If a church is mentioned there, it means that at least some records are available for it at the LSHA, and its records can potentially be obtained by contacting the Archive, or visiting it in person. It does not guarantee in any way that the corresponding records are going to be available online. However, you can use the collection and inventory numbers obtained from the index to do targeted searches on the E-Reading Room website, provided by the archive. In order to do that, change the interface language to English, click on "Advanced Search", enter the collection and inventory numbers from the index into the "Fond No" and "Series No" fields (respectively) and set "Fond holder" to "Lietuvos valstybės istorijos archyvas". If, on the results page, you see a photo thumbnail next to the entry, then this book is available online, and you can click on it to view the pages. Otherwise, it may be available from other sources, as detailed in the rest of this page.
See also Guide to LSHA collection 669.
Also has books for many parishes in Poland (both Catholic and other confessions).