[ad_1]
I am a bit confused, working on some parts of a website that are password protected because they are not ready yet. However I just read that security web crawlers can find the pages although they are password protected.
How to disallow any crawler from finding the pages , If that is the case, then can Security web crawlers also crawl a memberships plugin private pages ?
Those password protected pages, do they render HTML to say the page is protected ?
​
>then can Security web crawlers also crawl a memberships plugin private pages ?
Who is going to create a membership account for those security web crawlers?
The URL of the page could be found by a crawler potentially if it’s linked on the website somewhere or not excluded from the sitemap file (most SEO plugins have an option to remove pages from sitemaps if you do not wish the page to be shown in search engine results), so the URL would be public.
The actual content on the page however would need the password entered to be accessed. There’s the odd case/bug I’ve seen where password protected pages were cached by the server or the excerpt or SEO description was leaking information within the content so it’s not 100% secure but still useful for hiding content in some cases. A more secure option would be to require the user to login to access content such as membership plugins use. A crawler may still request those pages but won’t be able to access the content that would require a user to login.