Hello.
There is a PRO version on my website. Google indexes more than 15,000 pages a day. These are the filter pages. There are many of them, and using the usual brute force method, it turns out that the search robot will crawl hundreds of thousands of useless pages. All pages have meta tags robots=”noindex,nofollow”. My terrible mistake is not understanding how it works.
Google told me that the robot will still download the page to see the meta tags robots=”noindex,nofollow”.
Could you tell me, have your clients already encountered this problem? How was it solved?
As I understand it, ideally you should prohibit indexing not through meta tags, but in the server response httr header – X-Robots-Tag. And also, so that the filters are not in the form of links with the tag. Or at least assign all links by default rel=”nofollow”