# So, I built a wordpress site for a client (designer, not a developer) and it’s not being crawled by google. (Note: it’s been more than 3 weeks). I hire an SEO guy who does all the things – optimization, robots, etc. The site is STILL not being crawled.
When I look into google search console, The Live test is good. I can literally see my pages when I go to tested pages, but it’s STILL not being crawled. Under “more info” in the tested pages, there are 6 out of 148 resources that aren’t being delivered. 4 of them are warnings and 2 of them have the following:
Googlebot blocked by [robots.txt](https://events.api.secureserver.net/robots.txt) XHR [https://events.api.secureserver.net/t/1/tl/event?dh=](https://events.api.secureserver.net/t/1/tl/event?dh=)
The [event.api.secureserver.net](http://event.api.secureserver.net/) info is external – Godaddy. So I spend 2 hours on the phone with them saying, “Help. Your bot is blocking googlebots from crawling.” Apparently, this is a wordpress compatability premium issue and I haven’t paid for that assistance, so the guy suggested that I simply update the robots.txt file to bypass this error. He’s speaking Alien and I’m speaking Farmer, but okay here we go.
I update the robots.txt file and add the second line followed by the sitemap. I save, clear the cache, go back to search console and re-run the test. NOTHING CHANGES. Same errors. Same problem.
https://preview.redd.it/81094bye319d1.png?width=952&format=png&auto=webp&s=8e83f06d5d6fdfdd0e5952ec082c25d8e1830245
Has anyone ever dealt with this? Any suggestions? If I delete the site and reload it onto a fresh wordpress and theme will that help?
[ad_2]
Go to the admin dashboard – Settings – Reading. Ensure the box labeled Discourage search engines from indexing this site is not checked.
Also go to the homepage – right click and view page source. Do a search for noindex, this should not be found in the source code. If it is, something is blocking search engines.
If everything is good on the website, check out [https://tagparrot.com/](https://tagparrot.com/).
I use it to help speed up indexing
Change the Allow line after Googlebot from a / to a *
Update: I read elsewhere that it takes 24 hours for Google to crawl for new robots.txt updates. So I requested the call and I’ll wait until tomorrow to see.