I’m out of my depth with this and I don’t even know where to go now.

# So, I built a wordpress site for a client (designer, not a developer) and it’s not being crawled by google. (Note: it’s been more than 3 weeks). I hire an SEO guy who does all the things – optimization, robots, etc. The site is STILL not being crawled.

When I look into google search console, The Live test is good. I can literally see my pages when I go to tested pages, but it’s STILL not being crawled. Under “more info” in the tested pages, there are 6 out of 148 resources that aren’t being delivered. 4 of them are warnings and 2 of them have the following:

Googlebot blocked by [robots.txt](https://events.api.secureserver.net/robots.txt) XHR [https://events.api.secureserver.net/t/1/tl/event?dh=](https://events.api.secureserver.net/t/1/tl/event?dh=)

The [event.api.secureserver.net](http://event.api.secureserver.net/) info is external – Godaddy. So I spend 2 hours on the phone with them saying, “Help. Your bot is blocking googlebots from crawling.” Apparently, this is a wordpress compatability premium issue and I haven’t paid for that assistance, so the guy suggested that I simply update the robots.txt file to bypass this error. He’s speaking Alien and I’m speaking Farmer, but okay here we go.

I update the robots.txt file and add the second line followed by the sitemap. I save, clear the cache, go back to search console and re-run the test. NOTHING CHANGES. Same errors. Same problem.


Has anyone ever dealt with this? Any suggestions? If I delete the site and reload it onto a fresh wordpress and theme will that help?

  1. Go to the admin dashboard – Settings – Reading. Ensure the box labeled Discourage search engines from indexing this site is not checked.

    Also go to the homepage – right click and view page source. Do a search for noindex, this should not be found in the source code. If it is, something is blocking search engines.

  2. Update: I read elsewhere that it takes 24 hours for Google to crawl for new robots.txt updates. So I requested the call and I’ll wait until tomorrow to see.


This site will teach you how to build a WordPress website for beginners. We will cover everything from installing WordPress to adding pages, posts, and images to your site. You will learn how to customize your site with themes and plugins, as well as how to market your site online.

Buy WordPress Transfer