Google Search Console won’t crawl, Google states that Robot.txt blocking, thus it won’t recrawl

[ad_1]

Hello all,

I am trying to have my new wordpress site on HostGator crawled by Google Search Console after migrating from Squarespace. My last crawl was indexed with Squarespace May 30th. Since moving, I have received the following error messages on GSC.

Page Fetch – Failed: Robots.txt unreachable

I have worked with several agents at HostGator and they have said all security mods & DNS look fine and shouldn’t cause issues. They said there are no firewalls blocking the bot from crawling. They are unable to assist any longer, blaming site issues on coding.

There is no mention of “nofollow” in my HTML

I have installed LiteSpeed Cache, Yoast SEO, & WP External Links plugins, but didn’t notice any setting that would trigger a nofollow. I have disabled CDN on LSCache. I have tried to deactivate these plugins and recrawl with the same result.

I have amended my robot.txt several times and settled on allowing Yoast to create it. The same goes with my sitemap. I have tested RRT with google and crawl fails with the “URL is not available to Google” “Robots.txt unreachable”.

I have tested the site on 3rd party sites like [Technicalseo.com](http://Technicalseo.com) and the robot file is shown with no crawl restrictions. I have tested it on 4-5 others with no issues.

I am at a loss and would appreciate any help with the error.

**Details:**

WordPress: Discourage Search Engines – unchecked

https://preview.redd.it/zxelbsva4g6d1.png?width=567&format=png&auto=webp&s=9d9294c3a3fa59330eb2d312075c6da5765805ac

**Yoast SEO Sitemap:**

[https://www.rusticluxurycabins.com/sitemap\_index.xml](https://www.rusticluxurycabins.com/sitemap_index.xml)

**Robot.txt:**

[https://www.rusticluxurycabins.com/robots.txt](https://www.rusticluxurycabins.com/robots.txt)

# START YOAST BLOCK
# —————————
User-agent: *
Disallow:

Sitemap: https://www.rusticluxurycabins.com/sitemap_index.xml
# —————————
# END YOAST BLOCK

**GSC Error:**

https://preview.redd.it/a6zet7fs3g6d1.png?width=919&format=png&auto=webp&s=9e8fc1be70499db88fcdf42d08f09f844d7742bb

Thank you,

Kevin

[https://www.rusticluxurycabins.com](https://www.rusticluxurycabins.com)

[ad_2]