Desktop page not crawlable due to robots.txt – How to format in the plugin?
Desktop page not crawlable due to robots.txt – How to format in the plugin?
[ad_1]
I’m working on setting up my Google Merchant Center account, and I got an error message that some of my products were not crawlable. When I looked at the solution from Google, this is the support page they provide: https://support.google.com/merchants/answer/6098185?hl=en
On the page, they suggest this:
You can allow a full-site crawl by changing your robots.txt file as follows:
User-agent: Googlebot Disallow:
User-agent: Googlebot-image Disallow:
I have two problems.
I don’t know how to get the plugin to format the robots.txt like Google wants. If you look, the directory path seems to require a path. What should I put here?
2. When I look at my actual robots.txt file, it shows differently.
https://automospeedcrew.com/robots.txt
Any ideas?
The page I need help with: [log in to see the link]
