Robots.txt and Wordpress

by 1 replies
1
Normally when building a Wordpress site I set it to private before its finished. On my new site I did this and then took the privacy off on completion. I noticed Google still had not indexed it after a couple of days and when I submit the sitemap to Google Webmaster, it says its blocked by the Robots.txt.

I know Wordpress creates a virtual Robots so I created a manual one to see what happens, but still it says it is been blocked. Should I just wait and see - normally I have never had a problem. The Robots.txt look like this:

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes

Sitemap: http://www.xyxyx.com/sitemap.xml.gz

thanks
#search engine optimization #robotstxt #wordpress
  • Google may not have grabbed the new one yet. I think I read somewhere they only check once every 24 hours. Be patient and check again tomorrow.

    Your new one looks right.
Join the discussion

Next Topics on Trending Feed

  • 1

    Robots.txt and Wordpress

    Normally when building a Wordpress site I set it to private before its finished. On my new site I did this and then took the privacy off on completion. I noticed Google still had not indexed it after a couple of days and when I submit the sitemap to Google Webmaster, it says its blocked by the Robots.txt. I know Wordpress creates a virtual Robots so I created a manual one to see what happens, but still it says it is been blocked. Should I just wait and see - normally I have never had a problem. The Robots.txt look like this: