Googlebot is blocked from my domain

by 6 replies
7
I recently added my site to Google Webmaster Tools but it says Googlebot is blocked from my domain. It displays the following

User-agent: *
Disallow: /

(+ it lists the sitemap ok)

and in test results...

Blocked by line 2: Disallow: /
Detected as a directory; specific files may have different restrictions

However when I go to mydomain.com/robots.txt it displays this, which I think is ok...

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

(+ it lists the sitemap ok)

I have wordpress installed and I have checked Privacy settings to see if search engines are being blocked, and I have deactivated the plugins just in case.

Any suggestions?
#programming #blocked #domain #googlebot
  • A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages. (All respectable robots will respect the directives in a robots.txt file, although some may interpret them differently. However, a robots.txt is not enforceable, and some spammers and other troublemakers may ignore it. For this reason, we recommend password protecting confidential information.)
    You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file (not even an empty one).
  • Check your themes folder see if there happens to be a 2nd robot.txt in there.
  • Thanks for your suggestion Shoot but can't seem to see anything there
  • is this worth buying?
    • [1] reply
    • User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: Mediapartners-Google* Disallow: Allow: /* # digg mirror User-agent: duggmirror Disallow: / Sitemap: http://www.example.com/sitemap.xml
      • [1] reply
  • Banned
    [DELETED]
Join the discussion

Next Topics on Trending Feed