7 {{ upvoteCount | shortNum }}
7 {{ upvoteCount | shortNum }}

Googlebot is blocked from my domain

by mc9320
6 replies
I recently added my site to Google Webmaster Tools but it says Googlebot is blocked from my domain. It displays the following

User-agent: *
Disallow: /

(+ it lists the sitemap ok)

and in test results...

Blocked by line 2: Disallow: /
Detected as a directory; specific files may have different restrictions

However when I go to mydomain.com/robots.txt it displays this, which I think is ok...

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

(+ it lists the sitemap ok)

I have wordpress installed and I have checked Privacy settings to see if search engines are being blocked, and I have deactivated the plugins just in case.

Any suggestions?
#blocked #domain #googlebot
  • Profile picture of the author shantanu
    A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages. (All respectable robots will respect the directives in a robots.txt file, although some may interpret them differently. However, a robots.txt is not enforceable, and some spammers and other troublemakers may ignore it. For this reason, we recommend password protecting confidential information.)
    You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file (not even an empty one).
    {{ DiscussionBoard.errors[5342583].message }}
  • Profile picture of the author Shoot
    Check your themes folder see if there happens to be a 2nd robot.txt in there.
    {{ DiscussionBoard.errors[5343351].message }}
  • Profile picture of the author mc9320
    Thanks for your suggestion Shoot but can't seem to see anything there
    {{ DiscussionBoard.errors[5343446].message }}
  • Profile picture of the author blaine w
    is this worth buying?
    {{ DiscussionBoard.errors[5343449].message }}
    • Profile picture of the author Shoot
      User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: Mediapartners-Google* Disallow: Allow: /* # digg mirror User-agent: duggmirror Disallow: / Sitemap: http://www.example.com/sitemap.xml
      {{ DiscussionBoard.errors[5343475].message }}

Trending Topics