6 {{ upvoteCount | shortNum }}
6 {{ upvoteCount | shortNum }}

GoogleBot Emergency

by piinc
5 replies
Can someone explain in simple Layman's terms how to unblock Googlebots Webmaster Tool from my site????

Here is the test result (I've already checked my privacy settings in wordpress)

Blocked by line 4: Disallow: /
Detected as a directory; specific files may have different restriction

User-Agent: MJ12bot
Disallow:
User-agent: *
Disallow: /
allow: /example/
allow: / example /
allow: / example /
allow: / example /
allow: / example /
allow: / example /
allow: / example /
allow: / example /
allow: / example /
allow: / example /
#disallow #emergency #googlebot #robot.txt
  • Profile picture of the author limerickbob
    This is a solution I found at developerside.net. Hope it helps:

    You should try to access your-domain\robots.txt URL in your browser to see what you get. It might not even exist. If it does, it can block bots, but only directly, and not via any type of a metric (ex: by # of accesses), so take a look in it.

    The only other configuration I can think of that could cause a problem is with mod_security rule sets but it's unlikely.
    {{ DiscussionBoard.errors[5574164].message }}
    • Profile picture of the author piinc
      Originally Posted by limerickbob View Post

      This is a solution I found at developerside.net. Hope it helps:

      You should try to access your-domainrobots.txt URL in your browser to see what you get. It might not even exist. If it does, it can block bots, but only directly, and not via any type of a metric (ex: by # of accesses), so take a look in it.

      The only other configuration I can think of that could cause a problem is with mod_security rule sets but it's unlikely.
      I've checked it and it shows the same syntax. What next?
      {{ DiscussionBoard.errors[5574966].message }}
  • Profile picture of the author ibacklinkpro
    It's the Disallow: / that is the culprit. It should be:

    Code:
    User-Agent: MJ12bot
Disallow:

User-agent: *

Allow: /

Disallow: /example/dontspiderthis/

    Your method should be allow everything and block the stuff that shouldn't be spidered... right now you are saying block everything and only allow these example directories.... that is a bad idea...
    Signature

    Here is How to Steal Your Competitor’s High PR Backlinks:
    http://www.warriorforum.com/warrior-...free-demo.html

    {{ DiscussionBoard.errors[5574204].message }}
    • Profile picture of the author piinc
      Originally Posted by ibacklinkpro View Post

      It's the Disallow: / that is the culprit. It should be:

      Code:
       
User-Agent: MJ12bot
Disallow:
 
User-agent: *
 
Allow: /
 
Disallow: /example/dontspiderthis/

      Your method should be allow everything and block the stuff that shouldn't be spidered... right now you are saying block everything and only allow these example directories.... that is a bad idea...
      Thanks for the advice. Here's what it shows now:

      "Allowed by line 4: allow: /
      Detected as a directory; specific files may have different restrictions"

      I'm doing this directly in GWT and then testing.
      {{ DiscussionBoard.errors[5574948].message }}
  • Profile picture of the author piinc
    OK, set it up as follows:

    User-agent: *
    Disallow: /wp-admin
    Allow: /

    User-agent: MJ12bot
    Disallow: /
    Disallow: /wp-admin

    The problem is, Google Webmaster Tools does not reflect mydomain/robots.txt syntax shown above thus is considering the wrong syntax and thus is blocking it still.

    Any advice appreciated - I know I'm very close!!!
    {{ DiscussionBoard.errors[5575462].message }}

Trending Topics