4 {{ upvoteCount | shortNum }}
4 {{ upvoteCount | shortNum }}

Google site diagnostics says my url is blocking their robots

by carnipotent
3 replies
  • SEO
    • |
Hi,

I seem to have an issue with google bots. The recommended action by google is to include the following in my robots.txt file as follows:

User-agent: Mediapartners-Google
Disallow:

If this is the case, does it mean that we are only allowing google bots to access my site and not other bots from other search engines?

Will this affect my Serp with other search engines such as Yahoo, Bing etc..

For some reasons, when my robots.txt file is as follows:

User-agent: *
Disallow:

Google bots are not able to access my sites even with the wildcard "*" in the user-agent line.

Can anyone provide some insight into why this is happening?

Thanks.
#blocking #diagnostics #google #robots #site #url
  • Profile picture of the author Crew Chief
    Originally Posted by carnipotent View Post

    Hi,

    I seem to have an issue with google bots. The recommended action by google is to include the following in my robots.txt file as follows:

    User-agent: Mediapartners-Google
    Disallow:

    If this is the case, does it mean that we are only allowing google bots to access my site and not other bots from other search engines?

    Will this affect my Serp with other search engines such as Yahoo, Bing etc..

    For some reasons, when my robots.txt file is as follows:

    User-agent: *
    Disallow:

    Google bots are not able to access my sites even with the wildcard "*" in the user-agent line.

    Can anyone provide some insight into why this is happening?

    Thanks.
    You need to change Disallow to Allow.

    Your robots.txt file is like that because that's how you left it. In some WP themes and other sites, that is the default setting.

    Change it and you will be fine!

    Giles, the Crew Chief
    Signature
    Tools, Strategies and Tactics Used By Savvy Internet Marketers and SEO Pros:

    ProSiteFlippers.com We Build Monetization Ready High-Value Virtual Properties
    {{ DiscussionBoard.errors[2803916].message }}
    • Profile picture of the author trustumar
      Check these

      google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

      robotstxt.org/robotstxt.html
      {{ DiscussionBoard.errors[2804004].message }}
  • Profile picture of the author mihu
    Banned
    [DELETED]
    {{ DiscussionBoard.errors[2804017].message }}
    • Profile picture of the author carnipotent
      Thanks for the help guys. Really appreciate it. I will do as Crew Chief says and see if it resolves the problem.

      Crew Chief, my robots.txt file is changed to :

      User-agent: *
      Allow:

      Is this correct?
      {{ DiscussionBoard.errors[2809097].message }}

Trending Topics