Google site diagnostics says my url is blocking their robots

carnipotent · 10-31-2010, 10:07 PM

Hi,

I seem to have an issue with google bots. The recommended action by google is to include the following in my robots.txt file as follows:

User-agent: Mediapartners-Google
Disallow:

If this is the case, does it mean that we are only allowing google bots to access my site and not other bots from other search engines?

Will this affect my Serp with other search engines such as Yahoo, Bing etc..

For some reasons, when my robots.txt file is as follows:

User-agent: *
Disallow:

Google bots are not able to access my sites even with the wildcard "*" in the user-agent line.

Can anyone provide some insight into why this is happening?

Thanks.

Crew Chief · 10-31-2010, 10:56 PM

Quote:

Originally Posted by carnipotent

Hi,

I seem to have an issue with google bots. The recommended action by google is to include the following in my robots.txt file as follows:

User-agent: Mediapartners-Google
Disallow:

If this is the case, does it mean that we are only allowing google bots to access my site and not other bots from other search engines?

Will this affect my Serp with other search engines such as Yahoo, Bing etc..

For some reasons, when my robots.txt file is as follows:

User-agent: *
Disallow:

Google bots are not able to access my sites even with the wildcard "*" in the user-agent line.

Can anyone provide some insight into why this is happening?

Thanks.

You need to change Disallow to Allow.

Your robots.txt file is like that because that's how you left it. In some WP themes and other sites, that is the default setting.

Change it and you will be fine!

Giles, the Crew Chief

trustumar · 10-31-2010, 11:26 PM

Check these

google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

robotstxt.org/robotstxt.html

carnipotent · 11-01-2010, 11:43 PM

Thanks for the help guys. Really appreciate it. I will do as Crew Chief says and see if it resolves the problem.

Crew Chief, my robots.txt file is changed to :

User-agent: *
Allow:

Is this correct?

10-31-2010, 10:07 PM	#1
carnipotent Warrior Member Join Date: Oct 2010 Posts: 8 Thanks: 0 Thanked 1 Time in 1 Post	Google site diagnostics says my url is blocking their robots Hi, I seem to have an issue with google bots. The recommended action by google is to include the following in my robots.txt file as follows: User-agent: Mediapartners-Google Disallow: If this is the case, does it mean that we are only allowing google bots to access my site and not other bots from other search engines? Will this affect my Serp with other search engines such as Yahoo, Bing etc.. For some reasons, when my robots.txt file is as follows: User-agent: * Disallow: Google bots are not able to access my sites even with the wildcard "*" in the user-agent line. Can anyone provide some insight into why this is happening? Thanks.

10-31-2010, 11:26 PM	#3
trustumar Active Warrior Join Date: Sep 2010 Posts: 91 Thanks: 6 Thanked 3 Times in 3 Posts	Re: Google site diagnostics says my url is blocking their robots Check these google.com/support/webmasters/bin/answer.py?hl=en&answer=156449 robotstxt.org/robotstxt.html
	Coming Soon!

11-01-2010, 11:43 PM	#4
carnipotent Warrior Member Join Date: Oct 2010 Posts: 8 Thanks: 0 Thanked 1 Time in 1 Post	Re: Google site diagnostics says my url is blocking their robots Thanks for the help guys. Really appreciate it. I will do as Crew Chief says and see if it resolves the problem. Crew Chief, my robots.txt file is changed to : User-agent: * Allow: Is this correct?