Googlebot and robots.txt

razztek · 09-18-2009, 10:16 AM

I was just in my Google Webmasters account adding my URL and sitemap, and I noticed that when I "tested" it, it said something about not being able to browse my site because the robots.txt had "Disallow: /".

If you look through the configuration options there is one where you want your site to be crawled, which will change it to "Allow: /" (or whatever the syntax is).

My question: typically do you want an "allow" or "disallow"? This is a Wordpress site and I did nothing to change anything, so I guess Disallow was on by default. My first thought would be that "Allow" is better.

Thoughts?

trevord92 · 09-18-2009, 10:33 AM

Hi

Check out The Web Robots Pages

It also includes a checker which will tell you what your file is telling the various robots.

Trevor

razztek · 09-18-2009, 12:02 PM

That's perfect. Thanks!!

Matt Cromie · 09-18-2009, 04:18 PM

or get the robots meta plugin for wordpress this will give you the ability to fine tune how robots crawl your site.

09-18-2009, 10:16 AM	#1
razztek Rob RodenParker War Room Member Join Date: Feb 2005 Location: Area 51 Posts: 809 Thanks: 201 Thanked 170 Times in 152 Posts Social Networking Contact Info	Googlebot and robots.txt I was just in my Google Webmasters account adding my URL and sitemap, and I noticed that when I "tested" it, it said something about not being able to browse my site because the robots.txt had "Disallow: /". If you look through the configuration options there is one where you want your site to be crawled, which will change it to "Allow: /" (or whatever the syntax is). My question: typically do you want an "allow" or "disallow"? This is a Wordpress site and I did nothing to change anything, so I guess Disallow was on by default. My first thought would be that "Allow" is better. Thoughts?
	[WF Group] The Kindle Connection - Join Our Kindle Mastermind Group! Children's Short Stories Collection - HIGHLY RECOMMENDED!!

09-18-2009, 10:33 AM	#2
trevord92 HyperActive Warrior War Room Member Join Date: Feb 2005 Location: , , United Kingdom. Posts: 215 Thanks: 0 Thanked 26 Times in 19 Posts Contact Info	Re: Googlebot and robots.txt Hi Check out The Web Robots Pages It also includes a checker which will tell you what your file is telling the various robots. Trevor

09-18-2009, 12:02 PM	#3
razztek Rob RodenParker War Room Member Join Date: Feb 2005 Location: Area 51 Posts: 809 Thanks: 201 Thanked 170 Times in 152 Posts Social Networking Contact Info	Re: Googlebot and robots.txt That's perfect. Thanks!!
	[WF Group] The Kindle Connection - Join Our Kindle Mastermind Group! Children's Short Stories Collection - HIGHLY RECOMMENDED!!

09-18-2009, 04:18 PM	#4
Matt Cromie Share The Love War Room Member Join Date: Jun 2009 Posts: 40 Thanks: 2 Thanked 0 Times in 0 Posts	Re: Googlebot and robots.txt or get the robots meta plugin for wordpress this will give you the ability to fine tune how robots crawl your site.