robots.txt errors

Lloyd Buchinski · 03-16-2010, 08:36 PM

I like to keep my error log down to help when there is a screw up. I always thought that this in the head was enough for the robots.

<meta name="robots" content="index, follow," />

But I keep getting them looking for public/robots.txt and creating an error line. (snarl)

Should I also include a text file in the public folder for them? I guess the content would be

user-agent: *
disallow:

I also hate clogging up my public folder so adding a file for 2 db seems a bit tedious too. (groan) (db is geek speak for dozen bytes)

Also while I was googling this I ran into quite a few comments that there is no point in having the meta tag I quoted above unless you want to disallow. The default behavior for robots is to follow. Guess i should delete them all?

Anybody have any comments about any of this and happen to feel a little chatty?

aarondwyer · 03-17-2010, 07:21 PM

Just put in a robots.txt file in your root web directory, and then make sure your website is in Google Webmasters and that it validates your robots.txt.

Also be aware that newer versions of Wordpress have a virtual robots.txt

Use the meta to dissallow as you have said.

Aaron

03-16-2010, 08:36 PM	#1
Lloyd Buchinski copy and paste geek War Room Member Join Date: Jan 2005 Location: Calgary Posts: 1,390 Thanks: 68 Thanked 168 Times in 140 Posts	robots.txt errors I like to keep my error log down to help when there is a screw up. I always thought that this in the head was enough for the robots. <meta name="robots" content="index, follow," /> But I keep getting them looking for public/robots.txt and creating an error line. (snarl) Should I also include a text file in the public folder for them? I guess the content would be user-agent: * disallow: I also hate clogging up my public folder so adding a file for 2 db seems a bit tedious too. (groan) (db is geek speak for dozen bytes) Also while I was googling this I ran into quite a few comments that there is no point in having the meta tag I quoted above unless you want to disallow. The default behavior for robots is to follow. Guess i should delete them all? Anybody have any comments about any of this and happen to feel a little chatty?
	There are 10 types of people, those that understand binary and those that don't. The KimW WSO

03-17-2010, 07:21 PM	#2
aarondwyer HyperActive Warrior Join Date: Feb 2006 Location: Brisbane, Australia. Posts: 180 Thanks: 4 Thanked 13 Times in 5 Posts Social Networking	Re: robots.txt errors Just put in a robots.txt file in your root web directory, and then make sure your website is in Google Webmasters and that it validates your robots.txt. Also be aware that newer versions of Wordpress have a virtual robots.txt Use the meta to dissallow as you have said. Aaron
	http://www.localranktracker.com - New Local Google Places Rank Tracker Beta. Windows, MAC, Linux http://backup-smart.com - Automatically And Safely Backup All Your cPanel Websites To Your Local Computer, Get your Free 7 Day Trial of this new software. Windows, MAC, Linux