Have forgotten what a good robots.txt should read like

David · 06-30-2010, 05:33 AM

I've been using CMS's so long I've forgotten many basic html syntax basics

this request is so 2003... what *should* a good robots.txt read like?

I googled this:

Writing a good robots.txt file | SEO Tips

the author (and this post is a bit dated) has a long line of sites to block

http://urbanoalvarez.es/robots.txt

is there any good reason for mimicing this guys robots.txt?

NOTE: we have nothing to block from google, no membership site, we do not sell an ebook (yet) that we want the spiders to not tell everyone where they can get the ebook we're selling for free

is there a sound arguement for having such an extensive robots.txt?

06-30-2010, 05:33 AM	#1
David Advanced Warrior War Room Member Join Date: Apr 2003 Location: Frederick, Maryland, USA. Posts: 994 Blog Entries: 11 Thanks: 290 Thanked 18 Times in 18 Posts Social Networking Contact Info	Have forgotten what a good robots.txt should read like I've been using CMS's so long I've forgotten many basic html syntax basics this request is so 2003... what should a good robots.txt read like? I googled this: Writing a good robots.txt file \| SEO Tips the author (and this post is a bit dated) has a long line of sites to block http://urbanoalvarez.es/robots.txt is there any good reason for mimicing this guys robots.txt? NOTE: we have nothing to block from google, no membership site, we do not sell an ebook (yet) that we want the spiders to not tell everyone where they can get the ebook we're selling for free is there a sound arguement for having such an extensive robots.txt?
	David Bruce Jr of Frederick Web Promotions Google Places Ranking Improved \|