Have forgotten what a good robots.txt should read like

by David 0 replies
I've been using CMS's so long I've forgotten many basic html syntax basics

this request is so 2003... what *should* a good robots.txt read like?

I googled this:

Writing a good robots.txt file | SEO Tips

the author (and this post is a bit dated) has a long line of sites to block

http://urbanoalvarez.es/robots.txt

is there any good reason for mimicing this guys robots.txt?

NOTE: we have nothing to block from google, no membership site, we do not sell an ebook (yet) that we want the spiders to not tell everyone where they can get the ebook we're selling for free

is there a sound arguement for having such an extensive robots.txt?
#search engine optimization #forgotten #good #read #robotstxt

Trending Topics