this request is so 2003... what *should* a good robots.txt read like?
I googled this:
Writing a good robots.txt file | SEO Tips
the author (and this post is a bit dated) has a long line of sites to block
is there any good reason for mimicing this guys robots.txt?
NOTE: we have nothing to block from google, no membership site, we do not sell an ebook (yet) that we want the spiders to not tell everyone where they can get the ebook we're selling for free
is there a sound arguement for having such an extensive robots.txt?