www.robotstxt.org

It's a file you include on your site to instruct the search engine spiders on how you wish your pages to be crawled.

Carlo has already given you the link to the definitive resource.

Bill