Why robots.txt for SEO!

In basic terms, the robots file is what tells the spiders from the search engines what to look at.

-CG

Robots.txt file is which tells the search engine bots or spiders what to index and what not to index. As far as SEO is considered the robots.txt file is important if you try to hide the category archives, the tag archives etc from being indexed (you know, the duplicate content). But I think the top three search engines Google, Yahoo and Bing know what is duplicate content and what is not. I have seen some of the big websites use no robots.txt file and most of their category archives and tag archives are indexed without any penalty. So, the conclusion is robots.txt may not be that important for SEO. What is important for a website is great content and quality backlinks.

In robots.,txt file you can easily manage by using patterns (including wildcards) to control what not to index or follow, and saves you time.

robots.text is a text file from this text file mention Allow or Dis-allow. If mention allow means search engine crawls your page, Dis-allow doesn't crawls your page.

Your Robots.txt file is what tells the search engines which pages to access and index on your website on which pages not to. For example, if you specify in your Robots.txt file that you don’t want the search engines to be able to access your thank you page, that page won’t be able to show up in the search results and web users won’t be able to find it. Keeping the search engines from accessing certain pages on your site is essential for both the privacy of your site and for your SEO.

Robots.txt file is created, typically with notepad, to tell the good robots-'crawlers' that you don't wish them to crawl/index certain sections or pages in your site. A robots.txt file is not really needed, but I would put one in that allows all crawling.
Now, if you do wish to have pages not be in the index of the search engines, it is best to put in a meta in the head of the page or pages that says
<meta name="robots" content="noindex,nofollow" />
That is a page by page instruction to the robots to just leave the page alone, so then you wouldn't necessarily need to make an entry in the robots.txt file. Another note is that if you have pages that have already been indexed, but you want them out, you would have to add the meta I spoke of in the head of the page/pages as it's too late to try via a robots.txt file.
If any more questions, be sure to ask before you create a robots.txt file. Of course you can google it as there is tons of info out there.

Robots.txt's main use is to restrict robots or spiders to crawl a specific page. It is advisable to check before one move to promote the website.