Robots.txt

[meetmalika]

Hello all friends,
What is the Robots.txt file and what is the use of this file.How we can create this file?

[furniture7]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

[adskumar]

Robots.txt is a text file its duty is to hide some files from google robots

[remmi]

robot.txt is a text file which you put on your site to tell search engine bot not to crawl or index the pages which you do want. For e.g. If you have two version of a particular page, and if Google index both the pages then it will penalize for using same content. In order to avoid this, you can use robot.txt file for not allowing to crawl that page.

[tradereader]

The robots exclusion standard, also known more commonly as Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. A Robots.txt file is frequently used by search engines to categorize and archive web pages, or by webmasters to proofread source codes.

[antony05]

There is a hidden, relentless force that permeates the web
and its billions of web pages and files, unbeknownst to the majority of us sentient beings. I'm talking about search engine crawlers and robots here. Every day hundreds of them go out and scour the web, whether it's Google trying to index the entire web, or a spam bot collecting any email address it could find for less than honorable intentions. As site owners, what little control we have over what robots are allowed to do when they visit our sites exist in a magical little file called "robots.txt."

[webgnomes]

The robots.txt file is a text file that informs search engine crawlers which pages you'd like them NOT to index. For example, if you want to keep them from indexing everything under your private directory, you would include a Disallow: /private/ field. For even more information about robots.txt, check out this guide: A robots.txt File Guide That Won’t Put You to Sleep