Robots.txt???

[Ruby Tyagi]

what is Robots.txt??? how it is used???

[outdoorfountains]

robot.txt is a file that allow and disallow your website pages from search engines crawling.

[Pawpoint]

You can use a special robots.txt to tell search engine spiders not to index the content of a page, and/or not scan it for links to follow. You dont have to use it, but some people like parts of their pages ignored by the spiders if they are not relevant to what the site is about

[mukeshkumar@]

Robots.txt file is used to request Search Engine Spiders about the pages you want them to crawl or index or not.
It is an important factor for the webpages to be indexed properly.

[ghazia]

Robots.txt is a text file which can be uploaded in the root folder of your site. Sometimes you don't want the robots just roaming anywhere they like on your site.You can use robots.txt to block some pages on your site not be crawled by crawlers. Have a visit at this link to know about the effective use of robots.txt How to Write a robots.txt File

[Braylen]

Robots.txt file is a set of instructions that tell search engine robots which pages of your site to be crawled and indexed. In most cases, your site is consist of many files or folders i.e. admin folders, cgi-bin, image folder, which are not relevant to the search engines. Robots.txt helps tell spiders what is useful and public for sharing in the search engine indexes and what is not.

[ricky pounting]

rebotex is file use to restrict google crawler to index your website pages

[John Conner]

Robots.txt is a text files that used to tells engines that what to index and what not to index on a certain websites.


It is written as:


User-agent: *
Disallow: /

[Webster Logan]

A site that hides part that user doesn't want to be crawled by Google.

[webpageworker]

hi
robot.txt is a file that allow and disallow your website pages from search engines crawling. So if ur page is in process disallow it..

[Ruby Tyagi]

Thanx for giving info..but how i will create it??

[Yulia from DNP]

I think you need to read very detailed manual : The Web Robots Pages

[John Conner]

Originally Posted by Ruby Tyagi

Thanx for giving info..but how i will create it??

Hey Ruby,

I already answered your question in above my thread.

[seanpearse]

here's a handy robots.txt generator.....Robots.txt Generator - McAnerin International Inc.

[Ruby Tyagi]

Thanx all of you in making me understand one new thing..

[mogulskiholiday]

A robots.txt is a permissions file that can be used to control which webpages of a website a search engine indexes. The file must be located in the root directory of the website for a search engine website-indexing program (spider) to reference,

[AlbertSmiths]

The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages. For those spiders that obey the file, it provides a map for what they can, and cannot index. The file must reside in the root directory of your web. The URL path (web address) of your robots.txt file should look like this...

/robots.txt

[Webster Logan]

What is the kind of information that site owners does not want to be crawled?