Robots.txt???

by 17 replies
19
what is Robots.txt??? how it is used???
#search engine optimization #robotstxt
  • robot.txt is a file that allow and disallow your website pages from search engines crawling.
  • You can use a special robots.txt to tell search engine spiders not to index the content of a page, and/or not scan it for links to follow. You dont have to use it, but some people like parts of their pages ignored by the spiders if they are not relevant to what the site is about
  • Robots.txt file is used to request Search Engine Spiders about the pages you want them to crawl or index or not.
    It is an important factor for the webpages to be indexed properly.
  • Robots.txt is a text file which can be uploaded in the root folder of your site. Sometimes you don't want the robots just roaming anywhere they like on your site.You can use robots.txt to block some pages on your site not be crawled by crawlers. Have a visit at this link to know about the effective use of robots.txt How to Write a robots.txt File
  • Banned
    [DELETED]
    • [1] reply
  • rebotex is file use to restrict google crawler to index your website pages
  • Robots.txt is a text files that used to tells engines that what to index and what not to index on a certain websites.


    It is written as:


  • A site that hides part that user doesn't want to be crawled by Google.
  • hi
    robot.txt is a file that allow and disallow your website pages from search engines crawling. So if ur page is in process disallow it..
  • Thanx for giving info..but how i will create it??
    • [1] reply
    • Hey Ruby,

      I already answered your question in above my thread.
  • I think you need to read very detailed manual : The Web Robots Pages
  • here's a handy robots.txt generator.....Robots.txt Generator - McAnerin International Inc.
  • Thanx all of you in making me understand one new thing..
  • Banned
    [DELETED]
  • A robots.txt is a permissions file that can be used to control which webpages of a website a search engine indexes. The file must be located in the root directory of the website for a search engine website-indexing program (spider) to reference,
    • [1] reply
    • The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages. For those spiders that obey the file, it provides a map for what they can, and cannot index. The file must reside in the root directory of your web. The URL path (web address) of your robots.txt file should look like this...

      /robots.txt
      • [1] reply
Join the discussion

Next Topics on Trending Feed