3 replies
  • SEO
  • |
Can any one suggest me what is robots.txt?
#robotstxt
  • Profile picture of the author hernan10
    robots.txt is a file which tells the spiders (googlebot, yahoobot) which pages to search and index and which pages to leave alone...

    even if you want all pages to be indexed, its better if you have a robots.txt
    {{ DiscussionBoard.errors[711858].message }}
  • Profile picture of the author angilina
    Robots.txt file simply tell search engines about the pages that you do not/or do want to be crawled and get indexed. You can find more info about robots.txt file here:

    en.wikipedia.org/wiki/Robots.txt
    {{ DiscussionBoard.errors[712063].message }}
  • Profile picture of the author obj111
    handy for preventing dupe content getting into the index

    not all bots obey robots.txt
    {{ DiscussionBoard.errors[721434].message }}

Trending Topics