4 {{ upvoteCount | shortNum }}
4 {{ upvoteCount | shortNum }}

Robots.txt?

by jesicawillss
3 replies
  • SEO
    • |
What is robots.txt? Is it a part of on-page?
#robotstxt
  • Profile picture of the author marketgoal
    Robots.txt is also the factor of On page optimization. Basically Robots.txt is a text file you put on your website to tell search robots which pages you would like them not to visit. Here in this text file you can define that pages which you don't want to crawl by the crawler.
    Signature
    Free Samples
    {{ DiscussionBoard.errors[3045062].message }}
  • Profile picture of the author Devkumar
    Robots.txt is a text file where we can block the spider from calling some important files.
    {{ DiscussionBoard.errors[3045258].message }}
    • Profile picture of the author paulgl
      A normal website does not really need a robots.txt. The search bots will
      automatically look for one first, and if not found, will give a not found error.
      No big deal. Using robots.txt to block some pages is the correct way of doing
      it. There is no need to block anything, unless you have sensitive material
      readily found. There is no need to block privacy policy, etc., as google is
      smart enough to know what to do with those pages. There is no danger
      in having a privacy policy being indexed or showing up in SERPs (unlikely
      anyway, and no big deal).

      If you don't like seeing errors, just create a robots.txt and leave it blank.

      Paul
      Signature

      If you were disappointed in your results today, lower your standards tomorrow.

      {{ DiscussionBoard.errors[3045280].message }}

Trending Topics