9 {{ upvoteCount | shortNum }}
9 {{ upvoteCount | shortNum }}

How much important is robots.txt file in SEO

by pjohn
8 replies
  • SEO
    • |
Is it very important to have a robots.txt file for the SEO of your website? What are the benefits for having it and lose for not having it on our website?
Please explain. Thanks in anticipation.
#file #important #robotstxt #robottxt #seo
  • Profile picture of the author Moneyerr
    Robots.txt file allows or restricts to crawl some parts of your webpages. Search engines also find the location of the sitemap file in robots.txt which is very important in SEO view point. If you don't have robots.txt file, your server will return a 404 error page which can hurt your web rankings.
    {{ DiscussionBoard.errors[3845836].message }}
  • Profile picture of the author zoobie
    Not important at all. Unless you want to block some part of your site from Search Engine access.
    {{ DiscussionBoard.errors[3846474].message }}
    • Profile picture of the author paulgl
      404 errors happen all the time. They don't affect any rankings. Unless
      you deleted a page by mistake. 404 pages are there for a reason. They
      can make the visitor not leave your site, but find something else. A
      404 error for an unneeded robots.txt is moot. 99.9999999% of normal
      websites do not need a robots.txt

      Webmasters who check stats just panic in seeing these errors. So
      make one. Leave it blank. But then, what's the point?

      Paul
      Signature

      If you were disappointed in your results today, lower your standards tomorrow.

      {{ DiscussionBoard.errors[3846605].message }}
  • Profile picture of the author gittar1122
    Robots.txt tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored. Keeping the search engines from accessing certain pages on your site is essential for privacy of your site.

    Thanks
    Signature
    {{ DiscussionBoard.errors[3846591].message }}
  • Profile picture of the author App Developers
    to my knowledge it is only used to block certain pages from the bots. I have never used it or needed to use it. I am not sure why you would want to block them.
    Signature
    www.attractionexplained.com
    www.austinflightcheck.com
    {{ DiscussionBoard.errors[3847036].message }}
  • Profile picture of the author aspnaren
    robot.txt file contains the list of url that you dont want search engine bots t crawl from your sites..In some cases, for e.g, labels in blogger can be considered as copy content by SEs and you have more chance of getting punished by SEs...So its very important that you have a robots.txt if you label or categorize your content in your blogs
    {{ DiscussionBoard.errors[3847177].message }}
  • Profile picture of the author GuidoB
    I don't think it really matters if you have one!
    {{ DiscussionBoard.errors[3847438].message }}
  • Profile picture of the author rijans
    You can use robots.txt file to block duplicate links from search engines. So it's matter for SEO.
    {{ DiscussionBoard.errors[3849921].message }}

Trending Topics