Is Robots.txt any important for SEO?

10 replies
  • SEO
  • |
Is Robots.txt any important for SEO?
What are its purpose?
Please answer.
#important #robotstxt #seo
  • Profile picture of the author LloydMS
    robots.txt is a way that you can tell search engines not to crawl/index part of your site. You can specify that it not crawl a particular page or even an entire directory.

    It's not necessary for SEO. In most cases you probably don't really need to have a robots.txt file. And you can certainly screw up your entire site by having an improperly configured robots.txt.

    You only really need to use one if you don't want part of your site indexed. I use it sometimes to prevent the engines from crawling part of a membership site, for example. Or maybe admin pages. Or you could have different versions of content on your site - like a mobile or printer version - and you don't want those to be indexed to avoid duplicate content penalties.

    But for most basic sites, you don't have to worry about it.
    {{ DiscussionBoard.errors[7779562].message }}
    • Profile picture of the author biznics
      There's no SEO influence on your site ranking if you don't have a Robots.txt file.

      BUT, I am sure every site should have this file as no one wants to see "ordernow.php" or "signupform.html" page on Google ranking. (Yes, it can happen, these pages also rank if not included in Robots file).

      So to avoid indexing these (and similar) unwanted pages, using a sitemap is important.

      Good day,
      {{ DiscussionBoard.errors[7974500].message }}
  • Profile picture of the author funtler
    O i see. Thanks for information.
    Signature
    Solo Ads Provider
    Make $1k Per Sale => Join Here <=
    {{ DiscussionBoard.errors[7973667].message }}
  • Profile picture of the author InitialEffort
    Robot.txt is a file placed in a root directory. It tells the spiders which pages to visit and index and which pages not to. It offers several benefits including saving your bandwidth (since spiders won’t be visiting those areas which don’t have relevant information); it provides a very basic level of protection (it will keep users from finding the stuff you don’t want easily accessible via engines); it cleans up your logs; it allows you to prevent penalties related to duplicated content (with the robot.txt, you can prevent the dup content, if any, from being indexed), etc.
    {{ DiscussionBoard.errors[7973973].message }}
  • Profile picture of the author nikswf
    Its not necessary for every website but if you want to track with webmaster tools and want to get indexed quickly it is important.

    I recommend to use it.
    {{ DiscussionBoard.errors[7974073].message }}
  • Profile picture of the author lavathemes
    Do read this article also A Brief Guide to Robots.txt and Five Mistakes to Avoid and this one also Official Google statement: 39 changes in May (with actionable items). Hope these 2 links can better understand you the use of robot.txt file.
    {{ DiscussionBoard.errors[7974325].message }}
  • Profile picture of the author rasel786
    I am agree with LloydMS.
    {{ DiscussionBoard.errors[7974619].message }}
  • Profile picture of the author RedchilliMedia
    Robots.text file is a very important text file in the website which is preventing the nofollow area on the website...For example, the following Robots.txt command:

    User-agent: *

    Disallow: /thankyou

    …would block all search engine robots from visiting the following page on your website:

    http://www.yoursite.com/thankyou
    {{ DiscussionBoard.errors[7974636].message }}
    • Profile picture of the author Hansons
      Originally Posted by RedchilliMedia View Post

      Robots.text file is a very important text file in the website which is preventing the nofollow area on the website...For example, the following Robots.txt command:

      User-agent: *

      Disallow: /thankyou

      ...would block all search engine robots from visiting the following page on your website:

      http://www.yoursite.com/thankyou
      Thanks, I understood it properly today.
      Signature

      Is your website Hacked? Try -> www.sitebeak.com
      Is Google Analytics installed Properly? Test -> www.GAtective.com
      Impersonal Google search? Check -> www.impersonal.me

      {{ DiscussionBoard.errors[7974690].message }}
  • Profile picture of the author SASA Techno
    Yes Robot.txt is an important because robot txt is Only your page online but Google not crawl that page....
    {{ DiscussionBoard.errors[7974750].message }}

Trending Topics