14 {{ upvoteCount | shortNum }}
14 {{ upvoteCount | shortNum }}

SEO Benefit in Robots.txt?

by Anteela
11 replies
  • SEO
    • |
My SEO Manager is pushing us to add a robots.txt file to all of our websites. He just wants the file there, even though 99% of them are not intended to have pages not indexed. I asked why we're doing this, he could not give me an answer other than, "it has an SEO benefit." I've tried to search online for SEO experts backing it up, but it's no use.

Another co-worker said that Google tries looking for the robots file for every website. When they don't find one at all, they generate a 404 error, which could hurt rankings. Given all of the millions of websites out there, many which do not have a robots file, I find this hard to believe. Maybe I'm wrong.

So my question is: Is there an SEO benefit to adding a Robots.txt file when you are not trying to block crawlers from exploring certain pages of your website? If so, what is the benefit?

Thanks for the help. I'm just trying to figure things out. We still use many inefficient or outdated SEO techniques in my office. I'm still learning, but I want to clean up our processes. You all rock.
#benefit #robotstxt #seo
  • Profile picture of the author HunterSnake
    Having a Robots.txt is good I think, even if it's just...

    Code:
    User-agent: *
Allow: /

Sitemap: http://www.site.com/sitemap.xml
    But I don't think that you get a huge, if any benefit, from it if you're not actually using it for what its intended.


    Block or remove pages using a robots.txt file - Webmaster Tools Help

    Robots exclusion standard - Wikipedia, the free encyclopedia
    {{ DiscussionBoard.errors[2482089].message }}
  • Profile picture of the author Mark Brian
    It's true that search engines look for the robots.txt. It's good to have it as it somewhat serves as an invitation to the spiders. The spiders will read that first before they index your site.

    I usually put this inside my robots.txt

    User-Agent: *
    Allow: /
    Signature

    {{ DiscussionBoard.errors[2482094].message }}
    • Profile picture of the author D Baker
      I can't say for a fact that having the file provides an SEO benefit, but I can say that I think it is a good idea to have the file and even a better idea to actually use it.

      If you are using wordpress than there are folders that you don't need the robots to index and look inside.
      Signature
      SENuke Review
      SENuke XCr Review
      Article Builder
      Unique Article Wizard
      {{ DiscussionBoard.errors[2482115].message }}
      • Profile picture of the author paulgl
        For 99.999999% of webmasters, no robots.txt is needed.
        We have no reason to block anything, really, Google is smart
        enough to know what to index and ignore for serps.

        The spiders will always look for one. Why? Because they
        want to see what part of your site to not crawl.
        (See above)

        If you have no robots.txt, you get a 404 error. No big deal.
        But many webmasters choke when the view stats and see
        page not found errors. Quite meaningless, really, so just
        add some sort of robots.txt if you wish. Set your mind
        at ease.

        However, speaking of 404, you need a custom one. You want
        a gentle landing. You don't want an error message to show.
        You want a soft landing, with a page of links to the main part
        of your site. Robots.txt or not, there are typos that will send
        people scrambling out of your site.

        Paul
        Signature

        If you were disappointed in your results today, lower your standards tomorrow.

        {{ DiscussionBoard.errors[2482370].message }}
        • Profile picture of the author Anteela
          Thank you so much! We use a CMS for most of our websites, which automatically generate one, however some of our "revamp" deals are in HTML. I didn't even consider customizing the 404 page. Thanks, paulgl. This is something that we need to pursue, definitely.
          {{ DiscussionBoard.errors[2483396].message }}
          • Profile picture of the author McBrett
            @Anteela - That really sucks that your manager can't give you a straight answer as to why you need to add a Robots.txt other than "it's good for SEO". He/she should really be able to do that.

            To play devil's advocate for your manager though, if you work at some sort of an agency one benefit of adding a robots.txt file is that it will look like you're doing more SEO work for your clients that requires very little time.
            Signature
            www.500aMonth.com - This is my blog.
            {{ DiscussionBoard.errors[2483630].message }}
            • Profile picture of the author sandinfo
              Hey, very informative and interesting. Looks like you put a lot of effort and research into it. Keep up the good work!!
              {{ DiscussionBoard.errors[3652313].message }}
        • Profile picture of the author StoneWilson
          Originally Posted by paulgl View Post

          For 99.999999% of webmasters, no robots.txt is needed.
          We have no reason to block anything, really, Google is smart
          enough to know what to index and ignore for serps.

          The spiders will always look for one. Why? Because they
          want to see what part of your site to not crawl.
          (See above)

          If you have no robots.txt, you get a 404 error. No big deal.
          But many webmasters choke when the view stats and see
          page not found errors. Quite meaningless, really, so just
          add some sort of robots.txt if you wish. Set your mind
          at ease.

          However, speaking of 404, you need a custom one. You want
          a gentle landing. You don't want an error message to show.
          You want a soft landing, with a page of links to the main part
          of your site. Robots.txt or not, there are typos that will send
          people scrambling out of your site.

          Paul
          Don't agree with this, I've seen lots of webmasters asked why their "Index of/" showed up on SERP? Why their sitemap showed up on SERP?... Robots.txt can protect those private information of our websites, and reduce the rate our website got hacked, especially when you're using wordpress.
          Signature
          Looking for godaddy renewal coupon? Check GodaddyRenewal.com!
          {{ DiscussionBoard.errors[3653198].message }}
  • Profile picture of the author Nomar86
    like the answers above, its easy and fast to add and se like it
    Signature

    Web Development, SEO, Wordpress, Money Making Ideas

    - Nomar

    {{ DiscussionBoard.errors[2482099].message }}
  • Profile picture of the author Devan Colby
    robots.txt in very important to business or company sites, you can use this file to restrict search engines to index your confidential part of you website.
    {{ DiscussionBoard.errors[3652391].message }}
  • Profile picture of the author zoobie
    You don't really need it unless you have something that wants to block SE access.
    {{ DiscussionBoard.errors[3653032].message }}

Trending Topics