SEO Benefit in Robots.txt?

[Anteela]

My SEO Manager is pushing us to add a robots.txt file to all of our websites. He just wants the file there, even though 99% of them are not intended to have pages not indexed. I asked why we're doing this, he could not give me an answer other than, "it has an SEO benefit." I've tried to search online for SEO experts backing it up, but it's no use.

Another co-worker said that Google tries looking for the robots file for every website. When they don't find one at all, they generate a 404 error, which could hurt rankings. Given all of the millions of websites out there, many which do not have a robots file, I find this hard to believe. Maybe I'm wrong.

So my question is: Is there an SEO benefit to adding a Robots.txt file when you are not trying to block crawlers from exploring certain pages of your website? If so, what is the benefit?

Thanks for the help. I'm just trying to figure things out. We still use many inefficient or outdated SEO techniques in my office. I'm still learning, but I want to clean up our processes. You all rock.

[HunterSnake]

Having a Robots.txt is good I think, even if it's just...

User-agent: *
Allow: /

Sitemap: http://www.site.com/sitemap.xml

But I don't think that you get a huge, if any benefit, from it if you're not actually using it for what its intended.

YouTube - Can I use robots.txt to optimize...

Block or remove pages using a robots.txt file - Webmaster Tools Help

Robots exclusion standard - Wikipedia, the free encyclopedia

[Mark Brian]

It's true that search engines look for the robots.txt. It's good to have it as it somewhat serves as an invitation to the spiders. The spiders will read that first before they index your site.

I usually put this inside my robots.txt

User-Agent: *
Allow: /

[Nomar86]

like the answers above, its easy and fast to add and se like it

[D Baker]

I can't say for a fact that having the file provides an SEO benefit, but I can say that I think it is a good idea to have the file and even a better idea to actually use it.

If you are using wordpress than there are folders that you don't need the robots to index and look inside.

[paulgl]

For 99.999999% of webmasters, no robots.txt is needed.
We have no reason to block anything, really, Google is smart
enough to know what to index and ignore for serps.

The spiders will always look for one. Why? Because they
want to see what part of your site to not crawl.
(See above)

If you have no robots.txt, you get a 404 error. No big deal.
But many webmasters choke when the view stats and see
page not found errors. Quite meaningless, really, so just
add some sort of robots.txt if you wish. Set your mind
at ease.

However, speaking of 404, you need a custom one. You want
a gentle landing. You don't want an error message to show.
You want a soft landing, with a page of links to the main part
of your site. Robots.txt or not, there are typos that will send
people scrambling out of your site.

Paul

[Anteela]

Thank you so much! We use a CMS for most of our websites, which automatically generate one, however some of our "revamp" deals are in HTML. I didn't even consider customizing the 404 page. Thanks, paulgl. This is something that we need to pursue, definitely.

[McBrett]

@Anteela - That really sucks that your manager can't give you a straight answer as to why you need to add a Robots.txt other than "it's good for SEO". He/she should really be able to do that.

To play devil's advocate for your manager though, if you work at some sort of an agency one benefit of adding a robots.txt file is that it will look like you're doing more SEO work for your clients that requires very little time.

[sandinfo]

Hey, very informative and interesting. Looks like you put a lot of effort and research into it. Keep up the good work!!

[Devan Colby]

robots.txt in very important to business or company sites, you can use this file to restrict search engines to index your confidential part of you website.

[zoobie]

You don't really need it unless you have something that wants to block SE access.

[StoneWilson]

Originally Posted by paulgl

For 99.999999% of webmasters, no robots.txt is needed.
We have no reason to block anything, really, Google is smart
enough to know what to index and ignore for serps.

The spiders will always look for one. Why? Because they
want to see what part of your site to not crawl.
(See above)

If you have no robots.txt, you get a 404 error. No big deal.
But many webmasters choke when the view stats and see
page not found errors. Quite meaningless, really, so just
add some sort of robots.txt if you wish. Set your mind
at ease.

However, speaking of 404, you need a custom one. You want
a gentle landing. You don't want an error message to show.
You want a soft landing, with a page of links to the main part
of your site. Robots.txt or not, there are typos that will send
people scrambling out of your site.

Paul

Don't agree with this, I've seen lots of webmasters asked why their "Index of/" showed up on SERP? Why their sitemap showed up on SERP?... Robots.txt can protect those private information of our websites, and reduce the rate our website got hacked, especially when you're using wordpress.