4 {{ upvoteCount | shortNum }}
4 {{ upvoteCount | shortNum }}

Robots.txt need/don't need/ what's the skinny?

by David
3 replies
Hi gang, I seldom post in this sub section since I've been using CMS progs off the shelf and lost most of my coding skills (concentrating on writing content)

While researching on Panda changes in the algo, I noticed a trend:

old tactics seemed to just not get talked about,
  • no more posts on x after 2007
  • no posts on Y after 2008
  • no posts on Z after 2009
and so on... got me remembering a few sites that talked about blackhat or greyhat, or greyish hat.

This post put forth the idea that if your Robots.txt file was like this one, you'd be telling the SE to not index your site.

Web Designer Mag should fix its SEO - Yoast.


Out of curiosity I checked my robots.txt file and it was "bad?" like the one described.

SOoooo, I googled What should my robots.txt file say

this one said:
The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
kinda scary...

this one from an official Google Webmaster Central Support Forum said:
User-agent: *
Disallow:

Sitemap: http://www dot somesite dot com/sitemap.xml

that'll be fine (and no, that doesn't disallow everything)
----
so who is full of it?

These 2 are contradicting each other.. either that or I'm not understanding the premise

any one shed light on this gang?
#need or #need or dont #robots.txt #robotstxt #robotstxt file #skinny
  • Profile picture of the author DavidG
    Robots are something I don't bother. If there is something you don't want google to see then use Robots otherwise it doesn't matter.

    I don't touch them and I still rank high.

    regZ
    {{ DiscussionBoard.errors[4539012].message }}
    • Profile picture of the author unnatural
      Originally Posted by regZ View Post

      Robots are something I don't bother. If there is something you don't want google to see then use Robots otherwise it doesn't matter.

      I don't touch them and I still rank high.

      regZ
      Agreed, I don't bother with robots files unless there is something I really don't want to be indexed and I do just fine.

      The less time spent doing mundane tasks the better
      {{ DiscussionBoard.errors[4551880].message }}
  • Profile picture of the author Ken Durham
    notice the difference,
    Disallow: /
    Disallow:

    two different things.
    The first gives a command of what to block. From the root on in this case.
    The second is empty, so it is not giving any directories to block.
    Blocking admin sections are good, but I would rather block them from the actual pages with:
    <meta name="robots" content="noindex,nofollow">

    This way a human reading your robots.txt will not find directories/files that you want to stay hidden and keep out of the searches
    Signature

    yes, I am....

    {{ DiscussionBoard.errors[4552594].message }}

Trending Topics