6 {{ upvoteCount | shortNum }}
6 {{ upvoteCount | shortNum }}

robots.txt question

by timpears
5 replies
  • SEO
    • |
I just barely understand this stuff so I got a question. If my robots.txt showa as follows;
User-agent: *
Disallow: /wp-admin/

Is that allowing googlebot to review my site, except the admin area? Or not? I have had my site blocked from bots for the last couple days while I fix a mucked up site with many errors. But I remembered that today and changed it in the WP admin area. I didn't look at my robots.txt before, but5 this one I have now looks different than what WMT showed earlier. I am trying to get googlebot to do a search, but not sure I am doing it right.
#question #robotstxt
  • Profile picture of the author patadeperro
    Originally Posted by timpears View Post

    I just barely understand this stuff so I got a question. If my robots.txt showa as follows;
    User-agent: *
    Disallow: /wp-admin/

    Is that allowing googlebot to review my site, except the admin area? Or not? I have had my site blocked from bots for the last couple days while I fix a mucked up site with many errors. But I remembered that today and changed it in the WP admin area. I didn't look at my robots.txt before, but5 this one I have now looks different than what WMT showed earlier. I am trying to get googlebot to do a search, but not sure I am doing it right.
    Here you go Let me google that for you
    Signature
    {{ DiscussionBoard.errors[10197378].message }}
    • Profile picture of the author timpears
      Originally Posted by patadeperro View Post

      Yea well...
      Signature

      Tim Pears

      {{ DiscussionBoard.errors[10197458].message }}
      • Profile picture of the author patadeperro
        Originally Posted by timpears View Post

        Yea well...
        I mean if you cliked the link and read the first answer, you may have learned what you were asking for, because the first result explains how the robots.txt file is and how it should be configured
        Signature
        {{ DiscussionBoard.errors[10197463].message }}
        • Profile picture of the author paulgl
          robots.txt should be blank for the most part.

          Google sometimes wants us to block things like search results.

          If your admin directory is not password protected, do that now.
          You should not really need to, as you already must login, I assume.
          But the added login is just a precaution.

          If it is protected, you most certainly do not need to put that in a robots.txt.

          After all, google crawling is a moot point, but you care if people can
          hack in.

          Paul
          Signature

          If you were disappointed in your results today, lower your standards tomorrow.

          {{ DiscussionBoard.errors[10197580].message }}
  • Profile picture of the author nmwf
    Originally Posted by timpears View Post

    I just barely understand this stuff so I got a question. If my robots.txt showa as follows;
    User-agent: *
    Disallow: /wp-admin/

    Is that allowing googlebot to review my site, except the admin area? Or not?
    Yes, that's correct, Tim. All search bots (not just Google's) will scan your site except for the /wp-admin/ folder.

    Of course, there are some naughty bots that ignore the robots.txt file, but that's a separate issue. For now, what you have is sufficient for rule-abiding bots.
    Signature
    Write comprehensible articles on *any* topic in seconds with First Draft...
    First Draft's: Download | Add-Ons | Templates | Purchase | Support | Affiliates
    {{ DiscussionBoard.errors[10197410].message }}

Trending Topics