11 {{ upvoteCount | shortNum }}
11 {{ upvoteCount | shortNum }}

Google bot can't access my robots.txt - need help in fixing it.

by satrap
8 replies
  • SEO
    • |
Hi guys,

While checking my ranking for one of my sites, I noticed all of them had dropped to nowhere.

So, I started digging and I found this message in my Google WMT:

Over the last 24 hours, Googlebot encountered 12 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%.

I have no clue how to go about this. Everything was working fine until yesterday. The Google bot could access the site and everything, but now this.

I would appreciate any help as to how to fix the issue.

By the way its a WP site and I am using WordPress SEO BY Yoast plugin (I don't know if it has anything to do with this or not, but just thought to include it in case.). Thanks in advance.
#access #bot #fixing #google #robotstxt
  • Profile picture of the author yukon
    Banned
    First check that you can access the robots.txt file by doing a search on your browser for:
    http://domain.com/robots.txt
    If you can't access the robots.txt file Google can't either.
    {{ DiscussionBoard.errors[8340885].message }}
  • Profile picture of the author dadamson
    Ensure that your robots.txt file is in the top directory of your website, and there are no duplicates of it.

    Check your robots.txt file for errors inside it. Perhaps it has been corrupted?

    Have you tried to recreate the robots.txt file and replace with the current one and then reset the Google WMT warnings?

    ^ If that doesn't do it I don't know what will.

    You can create a quick robots.txt file via a free online generator or a wordpress plugin.
    Signature
    {{ DiscussionBoard.errors[8340890].message }}
    • Profile picture of the author satrap
      I can access my file on my browser. All seems to be correct. In fact here is the actual content of my robots.txt file:

      Code:
      User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
      Isn't that what it supposed to look like?...

      Thanks again guys.
      Signature
      60 Awesome Ways to Make Money Without a Job
      .................................
      Check out my blog Survey Satrap featuring honest reviews of paid survey sites.
      {{ DiscussionBoard.errors[8343410].message }}
      • Profile picture of the author satrap
        I just noticed another strange thing.

        When I try to access my sitemap (which is created by Yoast SEO plugin) without the www, it works fine. But when I use www, it gives an error saying style sheet couldn't be loaded.
        I wonder if that has anything to do with this!?

        I never had any issue like this before on any of my sites and I haven't done anything different with this site, so I am really confused as to what the problem is.
        Signature
        60 Awesome Ways to Make Money Without a Job
        .................................
        Check out my blog Survey Satrap featuring honest reviews of paid survey sites.
        {{ DiscussionBoard.errors[8343592].message }}
        • Profile picture of the author satrap
          Ok, After digging deeper and seeing that everything looked fine from my side, I contacted the host.

          The support told me that "Google bot was blocked by their firewall" and that they have since fixed the issue and G bot should be able to access the site within 24-48 hours once DNS setting resolves.

          Has anyone had this happen with their host? ...

          I got this host just for this new site as I had read lots of good reviews about them. I dont know if this is something that is out of their hand and can happen to any host or what!?

          I never had this issue with any other host.

          Anyway, I just thought to share this in case anyone else ever comes across the same problem, they can know one more place to look for possible issues.
          Signature
          60 Awesome Ways to Make Money Without a Job
          .................................
          Check out my blog Survey Satrap featuring honest reviews of paid survey sites.
          {{ DiscussionBoard.errors[8343820].message }}
          • Profile picture of the author yukon
            Banned
            Originally Posted by satrap View Post

            Ok, After digging deeper and seeing that everything looked fine from my side, I contacted the host.

            The support told me that "Google bot was blocked by their firewall" and that they have since fixed the issue and G bot should be able to access the site within 24-48 hours once DNS setting resolves.

            Has anyone had this happen with their host? ...

            I got this host just for this new site as I had read lots of good reviews about them. I dont know if this is something that is out of their hand and can happen to any host or what!?

            I never had this issue with any other host.

            Anyway, I just thought to share this in case anyone else ever comes across the same problem, they can know one more place to look for possible issues.
            Who's your host?

            I've never had Hostgator block a robots.txt, I'm not sure why a host would even want to block that file.
            {{ DiscussionBoard.errors[8343852].message }}
            • Profile picture of the author satrap
              Originally Posted by yukon View Post

              Who's your host?

              I've never had Hostgator block a robots.txt, I'm not sure why a host would even want to block that file.
              Its webhostingpad.

              No it didn't block the file it blocked the Google bot from accessing it and it turn the site itself.

              I think it may have been an accidental thing where somehow their firewall saw Google bot as a fishy bot perhpas!? I am not sure though.
              Signature
              60 Awesome Ways to Make Money Without a Job
              .................................
              Check out my blog Survey Satrap featuring honest reviews of paid survey sites.
              {{ DiscussionBoard.errors[8344051].message }}
  • Profile picture of the author satrap
    The problem still is there. I talked to support team today and they told me I need to run a trace route of my domain. They think the issue may be from my server.
    I said I have 20 or so other sites (all hosted with other companies) and none of them are having this issue. If it was something from my server wouldn't that cause all of these sites to not be accessible to Google bot. Of course the reply was, "please run the trace route and email it to us and we will take a look at it".

    So, I did and am now waiting for their email.

    I don't know if I am jumping guns here or what, but I am already thinking about changing hosts as I have had the same issue a few weeks ago and I have been with this host only for the past 3-4 months with this one site.

    The site was already making money with all its ranking and now because of inaccessibility issue, of course all the rankings are gone from the index and so...

    I had read lots of good reviews about this host. So, I am still thinking to give them a chance, maybe this really is something from my server that I don't understand. Anyone had this issue before?..

    Sorry, rant over. I just wanted to get this off my chest, lol.
    Signature
    60 Awesome Ways to Make Money Without a Job
    .................................
    Check out my blog Survey Satrap featuring honest reviews of paid survey sites.
    {{ DiscussionBoard.errors[8351659].message }}

Trending Topics