11 {{ upvoteCount | shortNum }}
11 {{ upvoteCount | shortNum }}

Robot.txt and google bot

by the_viking
10 replies
  • SEO
    • |
Hi,
I just got an email from Google telling me that their bot can't access to my CSS and JS files on my domain. Further down the email they basically tell me that this will have a negative effect on my rankings.

My robot.txt contains these lines:
Code:
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-content/
Disallow: /wp-admin/
Disallow: /wp-includes/
Anybody know what I should do? I haven't changed my robot.txt in years...
#bot #google #robottxt
  • Profile picture of the author paulgl
    Leave the robots.txt blank.

    your wp-includes or content probably contains the css, javascript.

    Google nowadays wants to see everything that makes your site run.

    Paul
    Signature

    If you were disappointed in your results today, lower your standards tomorrow.

    {{ DiscussionBoard.errors[10191727].message }}
  • Profile picture of the author the_viking
    Thanks, I will leave the robots.txt blank.
    {{ DiscussionBoard.errors[10191773].message }}
  • Profile picture of the author yourfriend78640
    Your robots.txt file should contain only following code

    User-agent: *
    Disallow: /wp-admin/
    these 2 lines are more than enough and it will resolve your issue
    {{ DiscussionBoard.errors[10191784].message }}
    • Profile picture of the author stolf
      Got the same message today for a few of my websites and all they contain in robot.txt is:


      User-agent: *
      Disallow: /wp-admin/


      So Im not sure what google have been up to today.
      {{ DiscussionBoard.errors[10191900].message }}
      • Profile picture of the author paulgl
        I thought google changed that a long time ago. You would have
        gotten that message a couple of years ago or so.

        Checking to see if google did another update with that, but
        I have found nothing.

        As an after thought as I'm typing, I am wondering about mobile
        tweaked sites. Many people fell into that trap and just junked up
        their site with nonsense, thinking mobile friendly was going to
        do something. Perhaps the mobile code is just the $%#@s.

        I see no reason to block the admin.

        Paul
        Signature

        If you were disappointed in your results today, lower your standards tomorrow.

        {{ DiscussionBoard.errors[10192030].message }}
      • Profile picture of the author yourfriend78640
        Originally Posted by stolf View Post

        Got the same message today for a few of my websites and all they contain in robot.txt is:


        User-agent: *
        Disallow: /wp-admin/


        So Im not sure what google have been up to today.
        ?????

        Okay remove this whole damn robots.txt file. Don't think Wordpress would have left something in wp-admin open that may cause security threat.
        {{ DiscussionBoard.errors[10192236].message }}
        • Profile picture of the author stolf
          I'm not sure if I should just leave it or what to do because of the 15 website I have only 5 got the message yesterday and they all contain the same txt in robot.txt file which is

          User-agent: *
          Disallow: /wp-admin/


          And when I fetch as google everything seems to work as it should.
          {{ DiscussionBoard.errors[10192735].message }}
  • Profile picture of the author yourfriend78640
    Hmm, I suggested you above code because that's what I did for my own site and I never received any error message from Google. This should work for you too.
    {{ DiscussionBoard.errors[10192752].message }}
    • Profile picture of the author stolf
      Okey I Think I found it.


      When I do a search for http://mywebsite.com/robots.txt I found:




      User-agent: *
      wp-admin
      wp-includes




      So I guess I just delete "wp-includes" and I should be good?
      {{ DiscussionBoard.errors[10192768].message }}
      • Profile picture of the author yourfriend78640
        Originally Posted by stolf View Post

        Okey I Think I found it.


        When I do a search for http://mywebsite.com/robots.txt I found:




        User-agent: *
        wp-admin
        wp-includes




        So I guess I just delete "wp-includes" and I should be good?
        yes deleting wp-includes from the code will help you reduce the errors. No need to include it in your robots.txt file
        {{ DiscussionBoard.errors[10192778].message }}

Trending Topics