10 {{ upvoteCount | shortNum }}
10 {{ upvoteCount | shortNum }}

robot.txt please help

by pennyroll
8 replies
  • WEB DESIGN
    • |
Google webmaster tools says:

Network unreachable: robots.txt unreachable We were unable to crawl your Sitemap because we found a robots.txt file at the root of your site but were unable to download it. Please ensure that it is accessible or remove it completely.

I've checked in the root to remove it but it's not there and I'm really not sure how to fix this?

Edit: I should have added the website I'm having issues with.
#robottxt
  • Profile picture of the author skyelans
    Banned
    [DELETED]
    {{ DiscussionBoard.errors[7001787].message }}
    • Profile picture of the author zotown
      Originally Posted by skyelans View Post

      not always visable via FTP but you can typically replace it by uploading another robots.txt file. Also, make sure you upload in asicII mode and not binary. You can also log into the hosting control panel and use the file directory to locate the file.
      agreed. login to your control panel and locate file there. if you still cant see it. refresh and upload another robots.txt. this will solve your problem.
      Signature
      {{ DiscussionBoard.errors[7001906].message }}
    • Profile picture of the author pennyroll
      When I attempt to login to my Control Panel on godaddy, it says " error can't get directory info... I'm guessing that this is the real problem not the robots.txt file itself.
      Signature
      Pennyroll.com
      Tips to make money that work!
      {{ DiscussionBoard.errors[7002011].message }}
  • Profile picture of the author jamaks
    Hi, more than likely since you do have a robots.txt file on the root. The only thing I am not sure of is wether or not the sitemap should be compressed.
    Code:
    User-agent: *Disallow: /wp-admin/Disallow: /wp-includes/Sitemap: http://www.pennyroll.com/sitemap.xml.gz
    I am just more used to seeing sitemap.xml and do not know if the sitemap.xml.gz would make a difference. Jim
    {{ DiscussionBoard.errors[7002134].message }}
    • Profile picture of the author pennyroll
      Originally Posted by jamaks View Post

      Hi, more than likely since you do have a robots.txt file on the root. The only thing I am not sure of is wether or not the sitemap should be compressed.
      Code:
      User-agent: *Disallow: /wp-admin/Disallow: /wp-includes/Sitemap: http://www.pennyroll.com/sitemap.xml.gz
      I am just more used to seeing sitemap.xml and do not know if the sitemap.xml.gz would make a difference. Jim
      I probably should have clarified, it's actually this site I'm having trouble with. Is there any way you could take a look at that one? It shows up but when I fetch the website as google bot, from webmaster tools, it errors and says it can't reach robots.txt?
      Signature
      Pennyroll.com
      Tips to make money that work!
      {{ DiscussionBoard.errors[7002193].message }}
  • Profile picture of the author jamaks
    Hi, robots.txt on that site looks perfect.
    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Cannot see any problem with that so as you have surmised the fault must be with the host. Jim
    {{ DiscussionBoard.errors[7002442].message }}
    • Profile picture of the author pennyroll
      Originally Posted by jamaks View Post

      Hi, robots.txt on that site looks perfect.
      User-agent: *
      Disallow: /wp-admin/
      Disallow: /wp-includes/
      Cannot see any problem with that so as you have surmised the fault must be with the host. Jim
      Well, it is godaddy so it's easy to imagine that it's with the host... They've gone so far down hill since I started using them. Switching this month. Thanks for taking a look
      Signature
      Pennyroll.com
      Tips to make money that work!
      {{ DiscussionBoard.errors[7002473].message }}
  • Profile picture of the author jamaks
    Hi, looks like they have not fully recovered from that outrage earlier in the month right enough. Meant to say that with something like robots.txt that you know is on the root of the domain it is simply a case of typing name-of-site.com/robots.txt into the browser address bar and you can check firstly that it is there and secondly that it is written correctly. Hope you get fully operational soon. Jim
    {{ DiscussionBoard.errors[7003003].message }}
  • Profile picture of the author BenQ
    I can see your robots.tx fine -- and it looks proper as well.
    Signature
    {{ DiscussionBoard.errors[7008865].message }}

Trending Topics