How do you fix robots.txt file?

by smodha
2 replies
  • SEO
  • |
Hey WF!

My site isn't indexing because something is messed up in the robots.txt file. The error I've found in Google Webmaster Tools is "Google couldn't crawl your site because we were unable to access the robots.txt file".

Any suggestions? I've never had a problem before. I only used this WordPress theme.

Sanj.
#file #fix #robotstxt
  • {{ DiscussionBoard.errors[6888563].message }}
  • Profile picture of the author paulgl
    That is weird because you don't need a robots.txt file.
    Googlebots will look for this file, try to find it, then
    move on. It's no big deal. Yes, it will show a not
    found error, but who cares? No big deal.

    So, it's probably not a robots.txt problem. Problem is,
    as I stated above, googlebots look first for that file.
    Something is preventing the googlebot from accessing
    basically the whole site, not just the robots.txt.

    You are probably not having anything crawled.

    I must reiterate the robots.txt is a non-issue, but
    not being able to "access" it is a tip off.

    Not knowing anything about your server, host, settings,
    etc., I cannot give any more info. That will take a little more
    digging.

    Repeat: The problem has nothing to do with a robots.txt file.
    It's some other tweak, maybe your host, maybe the server,
    maybe something else is preventing google from crawling.

    Paul
    Signature

    If you were disappointed in your results today, lower your standards tomorrow.

    {{ DiscussionBoard.errors[6888585].message }}

Trending Topics