4 {{ upvoteCount | shortNum }}
4 {{ upvoteCount | shortNum }}

It says there is a robots.txt file blocking crawlers, yet there isn't one there!

by phil.wheatley
3 replies
Hi Gang

I have plenty of wordpress blogs, and know how to use the sitemap builder plugin, submit them to sitemap submitter within webmaster tools, but one f**king little blog of mine keeps saying it couldn't access my urls due to a robots.txt file in the root directory.

I've check the folders online a million times now but it just doesn't exist.

Any ideas???

Cheers
Phil
#blocking #crawlers #file #robotstxt
  • Profile picture of the author Ben Bergmann
    Did you try to go to www.yourdomain.com/robots.txt in your browser and see if there's anything at all?
    {{ DiscussionBoard.errors[2287748].message }}
  • Profile picture of the author Quentin
    Just make a new one in Google webmaster tools and add it. Make it allow.

    That should fix it.

    Q
    Signature
    {{ DiscussionBoard.errors[2287770].message }}
    • Profile picture of the author wk_lee
      Banned
      [DELETED]
      {{ DiscussionBoard.errors[2287792].message }}
      • Profile picture of the author Elle Holder
        Is it an addon domain? Is the main domain blocked by robots.txt?

        I don't know if that's the way it's supposed to work, but when I switched to Hostgator, this made me crazy for months. My old host didn't use addons, each new domain was self sufficient. But when I moved to HG, a made my initial domain nofollow. Then for months, every frigging site I make would say it was blocked by robots.txt when there was no robots.txt! So I would have to make a robots.txt to override the one I couldn't find.

        Finally, I figured out it was due to the main domain being blocked.
        Signature

        {{ DiscussionBoard.errors[2288188].message }}

Trending Topics