9 {{ upvoteCount | shortNum }}
9 {{ upvoteCount | shortNum }}

I get that robot.txt problem for every site and can't get indexed

by phil.wheatley
7 replies
  • SEO
    • |
Hi Gang

I seem to have this problem every time I want to get a site indexed, it's always to do with that f****** robot.txt file. Last time I got around it by zapping the file altogether as Google Webmaster tools says you don't need it if you don't want to restrict the crawlers, and then you can add the following tag to pages you DO want to hide:

<meta name="robots" content="noindex">
The thing is, now it won't find any of my pages because it says: "robot.txt unreachable"

If you go on the safe assumption that I'm stupid, can anybody tell my how I can easily fix this.

Many thanks
Phil
#indexed #problem #robottxt #site
  • Profile picture of the author Ken
    Create /robots.txt file and leave it empty. Bots will find it and carry on as usual.
    {{ DiscussionBoard.errors[1657725].message }}
  • Profile picture of the author phil.wheatley
    Oh crikey,really, why can't they just say that on the site lol! Can I assume it should be placed in the main directory then where my idex.html and other pages sit? - Many thanks!
    Signature



    It's still not working for you??? Need direction?...
    ---->>>> BrainDirection.com <<<<----
    {{ DiscussionBoard.errors[1657736].message }}
    • Profile picture of the author Ken
      Right Phil. Main directory.
      {{ DiscussionBoard.errors[1657743].message }}
  • Profile picture of the author phil.wheatley
    Wow, cheers Ken, you've just managed to improve my life one just one line of text - Thanks very much, and this is my 500th post too hehe

    Thanks again
    Phil
    Signature



    It's still not working for you??? Need direction?...
    ---->>>> BrainDirection.com <<<<----
    {{ DiscussionBoard.errors[1657753].message }}
  • Profile picture of the author FredFarnes
    Originally Posted by phil.wheatley View Post

    Hi Gang

    I seem to have this problem every time I want to get a site indexed, it's always to do with that f****** robot.txt file. Last time I got around it by zapping the file altogether as Google Webmaster tools says you don't need it if you don't want to restrict the crawlers, and then you can add the following tag to pages you DO want to hide:

    <meta name="robots" content="noindex">
    The thing is, now it won't find any of my pages because it says: "robot.txt unreachable"

    If you go on the safe assumption that I'm stupid, can anybody tell my how I can easily fix this.

    Many thanks
    Phil
    Robot.txt is really easy to use, so I'm surprised you are having a problem.

    If you don't have the robots.txt file, you will be indexed. If you create an empty (blank) one, you will not receive that message.
    Signature

    So, you want to sell me another way to easily make "X" dollars in "X" days? ROFL too funny! IM success requires hard work and lots of time. Most newbies do not survive the steep learning curve. Anyone who says otherwise is probably selling you a fantasy.

    {{ DiscussionBoard.errors[1657803].message }}
  • Profile picture of the author phil.wheatley
    Hi Fred

    It's weird I know, I get this with all of my sites, it can take weeks or longer to get indexed for me despite having some great links, doing things in the right order and so on, I can't work it out. The first few times it was due to something the robt file was doing to prevent crawling, so after getting rid of that, I thought I wouldn't have problems, but I do.

    yeah, it should be simple right, I mean, I've done things 100 times more complex...but as for the indexing thing....it's that pesky robt file....haha, I bet when we start having robot wars in the future, I will be the first one on their list to exterminate! ;-) (don't worry, I'm not going mad, it's the Merlot talking, not helped by the fact I'm watching a Def Leppard convert from the 80s on DVD lol)
    Signature



    It's still not working for you??? Need direction?...
    ---->>>> BrainDirection.com <<<<----
    {{ DiscussionBoard.errors[1658297].message }}
  • Profile picture of the author magnusmora
    Phil, not sure if you have now sorted you problem. However I had something similar on my wordpress blogs. Anyhow to cut a long story short, found it was a setting in wordpress that was overiding the robot.txt file. To fix, go to settings, select privacy settings and then select

    I would like my blog to be visible to everyone, including search engines (like Google, Sphere, Technorati) and archivers

    This fixed it for me. Took about a week for Google to then index me. For some reason alot of my blogs default to the following setting

    I would like to block search engines, but allow normal visitors

    So now I check every site before launching. Cheers
    {{ DiscussionBoard.errors[1708935].message }}

Trending Topics