Question re: robots.txt. file

3 replies
In webmaster tools for one of my sites, the robots.txt file says that one url is blocked.

I have no idea what url this would be. I created a sitemap two years ago, but was the robots.txt file automatically created from that? On the sitemap, I recently (about six weeks ago) changed the sitemap to not include tag pages, author pages, things like that, but I never anywhere put in a url to be blocked. And I have no idea where to go to see what this url is.

Anyone reading this can probably tell that I barely know what I'm trying to talk about. I wasn't even sure which section of the forum to put this question in.

My site used to be ranked pretty well in google, but now it's nowhere to be found, although I'm still ranking in yahoo and bing. I did very little backlinking,and all of my content is original. However, google did cache my site just a few days ago. I did the "google fetch" function in webmaster tools for a few of the pages, and they were fetched with no problem.

But this thing about the the robots.txt file that's supposed to have one url blocked bothers me. Before I start a campaign to try to bring my site back, I want to make sure there's nothing on the technical side that is a problem.

Can anyone please educate me on this issue? Thanks so much.
#file #question #robotstxt
  • Profile picture of the author webpeon
    You should probably start by checking your main root directory to see if there is actually a robot.txt file in there.. If there isnt one available its possibly just a glitch from your webmaster tools, hard to say.. if there is one there, open the file and scroll through the list to see whats allowed and whats not..


    - The google ranking thing is most likely due to their recent penguin algorithm updates more so than whats on your site
    Signature
    Web 2 Mobile
    The Future of The Web
    {{ DiscussionBoard.errors[6431474].message }}
    • Profile picture of the author CatherineMay
      I opened the root directory of the site in question and couldn't see a robots.txt file listed. I wondered if it would be hidden in the sitemap or sitemap.xml.gz file, but couldn't open those without some other program.

      So, I did a search on that screen for robots.txt file and what came back was a robots.txt file for a site from a couple of years ago that I don't even own the domain for anymore.

      Very confusing...I guess I should just quit thinking about this?
      Signature

      {{ DiscussionBoard.errors[6434045].message }}
      • Profile picture of the author webpeon
        Originally Posted by CatherineMay View Post


        So, I did a search on that screen for robots.txt file and what came back was a robots.txt file for a site from a couple of years ago that I don't even own the domain for anymore.
        If its not relevant to the current site youre safe to delete it...
        Signature
        Web 2 Mobile
        The Future of The Web
        {{ DiscussionBoard.errors[6436818].message }}

Trending Topics