4 {{ upvoteCount | shortNum }}
4 {{ upvoteCount | shortNum }}

Problems with my Robots.txt

by Rhadoo7
3 replies
So I have a website with a custom domain bough through Blogger. I haven't made any modifications to the robots.txt file, so I guess this is the default for all blogspot blogs. It looks like this:

User-agent: Mediapartners-Google
Disallow: User-agent: *

Disallow: /search
Allow: /

The problem is with the "Disallow: /search". Because of this, I can't submit certain pages to "Fetch as Google". These are pages like : www.example.com/search/label/Category1

My question is: is it ok if I remove the "/search" ? Or should I just leave it like that? I never made any modifications to the robots.txt before, so I don't want to do anything bad...that's why I'm asking

Thanks!
#problems #robotstxt
  • Profile picture of the author dvduval
    I'm not sure what you are trying to avoid, but you could delete the robots.txt file and your site would operate fine.
    Signature
    It is okay to contact me! I have been developing software since 1999, creating many popular products like phpLD.
    {{ DiscussionBoard.errors[7462943].message }}
  • Profile picture of the author sunray
    Robots.txt is there to tell search engines which parts of the website they are NOT welcome to crawl. You don't have to enter Allow statements, only those you wish to avoid being indexed. I wouldn't delete this file though, since search engine crawlers ask for it, and if it's not there, it gives your system a 404 error.

    In your case, you can just delete the Disallow: /search line.
    Signature
    {{ DiscussionBoard.errors[7463187].message }}
  • Profile picture of the author aivil
    For users, specially Blogger Blogspot users this is one of the most trusted and working robots.txt file which you can use -

    User-agent: *
    Disallow:
    Allow: /
    Sitemap: http://www.example.com/atom.xml?redi...ax-results=500

    I have searched on my sites and found this to be most successful for my own Blogger blogs.

    I mentioned this for users out there who are searching for it. I hope they find it useful.
    Signature
    Top url Submission Sites | Get Free Backlinks
    Top 100 Ad Networks
    {{ DiscussionBoard.errors[8774743].message }}

Trending Topics