6 {{ upvoteCount | shortNum }}
6 {{ upvoteCount | shortNum }}

does my robots.txt block my URLs?

by wijasenna
5 replies
  • SEO
    • |
just found this on my webmasters tools -- crawler access:

Blocked URLs: 140
Downloaded: 20 hours ago
Status: 200 (success)

what does it mean? does my robots.txt block my urls to be crawled by spiders? if so, how to fix it?

thanks!!
#block #robotstxt #urls
  • Profile picture of the author socialbookmark
    Yes its possible. To re-generate your robots.txt, you can sue Google webmaster tools. It helps you to allow/disallow any part of your website to be indexed and then upload it to your server. If you tell us the link of your robots.txt file, it can be checked to see if it prevents indexing your pages.
    Signature

    I love warriorforum. zendegiyesabz

    {{ DiscussionBoard.errors[6011011].message }}
  • Profile picture of the author OliviaSSLGuru
    Try New Robots.txt with below enlisted text

    User-Agent: *
    Allow: /
    sitemap: [your xml site map]
    {{ DiscussionBoard.errors[6011153].message }}
  • Profile picture of the author Kreative4
    Robots.txt file are usually created to control crawling spiders of different search engines. So it is possible to block spiders. Check your robot file and find out the disallow areas/bots and fix it.
    Signature
    signature coming soon...
    {{ DiscussionBoard.errors[6011344].message }}

Trending Topics