What does "url blocked by robots.txt" mean in webmaster tools?

4 replies
  • SEO
  • |
Does this mean the google spiders can't find my website???
#tools #url blocked by robotstxt #webmaster
  • Profile picture of the author hassan007
    Yes.... The crawler will not crawl the directory or page that is present in robots.txt
    {{ DiscussionBoard.errors[6114534].message }}
  • Profile picture of the author addlinkweb
    Configure your robots.txt file and remove those pages name which you what Google to index, if you do not know how to make then make robots.txt in Google Webmaster Tool's, it will guide you how to create it
    {{ DiscussionBoard.errors[6114739].message }}
  • Profile picture of the author nicktyler
    Your robots.txt file is effectively banning the crawlers from indexing the url. You need to update/change it to allow/block the right url's. Search on Google or WMT for demos and tutorials about how to do this.
    Signature

    1000's of IT jobs in the UK online now at Dice

    {{ DiscussionBoard.errors[6114791].message }}
  • Profile picture of the author upendraets
    Yes, Google Can`t be found that website.
    {{ DiscussionBoard.errors[6116139].message }}

Trending Topics