What does "url blocked by robots.txt" mean in webmaster tools?

by 4 replies
6
Does this mean the google spiders can't find my website???
#search engine optimization #tools #url blocked by robotstxt #webmaster
  • Yes.... The crawler will not crawl the directory or page that is present in robots.txt
  • Configure your robots.txt file and remove those pages name which you what Google to index, if you do not know how to make then make robots.txt in Google Webmaster Tool's, it will guide you how to create it
  • Banned
    [DELETED]
  • Your robots.txt file is effectively banning the crawlers from indexing the url. You need to update/change it to allow/block the right url's. Search on Google or WMT for demos and tutorials about how to do this.
  • Yes, Google Can`t be found that website.

Next Topics on Trending Feed