Problem: URL restricted by robots.txt
- SEO |
Hi
In google webmaster tools I've found 3 crawl errors like "URL restricted by robots.txt".
This is a mystery to me. I suspect only 2 of the pages are restricted by robots.txt. One of the pages require user to be authenticated and the other is a error page. Both of these are pages which I don't want indexed. The last page is a page with no tag or anything in robots.txt indicating it is restricted by robots.txt.
Another why does google treat it as an error if I restrict googlebot access to a specific page.
any suggestions why I get the URL restricted by robots.txt?
In google webmaster tools I've found 3 crawl errors like "URL restricted by robots.txt".
This is a mystery to me. I suspect only 2 of the pages are restricted by robots.txt. One of the pages require user to be authenticated and the other is a error page. Both of these are pages which I don't want indexed. The last page is a page with no tag or anything in robots.txt indicating it is restricted by robots.txt.
Another why does google treat it as an error if I restrict googlebot access to a specific page.
any suggestions why I get the URL restricted by robots.txt?
-
loginname -
Thanks
{{ DiscussionBoard.errors[1913201].message }} -