Problem: URL restricted by robots.txt

loginname · 03-26-2010, 03:24 AM

Hi

In google webmaster tools I've found 3 crawl errors like "URL restricted by robots.txt".

This is a mystery to me. I suspect only 2 of the pages are restricted by robots.txt. One of the pages require user to be authenticated and the other is a error page. Both of these are pages which I don't want indexed. The last page is a page with no tag or anything in robots.txt indicating it is restricted by robots.txt.

Another why does google treat it as an error if I restrict googlebot access to a specific page.

any suggestions why I get the URL restricted by robots.txt?

loginname · 03-26-2010, 03:33 AM

My robots.txt is like this (a shorten version):
User-agent: *
Sitemap: http://www.mydomain.com/sitemap.xml
Allow: /
Disallow: /folder1/
Disallow: /folder2/
Disallow: /Error.htm

03-26-2010, 03:24 AM	#1
loginname HyperActive Warrior War Room Member Join Date: Oct 2009 Posts: 275 Thanks: 10 Thanked 4 Times in 4 Posts	Problem: URL restricted by robots.txt Hi In google webmaster tools I've found 3 crawl errors like "URL restricted by robots.txt". This is a mystery to me. I suspect only 2 of the pages are restricted by robots.txt. One of the pages require user to be authenticated and the other is a error page. Both of these are pages which I don't want indexed. The last page is a page with no tag or anything in robots.txt indicating it is restricted by robots.txt. Another why does google treat it as an error if I restrict googlebot access to a specific page. any suggestions why I get the URL restricted by robots.txt?

03-26-2010, 03:33 AM	#2
loginname HyperActive Warrior War Room Member Join Date: Oct 2009 Posts: 275 Thanks: 10 Thanked 4 Times in 4 Posts	Re: Problem: URL restricted by robots.txt My robots.txt is like this (a shorten version): User-agent: * Sitemap: http://www.mydomain.com/sitemap.xml Allow: / Disallow: /folder1/ Disallow: /folder2/ Disallow: /Error.htm