Should I use robots.txt to ban meta analysers ? if so how ?

cctvinstallers · 05-23-2011, 04:54 AM

Hi, I tried analysing a site the other day and the analyser returned an error something like forbidden by robots.txt.

It seems a good idea to do this to stop people prying into your site , but does anyone know how to do it ?
(I couldn't see anything in the pages source code)

Also is it a good idea, or could it ban search engines too ?
how would it know its a meta analyser spidering the site ?

agiletechnosys11 · 05-23-2011, 06:43 AM

Robot . txt is a small code in a web page which is mostly used to stop or continue crawling your page by search engine spider. You can use Dofollow to follow your page or Nofollow to skip it.

Thanks..!!

05-23-2011, 04:54 AM	#1
cctvinstallers HyperActive Warrior Join Date: May 2011 Location: london, uk Posts: 106 Thanks: 14 Thanked 1 Time in 1 Post Social Networking	Should I use robots.txt to ban meta analysers ? if so how ? Hi, I tried analysing a site the other day and the analyser returned an error something like forbidden by robots.txt. It seems a good idea to do this to stop people prying into your site , but does anyone know how to do it ? (I couldn't see anything in the pages source code) Also is it a good idea, or could it ban search engines too ? how would it know its a meta analyser spidering the site ?
	My site

05-23-2011, 06:43 AM	#2
agiletechnosys11 Warrior Member Join Date: May 2011 Posts: 14 Thanks: 0 Thanked 0 Times in 0 Posts	Re: Should I use robots.txt to ban meta analysers ? if so how ? Robot . txt is a small code in a web page which is mostly used to stop or continue crawling your page by search engine spider. You can use Dofollow to follow your page or Nofollow to skip it. Thanks..!!
	php web development company