Should I use robots.txt to ban meta analysers ? if so how ?
- SEO |
Hi, I tried analysing a site the other day and the analyser returned an error something like forbidden by robots.txt.
It seems a good idea to do this to stop people prying into your site , but does anyone know how to do it ?
(I couldn't see anything in the pages source code)
Also is it a good idea, or could it ban search engines too ?
how would it know its a meta analyser spidering the site ?
It seems a good idea to do this to stop people prying into your site , but does anyone know how to do it ?
(I couldn't see anything in the pages source code)
Also is it a good idea, or could it ban search engines too ?
how would it know its a meta analyser spidering the site ?
-
agiletechnosys11 -
Thanks
{{ DiscussionBoard.errors[3935393].message }} -