| | ||||||||
| |||||||||
 |
10-07-2011, 02:26 PM
| #1 |
| HyperActive Warrior Join Date: Dec 2008
Posts: 124
Thanks: 0
Thanked 4 Times in 4 Posts
|  Can anyone tell me the proper use of Robots.txt File?
I was searching robots.txt of anonymous websites. I found "disallow: signatures" over there in a website. Can anyone tell me what is this ? Is this is something for not crawling forum signature's Link ?
|
|  |
 |
|
| Tags |
| file, proper, robotstxt |
| Thread Tools | |
| |
 |
