| | ||||||||
| |||||||||
 |
10-04-2011, 03:44 PM
| #1 |
| HyperActive Warrior War Room Member Join Date: May 2009 Location: Vancouver, BC
Posts: 271
Thanks: 2
Thanked 6 Times in 6 Posts
|  Which folders should I disallow in Robots.txt?
I'm wondering which pages/folders of a site should I disallow using the robots.txt? I saw on someone else's site they have this: User-agent: * Disallow: /cgi-bin/ Disallow: /go/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /author/ Disallow: /page/ Disallow: /category/ Disallow: /wp-images/ Disallow: /images/ Disallow: /backup/ Disallow: /banners/ Disallow: /archives/ Disallow: /trackback/ Disallow: /feed/ User-agent: Googlebot-Image Allow: /wp-content/uploads/ User-agent: Mediapartners-Google Allow: / User-agent: duggmirror Disallow: / |
| | |
|  |
 |
|
| Tags |
| disallow, folders, robotstxt |
| Thread Tools | |
| |
 |
