robots.txt - What should it look like for wordpress site?
I am in the process of putting together a site that is using wordpress and I am wondering what a good robots.txt should look like. After some googling and combinig it with what I use on other sites of mine I have this so far:
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
User-agent: Googlebot
Disallow: /wp-content/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /feed/
Disallow: /archives/
Disallow: /index.php
Disallow: /*?
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: */feed/
Disallow: */trackback/
Disallow: /page/
Disallow: /tag/
Disallow: /category/
User-agent: Googlebot-Image
Disallow: /wp-includes/
User-agent: Mediapartners-Google*
Disallow:
User-agent: ia_archiver
Disallow: /
User-agent: duggmirror
Disallow: /
So, any one wants to add something or suggest I delete something?
Also, if I want to have a ebook that people can download, how should I go about it to somewhat protect it (it's free but I do want people to sign up to a list to get it) and keep google from indexing it?
So far I've made a folder in the main directory, put a Disallow: /foldername in robots.txt as well as put an empty index.html in the folder. Anything else I can/should do to keep search engines from indexing it?
Thank you for your feedback!
-
sober -
Thanks
{{ DiscussionBoard.errors[1266273].message }} -
-
TheRichJerksNet -
Thanks
{{ DiscussionBoard.errors[1266304].message }} -
-
KristiDaniels -
Thanks - 1 reply
{{ DiscussionBoard.errors[1266323].message }}-
Bruce Hearder -
Thanks
{{ DiscussionBoard.errors[1267164].message }} -
-