5 {{ upvoteCount | shortNum }}
5 {{ upvoteCount | shortNum }}

Robots, Wordpress and Duplicate Content

by Beldin
4 replies
  • SEO
    • |
Hi gang,

Wordpress and all its subdirectories are new to me.... Is there a standard robots.txt people use when they use wordpress so that only the pages and posts are indexed?

So far I've got:

User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /cgi-bin/
But think I should also have

Disallow: /tag/
Disallow: /category/
Disallow: /author/
Disallow: /admin/

And something for archives... /archives/? Maybe?

Any suggestions?

Thanks,
Stephen
#content #duplicate #robots #wordpress
  • Profile picture of the author Intrepreneur
    Let everything be indexed even your contact page, privacy policy etc.

    Having too many restrictions can affect your site performance.
    Signature
    SEO, copywriting and backlinks service UK
    {{ DiscussionBoard.errors[1442993].message }}
    • Profile picture of the author Cristian Lungu
      I should restrain SE bot crawl pages like terms of service and privacy policy and about us, since this dilutes the link juice passed between your important web pages. Just a suggestion...

      If you're new to wordpress just use All in One SEO pack plugin and your SEO will be taken care of; as your duplicate content issue.
      {{ DiscussionBoard.errors[1447012].message }}
  • Profile picture of the author tony-raymondo
    Originally Posted by Beldin View Post

    Hi gang,

    Wordpress and all its subdirectories are new to me.... Is there a standard robots.txt people use when they use wordpress so that only the pages and posts are indexed?

    So far I've got:

    User-agent: *
    Disallow: /wp-
    Disallow: /feed/
    Disallow: /trackback/
    Disallow: /cgi-bin/
    But think I should also have

    Disallow: /tag/
    Disallow: /category/
    Disallow: /author/
    Disallow: /admin/

    And something for archives... /archives/? Maybe?

    Any suggestions?

    Thanks,
    Stephen
    why would you want to?

    lots of those are important
    {{ DiscussionBoard.errors[1447190].message }}
    • Profile picture of the author Beldin
      I have no reason too... just searching the web and found some suggestions... so I came here to verify!
      {{ DiscussionBoard.errors[1451835].message }}

Trending Topics