Robots.txt file For SEO on Wordpress

5 replies
  • SEO
  • |
Hello, i'm optimizing my new Wordpress site and i'm considering on adding a robots.txt file with the kb robots plugin, i was wondering if any of you can recommend a standard robots.txt file that i should use for seo.

Thanks
#file #robotstxt #seo #wordpress
  • Profile picture of the author CyborgX
    That’s a great question. One of my blog I’ve a static robots.txt, so I’ve set up a dynamic example using code snippets from other sites: this robots.txt is what a user sees, and here is what various get on request of my robots.txt example.

    Of course crawlers don’t request a robots.txt file with a query string identifying themselves.
    {{ DiscussionBoard.errors[6360857].message }}
    • Profile picture of the author paulgl
      If you can decipher that previous reply, you're
      a better man then I am.

      I have no idea what that's about.

      You really don't need one, or just create a blank
      one.

      All search bots will look for a robots.txt first and
      follow any directions. Maybe.

      There is really no reason to block any pages from
      getting indexed. Unless you have a really good
      reason. 99.999% of sites have no reason to hide
      anything. Google is pretty smart as to what to show
      in SERPS. Just because it's indexed in no way, shape,
      or form does it mean it will show for anything.

      One thing that should be blocked from indexing is search
      results.

      Google has a built in one for blogspot:

      User-agent: *
      Disallow: /search

      I would assume wordpress is very similar.

      If someone does a search at the very instant the googlebot
      crawls your page, there is a chance that the search result
      page will be indexed.

      Hardly anything for concern, however. It's just that search
      results are useless pages, and, if lots of people do searches,
      it could result in thousands of these things getting indexed.
      But how many of your visitors do searches, and do them at the
      same time the bot comes? Very low odds.

      Paul
      Signature

      If you were disappointed in your results today, lower your standards tomorrow.

      {{ DiscussionBoard.errors[6361034].message }}
  • Profile picture of the author karga
    i have problem robot.txt on my wordpress site, can help me anyone please?
    {{ DiscussionBoard.errors[6521583].message }}
  • Profile picture of the author PatrickLSO
    Here is a basic one. Covers most stuff you would want.



    User-agent: *

    Disallow: /feed/
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-content/
    Disallow: /wp-includes/
    Disallow: /trackback/
    Disallow: /xmlrpc.php
    Disallow: ?wptheme=
    Disallow: /blackhole/
    Disallow: /transfer/
    Disallow: /tweets/
    Disallow: /mint/

    Allow: /tag/mint/
    Allow: /tag/feed/
    Allow: /wp-content/online/

    Sitemap: yoursitemapURL

    User-agent: ia_archiver
    Disallow: /
    {{ DiscussionBoard.errors[6521800].message }}
    • Profile picture of the author tjp33
      Shouldn't we also block our affiliate links folder in which case we do our redirects from?
      {{ DiscussionBoard.errors[6795440].message }}

Trending Topics