2 replies
I have a bunch of affiliate sites. When google indexes my sites, it shows multiple pages, because it shows the main one, and then the same one with category. It also sometimes shows my affiliate sites, even though I use a link cloaker. It will also index my admin panel, and several other things I don't want indexed.

I never had robots files on my sites. They are all wordpress blogs. I came up with a file, and want an opinion before I put it on my sites.

Also is it best to put it in the public_html of the domain, or is the wordpress plugin for a robots.txt file just as good. I also noticed that in the sitemap plugin, you can click an option to have it built in the root directory, and then edit it.

Which way is best?

Here is the robots.txt file I am thinking of using. Thanks Glenn
User-agent: *
Disallow: /feeds/
Disallow: /feeds
Disallow: /auction.php
Disallow: /auctions.php
Disallow: /get.php
Disallow: /wp-content/
Disallow: /wp-content
Disallow: /wp-content/plugins/
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /trackback
Disallow: /comments
Disallow: */trackback
Disallow: /category/*/*
Disallow: */comments
Disallow: /*?*
Disallow: /*?
#robotstxt
  • Profile picture of the author Karen Blundell
    your robots.txt file should be in your public_html directory of your domain

    the sitemap plugin puts it in the correct location
    Signature
    ---------------
    {{ DiscussionBoard.errors[1549262].message }}
  • Profile picture of the author customertools
    You can add a no index meta tag to the cloaked pages... VS messing up your website with a very restrictive robots.txt file.

    samples:
    <META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">
    <META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">
    <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

    There are a few wp plugins that allow you to add custom meta tags
    {{ DiscussionBoard.errors[1549892].message }}

Trending Topics