Need advice for robots.txt file

1 replies
I have a bunch of affiliate sites. When google indexes my sites, it shows multiple pages, because it shows the main one, and then the same one with category. It also sometimes shows my affiliate sites, even though I use a link cloaker. It will also index my admin panel, and several other things I don't want indexed.

I never had robots files on my sites. They are all wordpress blogs. I came up with a file, and want an opinion before I put it on my sites.

Also is it best to put it in the public_html of the domain, or is the wordpress plugin for a robots.txt file just as good. I also noticed that in the sitemap plugin, you can click an option to have it built in the root directory, and then edit it.

Which way is best?

Here is the robots.txt file I am thinking of using. Thanks Glenn
User-agent: *
Disallow: /feeds/
Disallow: /feeds
Disallow: /auction.php
Disallow: /auctions.php
Disallow: /get.php
Disallow: /wp-content/
Disallow: /wp-content
Disallow: /wp-content/plugins/
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /trackback
Disallow: /comments
Disallow: */trackback
Disallow: /category/*/*
Disallow: */comments
Disallow: /*?*
Disallow: /*?
#advice #file #robotstxt
  • Profile picture of the author Manuel Viloria
    Hi! If your domain name is example.com, then
    please put your robots.txt file in the root. For example...

    example.com/robots.txt

    What kinds of duplicate pages do you want to exclude
    from Google? If it's all right with you, please post
    examples of the URLs of such duplicate pages.

    Thanks,
    Manuel Viloria
    Signature
    >> Get More Traffic Now www.ManuelViloria.com <<
    {{ DiscussionBoard.errors[1550626].message }}

Trending Topics