2 {{ upvoteCount | shortNum }}
2 {{ upvoteCount | shortNum }}

Robots.txt to block search engines?

by debra
1 replies
  • SEO
    • |
I was trying to write a robots.txt file to block certain search engines that are bombarding one of my sites. Mainly the ones that the store doesn't sell too. Like FR and CN...etc. I wanted only US visitors. It's an amazon affiliate site.

There are Soooo Manyyyyy of them.

So I thought that I could use the disable or disallow in the robots.txt file for the friendly engines and then create a blackhole for the bots/crawlers/spiders that don't want to give respect to the robots file.

Does anyone have a bot/crawler/spider list of the majors and are not dead?

Another thing...has anyone included disallowing google.fr, google.com.es and had any adverse reactions to google.com indexing and ranking?

Google bots (not the one specific to the US, CA and UK) are indexing my images in foreign countries and now my bounce rate is over 85 percent. I Swear Sometimes I think google is just a retard and I can't figure out why ppl play up to them.
#block #engines #robotstxt #search
  • Profile picture of the author Terry Kyle
    Originally Posted by debra View Post

    I was trying to write a robots.txt file to block certain search engines that are bombarding one of my sites. Mainly the ones that the store doesn't sell too. Like FR and CN...etc. I wanted only US visitors. It's an amazon affiliate site.

    There are Soooo Manyyyyy of them.

    So I thought that I could use the disable or disallow in the robots.txt file for the friendly engines and then create a blackhole for the bots/crawlers/spiders that don't want to give respect to the robots file.

    Does anyone have a bot/crawler/spider list of the majors and are not dead?

    Another thing...has anyone included disallowing google.fr, google.com.es and had any adverse reactions to google.com indexing and ranking?

    Google bots (not the one specific to the US, CA and UK) are indexing my images in foreign countries and now my bounce rate is over 85 percent. I Swear Sometimes I think google is just a retard and I can't figure out why ppl play up to them.
    Hi Debra,

    This older guide from Danny Sullivan could help:

    Google Offers Robots.txt Generator

    I don't know if you can block Googlebot by specific country as I suspect it's universal and from there Google figures out which sites to show in which country's search results by hosting location, domain extension, language etc:


    Google Image traffic is a PITA and for most niches doesn't convert of course. I plan to try this solution and report back:

    How do I block google images? - Yahoo! Answers
    Signature
    {{ DiscussionBoard.errors[6939050].message }}

Trending Topics