Google.co.uk Robots.txt Exclusion?

clsupply10 · 11-14-2011, 03:22 AM

Hi,
I have a problem with duplicate content on a clients website as they have over 40 versions of the site on the same domain for different countries around the world. The problem is that the content has not been localised so there is actually 40 english versions of the site on the same domain.

What I am trying to do is to tell the google bots to only crawl the UK verson of the site so that 1 section is indexed. I realise this is possible with a simply change to the robots.txt file. however i would actually only like this to happen in Google.co.uk.

Is it possible to include exclusions in the robots.txt file for specific countries?

Any help would be greatly apprecaited.

Chris

11-14-2011, 03:22 AM	#1
clsupply10 clsupply10 Join Date: Nov 2011 Posts: 2 Thanks: 0 Thanked 0 Times in 0 Posts	Google.co.uk Robots.txt Exclusion? Hi, I have a problem with duplicate content on a clients website as they have over 40 versions of the site on the same domain for different countries around the world. The problem is that the content has not been localised so there is actually 40 english versions of the site on the same domain. What I am trying to do is to tell the google bots to only crawl the UK verson of the site so that 1 section is indexed. I realise this is possible with a simply change to the robots.txt file. however i would actually only like this to happen in Google.co.uk. Is it possible to include exclusions in the robots.txt file for specific countries? Any help would be greatly apprecaited. Chris