Remove Duplicate Content from Google's Index - Watch your Rankings Increase!

[strategic seo services]

You can improve your website's ranking in Google by eliminating your site's duplicate content from the Google index.

What counts as duplicate content and how do you know if it's showing in the index?
Well, let's say that you have a blog. You have "Post A", which is totally unique content. However, that "Post A" is also showing up on a "category" page of your site. That category page is placed in Google's supplemental index and is considered to be duplicate content by Google.

How do you check for this?

Do a search like this in Google: site:http://www.yoursite.com

Now, go to the last page of results, if you see a message like this:
"In order to show you the most relevant results, we have omitted some entries very similar to the 42 already displayed.
If you like, you can repeat the search with the omitted results included."

1. Click on "repeat the search with the omitted results included."
2. Now, find the pages that are repeats (eg: the Category Page with Post "A"). Get the URLs of those pages and submit content removal requests to Google through Webmasters Tools: Completely remove an entire page - Webmaster Tools Help
3. Now, the last step: put a disallow command in your robots.txt file so that URL is no longer crawled or indexed by Google.

URLs are usually removed from Google's index within 24 hours.

Use this strategy & witness a great improvement in your site's rankings! This strategy has helped me a great deal. Just thought I'd share it with you.

[gearmonkey]

The problem I am having with one of my guitar blogs is people are stealing the content as fast I put it up. I read an article over at SEOmoz that this is an ongoing problem that Google needs to address ASAP because a lot of people are affected by this in a negative way. Leecher splogs are outranking the original source in some cases.

This is a good read: Post-Panda, Your Original Content is Being Outranked by Scrapers & Partners | SEOmoz

Also read the comments.

Great advice, btw. If you are using Wordpress, I recommend using Robots Meta by Yoast plugin.

[strategic seo services]

Originally Posted by gearmonkey

The problem I am having with one of my guitar blogs is people are stealing the content as fast I put it up. I read an article over at SEOmoz that this is an ongoing problem that Google needs to address ASAP because a lot of people are affected by this in a negative way. Leecher splogs are outranking the original source in some cases.

This is a good read: Post-Panda, Your Original Content is Being Outranked by Scrapers & Partners | SEOmoz

Also read the comments.

Great advice, btw. If you are using Wordpress, I recommend using Robots Meta by Yoast plugin.

Thanks. I use Robots Meta. It works really well. Highly recommended.

[Icematikx]

Pro tip:

Develop a WordPress plugin that syndicates new posts through RSS feeds. Find new blogs that take 2-3 days to get indexed. Scrape those sources and post their articles to your domain without any references. By the time the original source gets indexed, their content is considered duplicate content, because the RSS aggregator indexed the content quicker.

It's a sly tactic, but it's only going to get fixed once it starts to have a hugely detrimental effect on webmasters.

[strategic seo services]

Originally Posted by Icematikx

Pro tip:

Develop a WordPress plugin that syndicates new posts through RSS feeds. Find new blogs that take 2-3 days to get indexed. Scrape those sources and post their articles to your domain without any references. By the time the original source gets indexed, their content is considered duplicate content, because the RSS aggregator indexed the content quicker.

It's a sly tactic, but it's only going to get fixed once it starts to have a hugely detrimental effect on webmasters.

That's a very interesting idea, but that type of plugin would definitely anger a lot of webmasters. I wonder if a plugin like that already exists??

[yukon]

Long term it's easier to create a better category.php template page inside your WP theme that doesn't add a full WP Post to the Category page. Instead of adding WP Post on the Category page, create a unique static description (per Category) then simply link out to the internal WP Post in the same category.

[related thread]
Any Good Silo Structure Plugins You Know Of?

No sense in constantly chasing down pages you don't want in the SERPs. There's also no reason to not rank a Category page, it's the mother of all pages related to the subject/category on your site.

[gearmonkey]

Originally Posted by yukon

Long term it's easier to create a better category.php template page inside your WP theme that doesn't add a full WP Post to the Category page. Instead of adding WP Post on the Category page, create a unique static description (per Category) then simply link out to the internal WP Post in the same category.

[related thread]
Any Good Silo Structure Plugins You Know Of?

No sense in constantly chasing down pages you don't want in the SERPs. There's also no reason to not rank a Category page, it's the mother of all pages related to the subject/category on your site.

Interesting point.

Do some people find that some wordpress themes rank better than others? For example Genesis vs heatmapthemes, both paid and premium themes.

[MikeFriedman]

So nobody has ever heard of a canonical tag around here?

[yukon]

Originally Posted by gearmonkey

The problem I am having with one of my guitar blogs is people are stealing the content as fast I put it up. I read an article over at SEOmoz that this is an ongoing problem that Google needs to address ASAP because a lot of people are affected by this in a negative way. Leecher splogs are outranking the original source in some cases.

This is a good read: Post-Panda, Your Original Content is Being Outranked by Scrapers & Partners | SEOmoz

Also read the comments.

Great advice, btw. If you are using Wordpress, I recommend using Robots Meta by Yoast plugin.

No offense to you gearmonkey, but that's exactly the kind of crap that Moz spews out that is totally ridiculous.

That article is dated April 20th, 2011 & Moz claims Panda is the reason scraped content pages out rank the original source pages content.

Scraped pages have been outranking original source pages since the beginning of Googles existence, this is far from being anything related to Panda & defiantly was started years before Panda or Moz ever existed.

There's no possible way Google can have their algo. determine which site created the original content, it's literally impossible. Google can't compare dates on who had the first crawl/index/cache because you might have some small site that doesn't care or know much about SEO that's creating awesome original content, then have SEO savvy scraper sites that know a lot about SEO & how to get their pages indexed ASAP (no real SERP competition).

Original sources of content means nothing to Googles algo., again it's literally impossible to figure out who created the original content. The only choice If anyone wants to rank pages is to have better SEO than the scraper sites (not optional).

[ucables]

Julia, thank you for this very interesting info!

do you know any tool to check repeated pages automatically?

when you have thouthands of pages indexed its not easy to check all duplicate pages indexed.

[strategic seo services]

Originally Posted by ucables

Julia, thank you for this very interesting info!

do you know any tool to check repeated pages automatically?

when you have thouthands of pages indexed its not easy to check all duplicate pages indexed.

You're welcome. Unfortunately I don't know of a tool that checks for repeated pages automatically. If I find one, I'll post it here.

[GGpaul]

Originally Posted by strategic seo services

You can improve your website's ranking in Google by eliminating your site's duplicate content from the Google index.

What counts as duplicate content and how do you know if it's showing in the index?
Well, let's say that you have a blog. You have "Post A", which is totally unique content. However, that "Post A" is also showing up on a "category" page of your site. That category page is placed in Google's supplemental index and is considered to be duplicate content by Google.

How do you check for this?

Do a search like this in Google: site:http://www.yoursite.com

Now, go to the last page of results, if you see a message like this:
"In order to show you the most relevant results, we have omitted some entries very similar to the 42 already displayed.
If you like, you can repeat the search with the omitted results included."

1. Click on "repeat the search with the omitted results included."
2. Now, find the pages that are repeats (eg: the Category Page with Post "A"). Get the URLs of those pages and submit content removal requests to Google through Webmasters Tools: Completely remove an entire page - Webmaster Tools Help
3. Now, the last step: put a disallow command in your robots.txt file so that URL is no longer crawled or indexed by Google.

URLs are usually removed from Google's index within 24 hours.

Use this strategy & witness a great improvement in your site's rankings! This strategy has helped me a great deal. Just thought I'd share it with you.

Is there an easier way to do this? This makes it difficult for me when I get 8,000+ results.

[strategic seo services]

I found this article that explains a process that seems a bit easier if you have a lot of pages on your site: How to Find Duplicate Content | SEOMention

It gives a lot of helpful tips too.

[gearmonkey]

Originally Posted by ucables

Julia, thank you for this very interesting info!

do you know any tool to check repeated pages automatically?

when you have thouthands of pages indexed its not easy to check all duplicate pages indexed.

If you're using wordpress, you can use this plugin
WordPress › Duplicate Post « WordPress Plugins

[satrap]

Originally Posted by gearmonkey

If you're using wordpress, you can use this plugin
WordPress › Duplicate Post « WordPress Plugins

This plugin doesn't help you find duplicate posts. It helps you create Clones of your already published posts!...

[pradeepkolanu]

Thanks a lot for the suggestion......

[gearmonkey]

Originally Posted by satrap

This plugin doesn't help you find duplicate posts. It helps you create Clones of your already published posts!...

I used one before that would mass find and mass delete duplicate posts. I just did a quick search and that popped up. Plugins are out there for those motivated enough to look.

[strategic seo services]

Originally Posted by gearmonkey

I used one before that would mass find and mass delete duplicate posts. I just did a quick search and that popped up. Plugins are out there for those motivated enough to look.

Has anyone come across the correct plugin yet?

[Rehmat]

Originally Posted by yukon

Long term it's easier to create a better category.php template page inside your WP theme that doesn't add a full WP Post to the Category page. Instead of adding WP Post on the Category page, create a unique static description (per Category) then simply link out to the internal WP Post in the same category.

Isn't it a better deal to prevent indexing of categories in WordPress using WordPress SEO or any other plugin? Lesser-experienced webmasters can't do it well what you have said.

[strategic seo services]

Originally Posted by Rehmat

Isn't it a better deal to prevent indexing of categories in WordPress using WordPress SEO or any other plugin?

Yes, prevention of duplicate content arriving in the Google index is a good first step. However, many sites already have duplicate content out there in the index, preventing their rankings from increasing to their full portential. So a plugin/program to detect the duplicate content in the index would really be a big help.

[paulgl]

Originally Posted by MikeFriedman

So nobody has ever heard of a canonical tag around here?

That would require working and learning.

Nobody has heard of much, when you think about it. How about
ditching WP and get either get a real CMS, your own coding, or
even blogspot?

In reality, duplicate content that causes any such penalties is
rare. Very rare. Threads like this just make people think the sky is
falling. This is 2013, people. 2013. Are you still next to
Rip Van Winkle?

I dare anyone to see how much "duplicate content" the wf,
amazon, wikipedia, NYT, etc. have.

Paul

[strategic seo services]

Originally Posted by paulgl

In reality, duplicate content that causes any such penalties is
rare. Very rare. Threads like this just make people think the sky is
falling. This is 2013, people. 2013. Are you still next to
Rip Van Winkle?

I dare anyone to see how much "duplicate content" the wf,
amazon, wikipedia, NYT, etc. have.

Paul

Really? It's funny that you cited the New York Times (NYT) in your "dare", as Matt Cutts has a video that addresses this very issue: Matt Cutts Addresses Duplicate Content Issue In New Video | WebProNews

You cannot compare authority sites like the NYT or Wikipedia to an "average" site on the web. Yes, we are in 2013, and the above article was written only 7 months ago.

My experience has shown me that by removing duplicate content from Google's index, a website's rankings will, in turn, increase within a 24 hour period.

[brettb]

I'm pretty sure that duplicate content sank my older sites. In one example I found one of my articles was stolen from my site but the stolen article had an older timestamp in Google!

Matt Cutts has no answer for this.

[AravGupta]

The better option to check with the duplicate onpage optimization is to configure your website with webmaster tools, click on HTML improvement section that comes under optimization section and then download the list of errors, fix them manually and resubmit the sitemap of your website. I have done this recently and benefiting a lot.

[GeorgR.]

That category page is placed in Google's supplemental index and is considered to be duplicate content by Google.

This is just a VERY odd way to go about this, let alone that I have doubts to believe it would impact your rankings significantly.

The CORRECT way to do it is right from the start, on your site. All archive pages, categories, tags, calendar pages etc..etc.. should ALWAYS only have excerpts. There should not be dupes on your site with the same content on your actual post AND category pages, tags or wherever else.

But even then, I doubt that Google is still so "dumb" and would rank down your site since there are dupes on your sites. This is pretty common and I cannot see this being a major ranking factor.

[yukon]

Originally Posted by Rehmat

Isn't it a better deal to prevent indexing of categories in WordPress using WordPress SEO or any other plugin? Lesser-experienced webmasters can't do it well what you have said.

This will create a unique Category page. If you don't know HTML/PHP find someone to make the changes, a decent web developer should be able to read that thread & have it finished in less than 1 hour for an average theme.

[strategic seo services]

Here are some great tools that will help you check for duplicate content:
12 Best Sites For Checking Duplicate Content

[successproducts]

The best way is to ping your article the minute you post it. I've heard this would put the stamp on YOUR article before anybody else.

[beseenontop]

Several people have suggested the best way to deal with this is prevention - configure your blog so it does not create the duplicate content pages in the first place.

Do you use Wordpress? This webinar from SEOMoz and @nickherinckx does a good job explaining how to prevent these duplicate content issues. Advanced WordPress SEO: Actionable Advice for Ensuring Your WordPress Content is Found - Webinar | SEOmoz. It was only published a few days ago.

This article - Guide to Bringing Your WordPress Blog Up to Today?s Google Algorithm Standards | Search Engine Journal - was on Search Engine Journal last month. It's an excellent how-to guide with helpful screen shots. It's not quite as comprehensive as the webinar though. For example, it does not speak to the php file modification, the one to ensure only excerpts get posted to archive pages. Nor does it suggest how to customize your category pages so they are unique.

The two together are gold.

[nik0]

Maybe I misunderstand something here but why no one has suggested to noindex the tags/category/author etc. pages?

Yukon's suggestion to create a unique category page is miles better though and excerpts is also a very good alternative, I think most WP themes are already setup that it creates excerpts isn't it?

[Joe Karl]

Originally Posted by strategic seo services

You can improve your website's ranking in Google by eliminating your site's duplicate content from the Google index.

What counts as duplicate content and how do you know if it's showing in the index?
Well, let's say that you have a blog. You have "Post A", which is totally unique content. However, that "Post A" is also showing up on a "category" page of your site. That category page is placed in Google's supplemental index and is considered to be duplicate content by Google.

How do you check for this?

Do a search like this in Google: site:http://www.yoursite.com

Now, go to the last page of results, if you see a message like this:
"In order to show you the most relevant results, we have omitted some entries very similar to the 42 already displayed.
If you like, you can repeat the search with the omitted results included."

1. Click on "repeat the search with the omitted results included."
2. Now, find the pages that are repeats (eg: the Category Page with Post "A"). Get the URLs of those pages and submit content removal requests to Google through Webmasters Tools: Completely remove an entire page - Webmaster Tools Help
3. Now, the last step: put a disallow command in your robots.txt file so that URL is no longer crawled or indexed by Google.

URLs are usually removed from Google's index within 24 hours.

Use this strategy & witness a great improvement in your site's rankings! This strategy has helped me a great deal. Just thought I'd share it with you.

HI there,

I was going to order a package of you and I stumbled across your duplicate content post and was wondering if you can help.


Now, find the pages that are repeats (eg: the Category Page with Post "A"). Get the URLs of those pages and submit content removal requests to Google through Webmasters Tools: Completely remove an entire page - Webmaster Tools Help

I cant find no pages that are repeats ? ok i must be doing something right them....

[strategic seo services]

Originally Posted by Joe Karl

HI there,

I was going to order a package of you and I stumbled across your duplicate content post and was wondering if you can help.


Now, find the pages that are repeats (eg: the Category Page with Post "A"). Get the URLs of those pages and submit content removal requests to Google through Webmasters Tools: Completely remove an entire page - Webmaster Tools Help

I cant find no pages that are repeats ? ok i must be doing something right them....

Hi Joe,
Yes, if you don't see pages that are duplicates in Google, then you should be fine.

[ProSence]

Originally Posted by strategic seo services

Yes, prevention of duplicate content arriving in the Google index is a good first step. However, many sites already have duplicate content out there in the index, preventing their rankings from increasing to their full portential. So a plugin/program to detect the duplicate content in the index would really be a big help.

Can you please suggest any WP plugin to remove these duplicate contents?

[gearmonkey]

Originally Posted by nik0

Maybe I misunderstand something here but why no one has suggested to noindex the tags/category/author etc. pages?

Yukon's suggestion to create a unique category page is miles better though and excerpts is also a very good alternative, I think most WP themes are already setup that it creates excerpts isn't it?

I really love this plugin - Meta Robots WordPress plugin WordPress plugin

[Paul Tovey]

Thanks for this, I was wondering how to do this for ages!

[seoed]

My experience has shown me that by removing duplicate content from Google's index, a website's rankings will, in turn, increase within a 24 hour period.

how much did your rankings or traffic increase?

[paulgl]

Originally Posted by strategic seo services

You cannot compare authority sites like the NYT or Wikipedia to an "average" site on the web. Yes, we are in 2013, and the above article was written only 7 months ago.

My experience has shown me that by removing duplicate content from Google's index, a website's rankings will, in turn, increase within a 24 hour period.

That's just bogus. Man, you must have had a ton of sites and tons of dupe
content. That's NOT the "average joe."

Maybe your business was junk to begin with. Dump the trash, then see.
That's different than cause and effect about "duplicate content."

As far as "average joe sites" and authoritative sites, that's exactly
how this "average joe" wants to do things. Exactly like the big boys.
If amazon, NYT, ebay, etc. do not give a rat's hat about dupe content,
neither will I.

It's about junk, not duplicate content.

Just more blind leading the blind down a rabbit hole.

Paul

[yukon]

Originally Posted by paulgl

That's just bogus. Man, you must have had a ton of sites and tons of dupe
content. That's NOT the "average joe."

Maybe your business was junk to begin with. Dump the trash, then see.
That's different than cause and effect about "duplicate content."

As far as "average joe sites" and authoritative sites, that's exactly
how this "average joe" wants to do things. Exactly like the big boys.
If amazon, NYT, ebay, etc. do not give a rat's hat about dupe content,
neither will I.

It's about junk, not duplicate content.

Just more blind leading the blind down a rabbit hole.

Paul

That's why so many struggle with SEO, they think big sites do magical things. Sites like Wikipedia don't do anything any other site can't do. The problem is some people are too lazy to dig in & research why big site pages rank. Sure they have authority but the big sites started out with a single web page just like everyone else. Wikipedia for example has awesome on-page SEO (internal links + relevant pages).

[strategic seo services]

Originally Posted by seoed

how much did your rankings or traffic increase?

Some of my keywords went from page 10 or 15 back to page 1 or 2.

[smodha]

Pro tip: - if you are worried about content from your sites being stolen then head over to Tynt. Signup and they will give a script that you can run on your wordpress/blogger/tumblr blog etc. As soon as somebody "copy & pastes" content from you, your site will be rewarded with a backlink from the their site.

This is a paid service but the free account will give you access to a 30 day analystics report about your site. It's nice to earn some backlinks on auto-pilot and in my tests it's pretty effective.

Note:- I am not affiliated to tynt.com in anyway. There are no affiliate links in this post.

[strategic seo services]

Originally Posted by smodha

Pro tip: - if you are worried about content from your sites being stolen then head over to Tynt. Signup and they will give a script that you can run on your wordpress/blogger/tumblr bog etc. As soon as somebody "copy & pastes" content from you, your site will be rewarded with a backlink from the their site.

This is a paid service but the free account will give you access to a 30 day analystics report about your site. It's nice to earn some backlinks on auto-pilot and in my tests it's pretty effective.

Note:- I am not affiliated to tynt.com in anyway. There are no affiliate links in this post.

Interesting...this will also help you keep track of who is attempting to steal your content.

[rossnmia]

Definitely true - I had all my wordpress tag pages showing as duplicate pages. Took those off and viola rank improved

[rossnmia]

By the way google webmaster tools is great for telling you whats duplicate - getting what they consider duplicate pages from the horses mouth so to speak.

Also duplicate content can be pages with little content e.g. category pages with two posts excerpts on them and lots of surrounding sidebar content and duplicate or no meta tags for those pages = big slap from google in my experience

[IMSince2003]

Sigh. Do you really think that Google doesn't know what WordPress is and how it works? There are a zillion sites that use WP and I can say fairly certainly that Google is not penalizing them just because they use a standard WP installation.