OMG WTF!?!? DE-INDEXED!!!

[GuerrillaIM]

I thought I would try to clear up a few things that I see people getting confused with.

De-indexed means that your site was removed from the google index. If your site does not show up for its main keyword, it DOES NOT mean your site was de-indexed. Even if you search for your domain name and your site does not come up, this also does not mean you have been de-indexed.

To see if your site is in the google index you search for this "site:yourdomain.com" making sure not to put a space after the ":". If that does not bring back any results, then you actually have been "de-indexed", assuming you were indexed to start with.

Sent to hell is one of the popular terms used is SEO community to describe when a site still exists in the index, but dissapears down the rankings to a page where no one will ever see you. This is usually due to google disregarding a large portion of your links or applying a ranking penalty of some sort.

I have done several experiements trying purposefully to get a site de-indexed, just to see how far you have to push it before it happens. Contrary to popular belief it is pretty hard to get a site de-indexed. I certainly think the chances of it happening by accident without someone purposefully doing something dodgy to effect their rankings are extremely remote.

The 3 surefire ways to get your site de-indexed I have found are:

1. Multiple subdomains on same domain with automatically generated content. BlueFartters have been doing this trick for years. They get a domain then keep spawning autoblog subdomains off it with generated content and then interlink the sub domains. I also did an experiement with addidng reciprocal links directories to sub domains and had the reciprocal link point to my main domain. This also got me de-indexed.

2. Cloaked content. People use rubbish autogenerated content to display to search engine spiders, but display CPA offers to humans. This will get you de-indexed for sure.

3. Interlinked network over multiple domains. This one is harder to get de-indexed for. Generally if you interlink loads of unrelated domains in an obvious pattern, leaving obvious footprints then google can manually remove the network from the index if they deem it to be spam.

What usually happens with interlinked networks when people start screaming "OMG ALL SITEZ R DEINDEXED", is that they were ranking due to the fact of all the interlinking between sites they had done. The google algo then detected the obvious pattern of interlinking and chose to disregard the links. This of course reduced their ranking power and sent them to hell. The sites still exist in the index, they just dont rank as their incoming links have been disregarded.

I have been doing SEO for a long time and personally test everything before I comment on it. I have done all of these techniques just as an experiement to see what happens. I have never had a site de-indexed by accident in 8 years of optimizing sites.

If your site does go to hell it doesn't mean you should panic. They usually come back given time, some new content and some quality incoming links.

If you are going to base all your income in settining up SEO networks then you should create multiple networks, multiple accounts, mask your domain whois and hide obvious fingerprints. This way you wont run the risk of loosing your whole network in one swoop.

Getting de-indexed is rare, even for interlinked domains. Google usually just dis-regards the links so the sites don't rank well. Google generally looks to de-index automatically generated content or cloaked pages. As long as you don't put out rubbish pages or use BlueFart scripts you should be ok.

I could put 100 pages on 100 domains and interlink them. As long as all of those pages were good information then Google has no reason to de-index them, although if I interlinked them all it would leave an obvious fingerprint and it would just be a matter of time before those links were disregarded.

Does your data show different? I only have my own data to base this on and the people I work withs data so if you have some evidence to the contrary then it would be good to hear from you as I am in process of setting up several networks at the moment.

[Andy Fletcher]

Awesome post. I've added this to a required reading list in the FAQ.

Keep up the good work.

[sterlingtek]

For your cloaking experiment what kind of cloaking did you try? There are ways of course to make high quality autoblogs, (rather then rubbish). There are from what I can see a ton of ways to cloak or redirect traffic based on, incoming keywords, page landing on, etc.. Supposedly there are also huge DBs of user agents etc. for the cloaking game to try to outfox the Search Engines that are updated daily. I have never used them but I am interested in redirecting based on incoming keywords with a couple of sites that are not producing, any thoughts?

[sezerb]

Great post. Really informative. Many times you see people claiming their site is gone from the index just because they dropped ranking for a main keyword.

[GuerrillaIM]

Originally Posted by Andy Fletcher

Awesome post. I've added this to a required reading list in the FAQ.

Keep up the good work.

No problem, happy to help.

Originally Posted by sterlingtek

For your cloaking experiment what kind of cloaking did you try? There are ways of course to make high quality autoblogs, (rather then rubbish). There are from what I can see a ton of ways to cloak or redirect traffic based on, incoming keywords, page landing on, etc.. Supposedly there are also huge DBs of user agents etc. for the cloaking game to try to outfox the Search Engines that are updated daily. I have never used them but I am interested in redirecting based on incoming keywords with a couple of sites that are not producing, any thoughts?

I used a script that scraped and auto generated content but showed a squeeze page to human visitors. The script auto updated itself with a list of search engine spiders but it still got detected eventually.

I know some guys who are serious autobloggers, they setup thousands of sites every day and use cloaking to push traffic to CPA offers. For them it is a numbers game. They know a domain will get burned, they just automate the process and make a bit of cash before it does. If you plan to make money cloaking then you will have to do it on a fairly large scale in my opinion as the sites wont last.

[udeze]

Quite informative, I just logged on to my Webmasters tools and discovered that sitemap for 3 of my sites are showing 0, one of them dropped from 700 indexed pages to zero, however the indexed pages are still showing on google, this article has now clarified everything, it appears other people are having the same issues

[Fraggler]

I just found this about the sitemap issue - http://webmaster-forum-announcements...url-count.html

It is just a display bug!

That was a great post by the way, GuerrillaIM. There are a lot of misunderstandings about Google that can be easily explained with some common sense and research.

[NiyazK]

Its good u made a topic regarding this. If anyone complains about their site being deindexed this topic will serve them better

[vinild]

Some really valuable information. I had a few squidoo lenses I created about 6 six months ago (was just trying out) and the irony was I hadn't even put a link to my site. Now I realize that I need to put a link to all sites even though I no longer use them or they are outdated.

[Fernando Veloso]

Some of my sites were sent to hell and never came back.

Guess the place must be damn good!!

[GuerrillaIM]

Originally Posted by Fernando Veloso

Some of my sites were sent to hell and never came back.

Guess the place must be damn good!!

If your site has good content and inbound links, and should be ranking somewhere but is buried deep for all keywords, then you should check the outgoing links.

Linking to a bad neighbourhood can get your site sent to hell. Also having badly formatted html or errors on the site will do the same.

Usually removing offending links and tidying up all the code will bring them back eventually.

[James Ross]

Wow, Thanks Guerrilla for a great writeup. I'm glad my main sites have come back but I'm still waiting in purgatory on others.

[Boris_yo]

@GuerrillaIM

I have question for you. I have a XFactor website - 5 pages of content + Privacy Policy, About Us, Contact Us and Sitemap.

It's been 4 months and only 2 pages are indexed, main page and 2nd page of content. I tried to re-bookmarking website to no avail, i tried pinging it to no avail.

Apparently there is something that Google doesn't like.

Do you want to take a look at the website and tell me what you think about it? If yes i will PM you website.

Thanks.

[GuerrillaIM]

Sure, PM away.

[CET]

Nice post Guerrilla!

I'm currently working with a network of 17 sites and a couple of them are "in hell". I've been working on this for a while, and I don't have any answers for my client.

They are all driven off of a single CMS and use the same HTML template, but they all have their own domain and unique content. I've come up with several hypothesis for the penalty, but none of them pan out, because we expect to see any penalty on one of them to be across all of them.

Why would only a couple of sites in a network rank far differently when they're all the same, except in actual content?

[GuerrillaIM]

Originally Posted by CET

Why would only a couple of sites in a network rank far differently when they're all the same, except in actual content?

There are lots of things that effect ranking. Competition of keywords, incoming links etc...

I don't know what sort of answer you are looking for. You are probably better off starting a new thread and giving more detailed information of the setup of your network.

[James Sol Radina]

Very helpful. Thank you. I am learning each day more and more SEO. I appreciate this!

[steveyb]

This has to be some of the most informative information I have read in this forum thanks.

[FreshPLR]

If you just focus on creating content and build backlinks in a steady, natural way, traffic will grow and in time you will make money. There seems to be too much focus on monetising today, instead of creating value for the searcher first, and then the dollars will follow tomorrow.

[claycath]

Ok, what do you do if your site actually has been de-indexed? And you have no idea why. It's not an auto blog, not interlinking to my other sites, etc.

Is there any way to get google to re-index it?

I am adding content to beef it up and will add some backlinks too. But not understanding why it was de-indexed in the first place, it makes it hard to know how to fix it.

[GuerrillaIM]

Originally Posted by claycath

Ok, what do you do if your site actually has been de-indexed? And you have no idea why. It's not an auto blog, not interlinking to my other sites, etc.

Is there any way to get google to re-index it?

I am adding content to beef it up and will add some backlinks too. But not understanding why it was de-indexed in the first place, it makes it hard to know how to fix it.

Add the site to google webmaster tools. This will help diagnose the errors.

You can request re-inclusion from the WMT control panel, although its a good idea to find out why it was removed first.

I have seen google removing sites more agressively than usual that have badly formatted HTML recently. This is simple to fix if it is the case.

Add it to WMT then wait a while for data to appear in there (takes a while usually). If you get stuck feel free to PM me.

[claycath]

Thanks for the quick reply GuerrillaIM. Actually it was already in webmaster tools and I submitted my sitemap too months ago. It says no data available for pretty much everything.

Can I PM you the URL and see if anything jumps out to you as being grounds for the deindexing?

[GuerrillaIM]

Originally Posted by claycath

Thanks for the quick reply GuerrillaIM. Actually it was already in webmaster tools and I submitted my sitemap too months ago. It says no data available for pretty much everything.

Can I PM you the URL and see if anything jumps out to you as being grounds for the deindexing?

Sure, feel free.

[LisaJordan]

I checked my site that had been on page 1 since March and sure enough when I type in site:www....com it's gone! Oh boy I guess I have some work to do. How do you find out why if there is such a thing.

[DNChamp]

I have a domain that is deindexed. Asked google 2x to reconsider and they have not. I have not touched the page so it does get a few type in traffic. Now with that domain is it a lost cause if I completly rebuild it? Should I just let it expire out?

[calfred]

Yeah.

But if you were using AdSense, Google can still track every websites you have, right?

[yukon]

Originally Posted by calfred

Yeah.

But if you were using AdSense, Google can still track every websites you have, right?

Well yea, Adsense, Analytics, Webmaster Tools. They all provide code snippets for tracking sites.

[GuerrillaIM]

Originally Posted by LisaJordan

I checked my site that had been on page 1 since March and sure enough when I type in site:www....com it's gone! Oh boy I guess I have some work to do. How do you find out why if there is such a thing.

The points I made in the original post are the most common causes I have found. In addition to them, autogenerated text content (markov) and badly formatted HTML or slow loading times can be to blame.

Log into Google webmaster tools and see if there are any error messages and also see when you last crawl date was.

[lroufs]

hope I never get de-indexed. Just do it fare!

[socialbookmark]

Originally Posted by DNChamp

I have a domain that is deindexed. Asked google 2x to reconsider and they have not. I have not touched the page so it does get a few type in traffic. Now with that domain is it a lost cause if I completly rebuild it? Should I just let it expire out?

Unfortunately when Google penalize a website, the domain will be penalized and changing the subject of website or template of it and like this changes, can not help you too much. And it takes time that Google index and rank it again. I suggest you to make only some quality backlinks for your website and wait that Google index and rank it again.

[michielnl]

if your domain (for example "best-tv.com") is de-indexed, to my experience the domain is de-indexed, but the content can be re-used easily.

this is what i do to minimize loss:
- get a new domain with small change in name (for example "my-best-tv.com"), 301 redirect old domain to new domain (to get max of link juice to old domain)
- let new domain serve old pages ( by transfering content to new domain or set .htaccess file on new domain to get content from old domain on backside )

[michielnl]

another way to get your site de-indexed I have found is:

get a crappy looking site ranking top 2 for a competitive term and wait for a human review member of google checks your site :-)

google actually does do human reviews (a instruction guide for that leaked a couple of years ago)

i always try to make sure that the main page of site should bring a very clear value to visitors (for example by publishing independent review list, etc etc) and that a first look at your site should not raise questions about user value (so make sure the layout is simple en user driven, for example by showing the top 5 best buy options etc)

[markowe]

Originally Posted by michielnl

another way to get your site de-indexed I have found is:

get a crappy looking site ranking top 2 for a competitive term and wait for a human review member of google checks your site :-)

google actually does do human reviews (a instruction guide for that leaked a couple of years ago)

i always try to make sure that the main page of site should bring a very clear value to visitors (for example by publishing independent review list, etc etc) and that a first look at your site should not raise questions about user value (so make sure the layout is simple en user driven, for example by showing the top 5 best buy options etc)

Yeah, I suspect I have had a human review put paid to sites of mine (which were pretty poor, it has to be said!) I just wonder what they think of all those Xfactor-type sites out there like KitchenCabinetDoorHinges.com or whatever. There are still gazillions of those out there, doing fine AFAICS.

I do interlink my sites too much, I do wonder sometimes if it's a good idea. Also, running a lot of sites off the same shared hosting. But then there are like 700 sites on the same server, I figure safety in numbers, ha ha!

[dvduval]

And overall the ones that get de-indexed are all about SEO but they just don't exude "quality". If you are going to do something, why not put something online that is worthwhile?

[flx89]

all the best tips in just one post , great

[marjames]

i knew it!!! why my site get de-indexed! from 1st spot to nowhere... its not in there (site:xxx.com) but then I started it without knowing about de-indexing causes. I put a lot of effort, unique content, backlinking.. but because of subdomains auto-generated content, google slapped me off!!

[John Williamson]

Yes, thank you, hopefully a lot of people will read this, because there definitely seems to be much confusion about slapped vs. de-indexed.

[GuerrillaIM]

Originally Posted by marjames

i knew it!!! why my site get de-indexed! from 1st spot to nowhere... its not in there (site:xxx.com) but then I started it without knowing about de-indexing causes. I put a lot of effort, unique content, backlinking.. but because of subdomains auto-generated content, google slapped me off!!

Yes, I discovered this through trial and error because for a while some years back (5 or 6) i was setting up multiple link directories on subdomains and getting really good results then all of a sudden those sites got de-indexed.

After that I read a BH guide showing how to gain traffic bursts by generating hundreds of subdomains with autogenerated content.

Some BH guys I spoke to were literally burning through hundreds of domains each week with this tactic.

By deleting the subdomains I managed to get the sites in question re-indexed without any problem.

[JSProjects]

Great post. Definitely helps clear up some of the confusion around being de-indexed / slapped / the Google dance, etc.

[ukbuddy2]

This is a massively helpful post
thankyou

I have had all my sites with hostgator deindexed
it seems that maybe some of them failed the litmus test so google slapped them all in the same hosting company

i have no idea now how to get them reindexed other than clear out any duplicate content and remove any kind of scraping tool.
I had a shop plugin on a few of the sites


anyway do you offer a service to help people with de0-indexing ?
if not maybe a niche role there

also do you know of any tools to use that will help find if site has been deindexed
or
how to get reindexed

cheers

uk

Originally Posted by GuerrillaIM

I thought I would try to clear up a few things that I see people getting confused with.

De-indexed means that your site was removed from the google index. If your site does not show up for its main keyword, it DOES NOT mean your site was de-indexed. Even if you search for your domain name and your site does not come up, this also does not mean you have been de-indexed.

To see if your site is in the google index you search for this "site:yourdomain.com" making sure not to put a space after the ":". If that does not bring back any results, then you actually have been "de-indexed", assuming you were indexed to start with.

Sent to hell is one of the popular terms used is SEO community to describe when a site still exists in the index, but dissapears down the rankings to a page where no one will ever see you. This is usually due to google disregarding a large portion of your links or applying a ranking penalty of some sort.

I have done several experiements trying purposefully to get a site de-indexed, just to see how far you have to push it before it happens. Contrary to popular belief it is pretty hard to get a site de-indexed. I certainly think the chances of it happening by accident without someone purposefully doing something dodgy to effect their rankings are extremely remote.

The 3 surefire ways to get your site de-indexed I have found are:

1. Multiple subdomains on same domain with automatically generated content. BlueFartters have been doing this trick for years. They get a domain then keep spawning autoblog subdomains off it with generated content and then interlink the sub domains. I also did an experiement with addidng reciprocal links directories to sub domains and had the reciprocal link point to my main domain. This also got me de-indexed.

2. Cloaked content. People use rubbish autogenerated content to display to search engine spiders, but display CPA offers to humans. This will get you de-indexed for sure.

3. Interlinked network over multiple domains. This one is harder to get de-indexed for. Generally if you interlink loads of unrelated domains in an obvious pattern, leaving obvious footprints then google can manually remove the network from the index if they deem it to be spam.

What usually happens with interlinked networks when people start screaming "OMG ALL SITEZ R DEINDEXED", is that they were ranking due to the fact of all the interlinking between sites they had done. The google algo then detected the obvious pattern of interlinking and chose to disregard the links. This of course reduced their ranking power and sent them to hell. The sites still exist in the index, they just dont rank as their incoming links have been disregarded.

I have been doing SEO for a long time and personally test everything before I comment on it. I have done all of these techniques just as an experiement to see what happens. I have never had a site de-indexed by accident in 8 years of optimizing sites.

If your site does go to hell it doesn't mean you should panic. They usually come back given time, some new content and some quality incoming links.

If you are going to base all your income in settining up SEO networks then you should create multiple networks, multiple accounts, mask your domain whois and hide obvious fingerprints. This way you wont run the risk of loosing your whole network in one swoop.

Getting de-indexed is rare, even for interlinked domains. Google usually just dis-regards the links so the sites don't rank well. Google generally looks to de-index automatically generated content or cloaked pages. As long as you don't put out rubbish pages or use BlueFart scripts you should be ok.

I could put 100 pages on 100 domains and interlink them. As long as all of those pages were good information then Google has no reason to de-index them, although if I interlinked them all it would leave an obvious fingerprint and it would just be a matter of time before those links were disregarded.

Does your data show different? I only have my own data to base this on and the people I work withs data so if you have some evidence to the contrary then it would be good to hear from you as I am in process of setting up several networks at the moment.

[Fernando Veloso]

Originally Posted by GuerrillaIM

If your site has good content and inbound links, and should be ranking somewhere but is buried deep for all keywords, then you should check the outgoing links.

Linking to a bad neighbourhood can get your site sent to hell. Also having badly formatted html or errors on the site will do the same.

Usually removing offending links and tidying up all the code will bring them back eventually.

Huh?

I am talking about the last "I kill all webmasters" move by Google that nuked thousands of sites across the web, including 100% whitehat sites.

These are NOT the usual greyish sites that get penalized - these are completly normal sites with unique relevant content, etc. It seems they got penalized simply cause they shared... ADSENSE.

Thats relevant info I don't see discussed.

Sadly.

[ukbuddy2]

so does that mean not to put adsense on all your sites using the same account
open multiple adsense accounts ??
please enlighten

[Fernando Veloso]

Originally Posted by yukon

Well yea, Adsense, Analytics, Webmaster Tools. They all provide code snippets for tracking sites.

Last "killing" a couple weeks ago was made using Google own codes ^^. They have used these codes and the ability to know what sites belong to who, and just nuked them ALL.

Many people got perfectly GOOD and AUTHORITY sites penalized simply cause they shared adsense, analytics or webmasters tools information.

Thats why webmasters will never use Google scripts/products again.

Never again.

....

And sites BUYING LINKS in public... keep #1!!!

I mean, sites that follow G Webmasters Guidelines get penalized for something we can't understand, completely irrational...

But sites buying links publicly get #1 rankings for their good keywords.

So yeah, unless Google solve this **** and start penalizing the right sites, I prefer to send my data and $ to Facebook.

Google no more.

[GuerrillaIM]

Originally Posted by Fernando Veloso

Huh?

I am talking about the last "I kill all webmasters" move by Google that nuked thousands of sites across the web, including 100% whitehat sites.

These are NOT the usual greyish sites that get penalized - these are completly normal sites with unique relevant content, etc. It seems they got penalized simply cause they shared... ADSENSE.

Thats relevant info I don't see discussed.

Sadly.

The majority of discussions on this topic are very emotional so I steer clear of them as no one really wants to look at the facts objectively.

If you have an example de-indexed URL (or several would be even better) then lets look at it and see what we can do.

I have an adsense network myself (with fairly low quality sites) and from time to time a site drops out of the index for "no reason".

My own theory on this is due to Googles housekeeping. Google actually index less pages than other search engines. I believe they have a system inplace for clipping junk sites out of their index and this routine has some collateral damage to genuine sites that have grown a little stale or perhaps on some base level share similarities with junk sites.

In all cases I have had my site re-indexed within 3 months by adding some new unique content and linking to it from homepage. Sometimes a domain gets re-indexed almost instantly, sometimes Im waiting months.

In the last 10 years I have only ever had 1 domain that I couldn't get re-indexed (even with a re-inclusion request) but this domain had a very shady past before my client bought it.

[GuerrillaIM]

Originally Posted by ukbuddy2

This is a massively helpful post
thankyou

I have had all my sites with hostgator deindexed
it seems that maybe some of them failed the litmus test so google slapped them all in the same hosting company

i have no idea now how to get them reindexed other than clear out any duplicate content and remove any kind of scraping tool.
I had a shop plugin on a few of the sites


anyway do you offer a service to help people with de0-indexing ?
if not maybe a niche role there

also do you know of any tools to use that will help find if site has been deindexed
or
how to get reindexed

cheers

uk

Were all domains on the same IP and name servers?

[DNChamp]

Question.....I do have Host gator baby package....have about 50 sites on the same nameservers......Are you saying this could be a red flag AND if so will I need to buy a complete seperate hosting package to do new sites on?

[Born2rule]

Thanks GuerrillaIM, for this interesting post, I had some of my adsense site listed in the same adsense account de-index recently. Never list all your site on that adsense account, e.g. "url channel".
You mention something about sub-domain, I wander if that is why my site was de-index, I don't know why it happen but somehow all the site that got deindex was index along with the primary/root domain. (hostgator baby share host acc). e.g "rootdomain.com/subdomain1", I check with seo quake, I discover that the primary domain has 19 pages index, when I only have about "6 pages" on the primary domain, further investigation reveal that the sub-domain were being index along with the primary domain. I wonder if this is normal?
I will try your re-index tips, I will add some post and quality back-link, hope that will reinstate my site!
I can't pm you insufficient post count,please I would like you to have a close look at the index pages?

[36burrows]

Great post my friend and will definitely clear up a LOT of the confusion on this forum.

[36burrows]

Originally Posted by DNChamp

Question.....I do have Host gator baby package....have about 50 sites on the same nameservers......Are you saying this could be a red flag AND if so will I need to buy a complete seperate hosting package to do new sites on?

NO, that is not a problem.

There would only be a problem if you decided to interlink all of the sites together in order to manipulate search results.

If none of them are linked together, you can have as many sites as you want.

[GuerrillaIM]

Originally Posted by DNChamp

Question.....I do have Host gator baby package....have about 50 sites on the same nameservers......Are you saying this could be a red flag AND if so will I need to buy a complete seperate hosting package to do new sites on?

No I am not saying that its bad to have sites in same package, this is normal to have multiple sites but if you do something google doesnt like with multiple sites that share a common footprint you may have the entire network of sites removed.

I have hosting accounts with 50+ sites in and h ave no problems but if I want to create hundreds of automatically generated sites and interlink them then you need to do a little bit of work to cloak it.

Sometimes you might even have problems with hackers exploiting your sites and thats why all your sites get removed.

When multiple sites are removed the ffirst step is to identify the common factors between thebsites to try and understand what is going.