Typical Time from Re-Crawl to Re-Index

consultant1027 · 07-28-2011, 01:08 PM

For existing sites, already in the Google index, on average how long are people seeing Google take before recrawling and discovering new pages or page changes, and when the changes are reflected in the index (you can see the Title or Meta Description change or see the page preview image change, or, see new pages appear)

Are people seeing new pages for existing sites take longer to show up than changes to existing pages?

How dependent is it on the size and age of the site?

~kev~ · 07-28-2011, 01:37 PM

Quote:

Originally Posted by consultant1027

How dependent is it on the size and age of the site?

A lot.

Google will crawl older sites with lots of backlinks faster then new sites. There was a show on the discovery channel a few months ago where an engineer from google was talking about how search engines work. Sites like yahoo are being crawled thousands of times a day to find new content.

On my blog, new articles are indexed within seconds of being posted.

New sites might only be crawled a few times a week.

consultant1027 · 07-28-2011, 01:57 PM

I've noticed that too.. blogs and especially forum posts showing up on Google almost instantly. I'd love to get more specific insight on what factors they use specifically to determine how quickly the page gets indexed and their relative importance - but wouldn't we all?

JamesGw · 07-28-2011, 02:03 PM

It has a lot to do with backlinks, as that's one way spiders can reach your site. I think Google traffic also has a lot to do with it, which kind of makes it difficult to increase your indexing rate. The more pages you have indexed, the more Google crawls them, but that doesn't help you when you're waiting to have the pages indexed. That said, having a crapload of pages definitely helps.

consultant1027 · 08-02-2011, 11:20 AM

It's interesting, I've got a site that is 7 years old with 2700+ backlinks. I'm seeing new pages appear in the index in 3-5 days. I've seeing updates to Title and Meta Description appear within 12 hours. I'm seeing the page preview being updated in a week or two, longer for sub-pages, shorter for home page.

It seems like the re-indexing takes even longer but it may just be there wasn't enough changes to the page to make it change position.

Why would a page not be re-indexed immediately after it is crawled?

lexilexi · 08-02-2011, 05:30 PM

Blog pages are often crawled immediately because the blog software (i.e. Wordpress) has a built in software that pings Google to let it know that the blog has been updated. You can also see how long it takes on average by looking at the "cached" link in the serps, it has a date on it and looking through any top 10 serp there are often pages which have not been crawled for 2-6 weeks.

07-28-2011, 01:08 PM	#1
consultant1027 Active Warrior Join Date: Oct 2009 Posts: 77 Thanks: 0 Thanked 0 Times in 0 Posts	Typical Time from Re-Crawl to Re-Index For existing sites, already in the Google index, on average how long are people seeing Google take before recrawling and discovering new pages or page changes, and when the changes are reflected in the index (you can see the Title or Meta Description change or see the page preview image change, or, see new pages appear) Are people seeing new pages for existing sites take longer to show up than changes to existing pages? How dependent is it on the size and age of the site?

07-28-2011, 01:57 PM	#3
consultant1027 Active Warrior Join Date: Oct 2009 Posts: 77 Thanks: 0 Thanked 0 Times in 0 Posts	Re: Typical Time from Re-Crawl to Re-Index I've noticed that too.. blogs and especially forum posts showing up on Google almost instantly. I'd love to get more specific insight on what factors they use specifically to determine how quickly the page gets indexed and their relative importance - but wouldn't we all?

07-28-2011, 02:03 PM	#4
JamesGw Senior Warrior Member Join Date: Apr 2011 Posts: 1,337 Thanks: 73 Thanked 132 Times in 121 Posts	Re: Typical Time from Re-Crawl to Re-Index It has a lot to do with backlinks, as that's one way spiders can reach your site. I think Google traffic also has a lot to do with it, which kind of makes it difficult to increase your indexing rate. The more pages you have indexed, the more Google crawls them, but that doesn't help you when you're waiting to have the pages indexed. That said, having a crapload of pages definitely helps.
	Adult Wholesale Direct \| Adult Home Party Kits

08-02-2011, 11:20 AM	#5
consultant1027 Active Warrior Join Date: Oct 2009 Posts: 77 Thanks: 0 Thanked 0 Times in 0 Posts	Re: Typical Time from Re-Crawl to Re-Index It's interesting, I've got a site that is 7 years old with 2700+ backlinks. I'm seeing new pages appear in the index in 3-5 days. I've seeing updates to Title and Meta Description appear within 12 hours. I'm seeing the page preview being updated in a week or two, longer for sub-pages, shorter for home page. It seems like the re-indexing takes even longer but it may just be there wasn't enough changes to the page to make it change position. Why would a page not be re-indexed immediately after it is crawled?

08-02-2011, 05:30 PM	#6
lexilexi HyperActive Warrior War Room Member Join Date: Nov 2007 Location: The Pod Posts: 329 Thanks: 129 Thanked 124 Times in 45 Posts	Re: Typical Time from Re-Crawl to Re-Index Blog pages are often crawled immediately because the blog software (i.e. Wordpress) has a built in software that pings Google to let it know that the blog has been updated. You can also see how long it takes on average by looking at the "cached" link in the serps, it has a date on it and looking through any top 10 serp there are often pages which have not been crawled for 2-6 weeks.
	The Link Building Black Book "Fear nothing in life but to waste the present moment"