Crawling process

alexbert · 07-21-2011, 06:26 AM

By using software a website is crawled called a spider in the case of Google. During this the crawler will looks for links to different pages from that website and look at the text on a website. The spider can crawl webpages more then one before they gets indexed and the spider will start from URLs found on previous indexed pages. A web crawler does not visit every page daily as this would be not needed when some pages have been never change and the crawler visit pages with frequent changes.

07-21-2011, 06:26 AM	#1
alexbert Warrior Member Join Date: Jul 2011 Posts: 18 Thanks: 0 Thanked 1 Time in 1 Post	Crawling process By using software a website is crawled called a spider in the case of Google. During this the crawler will looks for links to different pages from that website and look at the text on a website. The spider can crawl webpages more then one before they gets indexed and the spider will start from URLs found on previous indexed pages. A web crawler does not visit every page daily as this would be not needed when some pages have been never change and the crawler visit pages with frequent changes.
	Alex Bert SEO Company