What is Crawler?

by AndersonP Posted: 14 years ago 2 replies

A crawler is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "spider" or a "bot." Crawlers are typically programmed to visit sites that have been submitted by their owners as new or updated. Entire sites or specific pages can be selectively visited and indexed. Crawlers apparently gained the name because they crawl through a site a page at a time, following the links to other pages on the site until all pages have been read.

#search engine optimization #crawler

socialbookmark 14 years ago
Yes. There is a good article explains it well here:
Code:
```
en.wikipedia.org/wiki/Web_crawler
```
Warriorforum has many professional users that are familiar with these primitive information. So its better to write new things that users learn more.
Raman11 Banned 14 years ago

[DELETED]
csophie Banned 14 years ago

[DELETED]
jackjohns 14 years ago

A web crawler is a way for the search engines and other users to consistently ensure that their databases are up-to-date. Also,

Join the discussion

Next Topics on Trending Feed

3

What is Crawler?

AndersonP • 14 years ago • 2 replies

A crawler is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "spider" or a "bot." Crawlers are typically programmed to visit sites that have been submitted by their owners as new or updated. Entire sites or specific pages can be selectively visited and indexed. Crawlers apparently gained the name because they crawl through a site a page at a time, following the links to other pages on the site until all pages have been read.