What is Crawl-Delay ?

by SamDenial Posted: 12 years ago 4 replies

Hi warriors,

Can you please define crawl delay. I have seen Crawl-delay: 10 in robots.txt file, What does it mean??

Thanks!

Steve Michael 12 years ago

Crawl-delay is nothing but denotes the number of seconds to wait between two successive request to a same website.
aegisisc Banned 12 years ago

[DELETED]
- [1] reply
- SamDenial 12 years ago
  
  Thanks a lot aegisisc and Steve. I appreciate it.
perfectionnk Banned 12 years ago

[DELETED]
landen Banned 12 years ago

Hello,

I want to know if there are any advantages in modifying the crawl delay value?

Thanks in advance!
katherineeobryan Banned 12 years ago

[DELETED]
JohnMCarter 12 years ago

Crawl-delay is the directive used to by major websites who has very frequent content updation. In the case of Twitter; Google bot and other major bots love to visit Twitter since all they are providing real-time updates in their SERP. Since frequent visit by all bots will cause over-load to their servers, Twitter setting a time-interval for each crawling by using crawl-delay directive.
perfectionh Banned 12 years ago

[DELETED]

Next Topics on Trending Feed