What is Crawl-Delay ?

by 4 replies
Hi warriors,

Can you please define crawl delay. I have seen Crawl-delay: 10 in robots.txt file, What does it mean??

Thanks!
#search engine optimization #crawldelay
  • Profile picture of the author Steve Michael
    Crawl-delay is nothing but denotes the number of seconds to wait between two successive request to a same website.
  • Profile picture of the author aegisisc
    Banned
    [DELETED]
  • Profile picture of the author perfectionnk
    Banned
    [DELETED]
  • Profile picture of the author landen
    Banned
    Hello,

    I want to know if there are any advantages in modifying the crawl delay value?

    Thanks in advance!
  • Profile picture of the author katherineeobryan
    Banned
    [DELETED]
  • Profile picture of the author JohnMCarter
    Originally Posted by SamDenial View Post

    Hi warriors,

    Can you please define crawl delay. I have seen Crawl-delay: 10 in robots.txt file, What does it mean??

    Thanks!

    Crawl-delay is the directive used to by major websites who has very frequent content updation. In the case of Twitter; Google bot and other major bots love to visit Twitter since all they are providing real-time updates in their SERP. Since frequent visit by all bots will cause over-load to their servers, Twitter setting a time-interval for each crawling by using crawl-delay directive.
  • Profile picture of the author perfectionh
    Banned
    [DELETED]

Trending Topics