What is Crawl-Delay ?

4 replies
  • SEO
  • |
Hi warriors,

Can you please define crawl delay. I have seen Crawl-delay: 10 in robots.txt file, What does it mean??

Thanks!
#crawldelay
  • Profile picture of the author Steve Michael
    Crawl-delay is nothing but denotes the number of seconds to wait between two successive request to a same website.
    {{ DiscussionBoard.errors[6217268].message }}
  • Profile picture of the author aegisisc
    Banned
    [DELETED]
    {{ DiscussionBoard.errors[6217702].message }}
    • Profile picture of the author SamDenial
      Thanks a lot aegisisc and Steve. I appreciate it.
      {{ DiscussionBoard.errors[6296721].message }}
  • Profile picture of the author landen
    Banned
    Hello,

    I want to know if there are any advantages in modifying the crawl delay value?

    Thanks in advance!
    {{ DiscussionBoard.errors[6297418].message }}
  • Profile picture of the author JohnMCarter
    Originally Posted by SamDenial View Post

    Hi warriors,

    Can you please define crawl delay. I have seen Crawl-delay: 10 in robots.txt file, What does it mean??

    Thanks!

    Crawl-delay is the directive used to by major websites who has very frequent content updation. In the case of Twitter; Google bot and other major bots love to visit Twitter since all they are providing real-time updates in their SERP. Since frequent visit by all bots will cause over-load to their servers, Twitter setting a time-interval for each crawling by using crawl-delay directive.
    {{ DiscussionBoard.errors[6297581].message }}

Trending Topics