What is Crawl-Delay ?

by SamDenial 4 replies
Hi warriors,

Can you please define crawl delay. I have seen Crawl-delay: 10 in robots.txt file, What does it mean??

Thanks!
#search engine optimization #crawldelay
  • Profile picture of the author Steve Michael
    Crawl-delay is nothing but denotes the number of seconds to wait between two successive request to a same website.
    {{ DiscussionBoard.errors[6217268].message }}
  • Profile picture of the author aegisisc
    Banned
    [DELETED]
    {{ DiscussionBoard.errors[6217702].message }}
    • Profile picture of the author SamDenial
      Thanks a lot aegisisc and Steve. I appreciate it.
      {{ DiscussionBoard.errors[6296721].message }}
  • Profile picture of the author landen
    Banned
    Hello,

    I want to know if there are any advantages in modifying the crawl delay value?

    Thanks in advance!
    {{ DiscussionBoard.errors[6297418].message }}
  • Profile picture of the author JohnMCarter
    Originally Posted by SamDenial View Post

    Hi warriors,

    Can you please define crawl delay. I have seen Crawl-delay: 10 in robots.txt file, What does it mean??

    Thanks!

    Crawl-delay is the directive used to by major websites who has very frequent content updation. In the case of Twitter; Google bot and other major bots love to visit Twitter since all they are providing real-time updates in their SERP. Since frequent visit by all bots will cause over-load to their servers, Twitter setting a time-interval for each crawling by using crawl-delay directive.
    {{ DiscussionBoard.errors[6297581].message }}

Trending Topics

  • 93 {{ upvoteCount | shortNum }}

    Tons of Bad Advice In This Thread

    Most people that come to the forum are looking for advice on how to build their business or to otherwise make more money. Thanks to all those willing to help ... [read more]

  • 2 {{ upvoteCount | shortNum }}

    Mailchimp - Some contacts are missing?

    awesthei in Email Marketing

    Mailchimp - I recently imported a small .csv list (about 10 contacts) into my Mailchimp Contacts (about 900 contacts). These new contacts will not show up when I do a ... [read more]

  • 1 {{ upvoteCount | shortNum }}

    What is better sports affiliate marketing programs or networks for promoting live sports streaming ?

    Perfume Toronto in Ad Networks

    Please let me help if you know what will be better sports affiliate program or networks for promoting live sports streaming online ? I am using ad-center and edigitalplace. But ... [read more]

  • 28 {{ upvoteCount | shortNum }}

    Internet Marketing VS Network Marketing

    noobz in Internet Marketing

    Last night I was invited to a small meeting and this guy who is in this one Network Marketing company happens to be the speaker. He claimed to make 6 ... [read more]

  • 28 {{ upvoteCount | shortNum }}

    How i promote my business in particular city

    Nikhil Bansal in SEO

    I want to know how I promote my business in a particular city like Texas in America, what step I follow.