Freelancer.com

Go Back   Warrior Forum - The #1 Internet Marketing Forum & Marketplace > The Warrior Forum > Search Engine Optimization
Register Blogs Social Groups Advertise with usHelp Desk

Closed Thread
LinkBack Thread Tools
Unread 9th May 2012, 06:12 AM   #1
HyperActive Warrior
 
Join Date: 2011
Posts: 175
Thanks: 5
Thanked 4 Times in 4 Posts
Default What is Crawl-Delay ?

Hi warriors,

Can you please define crawl delay. I have seen Crawl-delay: 10 in robots.txt file, What does it mean??

Thanks!
SamDenial is offline  
Unread 10th May 2012, 01:07 AM   #2
Warrior Member
 
Join Date: 2012
Posts: 14
Thanks: 0
Thanked 0 Times in 0 Posts
Default Re: What is Crawl-Delay ?

Crawl-delay is nothing but denotes the number of seconds to wait between two successive request to a same website.

Steve Michael is offline  
Unread 23rd May 2012, 03:32 AM   #3
HyperActive Warrior
 
Join Date: 2011
Posts: 175
Thanks: 5
Thanked 4 Times in 4 Posts
Default Re: What is Crawl-Delay ?

Thanks a lot aegisisc and Steve. I appreciate it.
SamDenial is offline  
Unread 23rd May 2012, 06:15 AM   #4
Banned
 
Join Date: 2012
Posts: 49
Thanks: 0
Thanked 0 Times in 0 Posts
Default Re: What is Crawl-Delay ?

Hello,

I want to know if there are any advantages in modifying the crawl delay value?

Thanks in advance!
landen is offline  
Unread 23rd May 2012, 06:45 AM   #5
Warrior Member
 
JohnMCarter's Avatar
 
Join Date: 2012
Posts: 16
Thanks: 0
Thanked 1 Time in 1 Post
Post Re: What is Crawl-Delay ?

Quote:
Originally Posted by SamDenial View Post
Hi warriors,

Can you please define crawl delay. I have seen Crawl-delay: 10 in robots.txt file, What does it mean??

Thanks!

Crawl-delay is the directive used to by major websites who has very frequent content updation. In the case of Twitter; Google bot and other major bots love to visit Twitter since all they are providing real-time updates in their SERP. Since frequent visit by all bots will cause over-load to their servers, Twitter setting a time-interval for each crawling by using crawl-delay directive.

Hire PHP Developer A Team of Consistent Developers for Your PHP Needs
JohnMCarter is offline  
Closed Thread

  Warrior Forum - The #1 Internet Marketing Forum & Marketplace > The Warrior Forum > Search Engine Optimization

Bookmarks

Tags
crawldelay

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off




All times are GMT -6. The time now is 07:44 AM.