3 {{ upvoteCount | shortNum }}
3 {{ upvoteCount | shortNum }}

How to slow the crawl rate of yahoo?

by billmat
2 replies
Hello members,

How do I slow the crawl rate of yahoo?
its killing my bandwidth 700 bots a day and putting extra strain on my server.

Please tell me How to do this?
#crawl #rate #slow #yahoo
  • Profile picture of the author GMD
    Banned
    Originally Posted by billmat View Post

    Hello members,

    How do I slow the crawl rate of yahoo?
    its killing my bandwidth 700 bots a day and putting extra strain on my server.

    Please tell me How to do this?
    Well, you can't "slow" down anything. It's not like dimming the lights. You're either stuck with the crawl rate/frequency or you can block Yahoo's crawlers from your site (while still allowing other crawlers such as "Google"):

    Just change your robots.txt area:

    User-agent: Yahoo!-AdCrawlerDisallow: /
    {{ DiscussionBoard.errors[5383624].message }}
  • Profile picture of the author Webpromotion
    I would not do anything.
    What you have is not really a problem.

    You want the crawler at your site.

    People pay other people to get the crawler at their site.
    {{ DiscussionBoard.errors[5383819].message }}

Trending Topics