Help! Bing/msn bot using huge load of my server

4 replies
  • SEO
  • |
I am facing a big problem with my server. I have a website that keeps getting massive page requests coming from "Bing/Msn" bot every second or two and the ip changes now and then. Which is putting a heavy load on my server.

My CPU is constantly over 90%

I tried to block the bot from htaccess and robots.txt but they don't seem to have any effect.

If anyone has an idea how to defeat this it would be much appreciated.

Thank you in advance

#bing or msn #bot #huge #load #server
Avatar of Unregistered
  • Profile picture of the author dave_hermansen
    Blocking your website from search engines is a pretty drastic move unless you don't care if it is ever found.

    Sure seems like those are not normal bot errors unless you have a ton of URLs that no longer exist. If so, you need to 301 them to the correct pages. If there are no relevant pages to 301 them to, remove them from Bing.

    The following link shows you how to do that -
    Signature Learn How to Dropship the Right Way - Buy & Sell Websites - Partner with Coach
    My PROVEN ecommerce process, as seen on: Fox News, the NY Times & Flippa
    {{ DiscussionBoard.errors[11553738].message }}
    • Profile picture of the author kurosaki4d
      Thank you for your help, that's good to know, i'm sure it might come up handy someday !

      For now what i did is i reduced the crawl rate of Bing and Google to the absolute minimum from "Search console" and i added the crawl rate also in the robots.txt file, plus i added wordfence to block any malicious bots that pass on as google and to block it if it exceeds a certain amount of requests within 1 minute.

      That gave me a great result and relieved the extra pressure from the server.

      Thank you !
      {{ DiscussionBoard.errors[11555602].message }}
  • Profile picture of the author savidge4
    The IP set to would be the University of California ( San Diego )

    Knowing that I can for certain say that it is NOT a Bing / MSN bot. To further add to not just me saying this is so ( )

    I would block the entire set for a period of time and call it a day.
    Success is an ACT not an idea
    {{ DiscussionBoard.errors[11555652].message }}
  • Profile picture of the author sendizo
    Do you happen to have large files in your server?
    It's possible that your robots.txt file's directives are not being honored because of a formatting problem, Maybe!?

    Last, You could adjust Crwaler Speed from Bing Webmaster Tool Panel,
    Free Link & Click Tracking Software - Increase Profit by 33%.
    Free link tracking, Sales, Action & Engagement Tracking, CPA marketing, url rotator, A/B testing.
    {{ DiscussionBoard.errors[11555738].message }}
Avatar of Unregistered

Trending Topics