4 {{ upvoteCount | shortNum }}
4 {{ upvoteCount | shortNum }}

Is your host blocking SEMrush?

by Jeffery
3 replies
  • SEO
    • |
Many people on this forum use SEMrush. It seems the SEMrush bot is blocked or blacklisted by some hosts.

Most of the hosts are owned by EIG aka Endurance International Group - about 84 at time of this writing. To be certain it is necessary to contact your host support in the event of changes.

Two popular hosts and hosting packages recommended on this forum and others owned by EIG are BlueHost and HostGator - Shared Hosting. To see if your host is owned by EIG you can reference this article EIG list (not affiliated).

You can read more about it on Reddit (not affiliated):
FYI SEMrush bot is blacklisted by Bluehost server

A SEMrush representative confirmed it and basically says the SEMrush bot is blocked by the web host on shared hosting packages and they have tried to contact EIG, but EIG will not respond:
FeelTheRush11, Sam from SEMrush
Hi, Sam from SEMrush here! Thanks for your question! First of all we feel sorry for this negative experience you've faced! However, currently, we are experiencing a block of our crawlers from Bluehost hosting provider. We believe that such actions from Bluehost are not justified. We are constantly trying to open a conversation with their leadership but so far all our attempts to establish productive conversation with them have been ignored. I am afraid the common whitelisting of our bot doesn't work on their side for shared servers. The only solution at the moment is to switch to VPS / Dedicated Hosting and whitelist our bot. I would like to assure you that this is a top priority for us but we can't give you any specific timeframes as we solely depend on Bluehost in this matter. We hope for your understanding!

Referring to your question, if our crawler slows down your website and you do not have a crawl delay directive in your robots.txt file, you can tell SEMrush to crawl 1 URL per 2 seconds. This may force your audit to take longer to complete, but it will cause less potential speed issues for actual users on your website during the audit.
Some reddit users say they have contacted their respective host and verified the block owed to heavy server usage. Personally, sounds fair enough to me. It should be noted that SEMrush does in fact explain how to un-block the bot with the robots.txt file and there are a few WordPress plugins designed to do the same, however when a host blocks a bot (or anything for that fact) that will override the robots.txt file, so No Joy.

There are many a thread on this forum wherein some SEO Agencies recommend SEMrush to their clients. There are also many SEO Agencies that do "not" recommend SEMrush. There are also many a member that recommend SEMrush. There are also many a member that do "not" recommend SEMrush.

In my experience the main reason some (not all) SEO Agencies recommend SEMrush is because it gives the SEO Agency a reference to a metric that the customer is lead to believe as proof of the SEO Agency's competence.

Now that it is verified by some web host support staff (according to some reddit members) the SEMrush bot is blocked/blacklisted will SEO Agencies stop recommending SEMrush?

If SEO Agencies stop using SEMrush what are the alternitives and will the alternatives eventually be blocked by web hosts?
#blacklisted #blocking #bot #host #semrush
Avatar of Unregistered
  • Profile picture of the author freelancernational
    Banned
    [DELETED]
    {{ DiscussionBoard.errors[11619132].message }}
  • Profile picture of the author dsimms
    I would imagine from a host standpoint, it would make sense to block such tools, it does nothing for the host other than to consume BW, in other words, someone is spying on a site, and the host pays for it or someone has too, and I suspect a lot of similar tools are simply used to spy on the competition and to be honest, there is really nothing in it for the host other than it consumes data and resources.
    {{ DiscussionBoard.errors[11619140].message }}
    • Profile picture of the author Jeffery
      Originally Posted by dsimms View Post

      I would imagine from a host standpoint, it would make sense to block such tools, it does nothing for the host other than to consume BW, in other words, someone is spying on a site, and the host pays for it or someone has too, and I suspect a lot of similar tools are simply used to spy on the competition and to be honest, there is really nothing in it for the host other than it consumes data and resources.
      In this case the website owner is not spying on their own site because the website owner authorized the MOZ Bot.

      The take away is more about the host blocking MOZ bot(s).
      Signature
      In the minute it took me to write this post.. someone died of Covid 19. RIP.
      {{ DiscussionBoard.errors[11619142].message }}
Avatar of Unregistered

Trending Topics