A Message from Simon Grabowski CEO of GetResponse -

by 116 replies
161
Dear GetResponse Customer,

As you may know, GetResponse has had a pretty rough weekend… We suffered a major outage caused by a DDoS (distributed denial of service) attack on an unprecedented scale, which has flooded our network and our data center (Windstream) with malicious traffic.



On Saturday, after hours of working together with our data center Windstream (including their directors and VP-level executives), we received information that they have failed to mitigate the attack and of their decision to block our network connectivity until a powerful DDoS mitigation service can be fully implemented.

When GetResponse went down, the morale of the GetResponse community didn’t. We saw an amazing display of support and we would like to thank our customers for rooting for us.



We brought many of the services back online on Sunday (websites, web subscriptions, API, web forms, surveys, landing pages). We had another attack on Monday, but we fully mitigated it, with no downtime, thanks to our new mitigation solution.

We are still working on resolving the issues with email delivery and anticipate to have a complete resolution within the next couple of days. We are literally working on this around the clock, with involvement of VP-level executives from our data center and DDoS mitigation partners.



Over last 15 years we have repelled many DDoS attacks using our infrastructure. More recently, we employed CloudFlare Enterprise solution that has helped us to deal with previous attacks in March and April. Unfortunately, the scale of last weekend’s attack has been so huge that it overran our current mitigation solution, and we needed to come up with a new plan.

After a conference call with our data center, within minutes we got in touch with Akamai, the world’s leading DDoS mitigation provider, with 1.8 Tbps mitigation bandwidth. We asked them to deploy an “always on” clean-pipe DDoS scrubbing service for GetResponse. This is the most advanced type of protection available on the market (also most expensive). It scrubs all inbound traffic for malicious packets of data and returns only “clean” traffic to our data center.

In total, we are spending close to half a million dollars in mitigation solutions, hardware, connectivity and other upgrades.



This wasn’t the first attack that GetResponse has encountered recently. We endured several other attacks in March and April. And we weren’t alone…

Over the last two months this criminal has targeted many other large Internet companies, putting them offline as a consequence. These include Meetup, Shutterstock, MailChimp, Fotolia, Basecamp, oDesk, SurveyGizmo, MadMimi, OkCupid, HootSite, Typepad, Elance, Aweber and others. It is unfortunate, but these types of attacks are becoming much more frequent in today’s environment.

Why would anyone attack well-meaning companies disrupting their business?

This email explains it all:



We weren’t interested, and won’t ever be, even if they continue to attack us. The low “offer” ($750) was just bait, and we know they’d come back for more. Paying would only encourage them to attack other companies. Besides, we are confident that we can protect ourselves in the future with the mitigation plan we’ve put in place. But above all, paying criminals is simply the wrong thing to do.

I sincerely apologize for this outage. We care about you, our customers, and we know that you rely on us to get your emails out to the world. Since the attacks, we’ve been working day and night to get the mitigation in place.

I thank you for your trust, your support and your loyalty during these difficult moments.

Regards,

Simon Grabowski

CEO

GetResponse

PS.: If you have any questions about this issue, please contact our Customer Service at support.getresponse.com (GetResponse 360 customers, please contact your Account Managers)..
#main internet marketing discussion forum #ceo #getresponse #grabowski #message #simon
  • As a long-time customer I'm glad to see you posting an official statement here. I'm getting really tired of these DDoS attacks that are getting to be more and more of a nuisance for many companies and services. Good thing you didn't cave to his/hers demands, as you say it would probably only make things worse.

    Also... as a Bitcoin fan and proponent, this makes me a bit sad Guess the semi-anonymity of Bitcoin comes at a cost.
    • [ 1 ] Thanks
  • Thanks for the update

    Rootin' for you guys !
    • [ 1 ] Thanks
  • Thanks for the update Mick! So it's looking like we're still a few days out...

    That's good to know so we can all plan accordingly. Communication from GetResponse via Twitter and Facebook has been horrible.
    • [2] replies
    • Thanks for all the info, Mick. I'm glad this lowlife criminal isn't being bargained with. Let's hope we can put this all behind us as soon as possible.
    • Yes at least a few more days, If there was anything humanly possible to get us up sooner we'd be doing it.

      I'll pass on your comments about the communication on Facebook and Twitter.

      Apparently I scooped our own blog.

      I'll keep passing on the information as I get it.

      Cheers,

      Mick
      • [ 4 ] Thanks
      • [1] reply
  • its becoming more common because people pay the extortion..
    • [1] reply
    • Hi Mick,

      thanks for the update! I cannot believe that they were actually asking for money!
      • [2] replies
  • Sorry to hear of these attacks man.

    Everyone was bashing Aweber when it happened to them...

    But the truth is that these odiously widespread DDoS attacks have potential to overload even the most massive of servers. (I would say medium-sized businesses like Aweber and GetResponse are at more risk than a MASSIVE server like Google/Facebook/Amazon).

    Still... DDoS attacks are a significant yet little-understood problem.

    I wish all the companies the best, and am sorry for everyone who lost cash as a result of these abhorrent attacks.

    • [ 1 ] Thanks
  • Damn peanut butter thieves! They are out of control!

    Hey Mick,

    Is there any relief provided/offered/available from our fearless governments?

    Seems like this type of terrorism would warrant some resources.

    Brent
    • [ 1 ] Thanks
    • [2] replies
    • Not to my knowledge. I haven't heard anything about government involvement, but I can ask.
      • [ 1 ] Thanks
    • Nyarnyarnyarnyar! (it's my version of lol.)

      Probably funds are low for these culprits, and the debts are high..
  • Thanks for revealing the problem in front of us, Mickk, now things are much more clear...
    • [1] reply
    • I feel the same way. I like it that you explained what was going on.
      I am APALLED that they dare ask for money - it is - what is the word? a RACKET, that is what it is. You did well not to pay. I agree with you, ut above all, paying criminals is simply the wrong thing to do.
      Take care, you and all the team that have been working around the clock!
      • [ 1 ] Thanks
  • Very interesting, thanks for sharing what goes inside with us.
  • Thanks for the heads' up - looks like a good time for me to take off for a few days!
    • [1] reply
    • I love Get Response and will continue to use them.

      I'm sure once things settle down we'll be hearing how they plan to compensate people who lost out on promotions, etc while the site was down.

      Here's hoping it gets fixed soon.

      Tim
      • [ 1 ] Thanks
  • Thanks Mick for an update.
    Bitcoin transactions are not exactly untraceable - all transactions are visible within blockchain.
    Having few of these marked as "criminal" leaves a trail for math geniuses to investigate and chance for authorities to track.
    At some point criminal need to exchange them for goods or cash - and that leaves a small, albeit above zero probability of tracking the offender.

    It is a fact that expensive infrastructure upgrade is a must and kudos to GetResponse.
    But, contrary to public outcry, I wonder if paying criminal a ransom would bring lot more benefits than not:
    1. Buying time to upgrade infrastructure while customers are happy.
    2. Creating a digital trail for authorities to investigate therefore increasing a chance of capturing and prosecuting the criminal.

    2 weeks ago I just referred two customers to GetResponse, one of them is a market trader who ought to deliver time sensitive information hourly to his list - and I convinced him to leave another autoresponder service and take customer list to GetResponse.

    With GetResponse API (I just got respond to my ticket confirming that) is not operational and no ETA for the fix - the amount of money and service trust lost is humongous. Hence my rant about ransom strategy

    Gleb
    • [ 1 ] Thanks
    • [1] reply
    • Hi Gleb,

      Assuming that paying off the cyber terrorist wouldn't make it worse for our customers. (I think you are wrong by the way, paying extortionist is never the right move)

      Windsteam (our data center) didn't give us the option you suggested.

      "On Saturday, after hours of working together with our data center Windstream (including their directors and VP-level executives), we received information that they have failed to mitigate the attack and of their decision to block our network connectivity until a powerful DDoS mitigation service can be fully implemented."

      Cheers,

      Mick
      • [1] reply
  • I had one of my sites hit with a DDOS attack and a script kiddie from Bulgaria asked me for $1500. Unfortunately for the script kiddie he got setup by the UK police and they extradited him to the UK.
    • [ 3 ] Thanks
  • I'm curious as to how you even know that the extortionist is, in fact, the perpetrator.

    I second any and all questions about government involvement.

    After a client of mine was hit with an attack by a gang of thieves from another country...I suddenly realized how vulnerable small and medium (and even large) businesses are to cybercrime ...

    ...but where to go?

    I have even proposed (just throwing the idea out there) ... that all those concerns about internet sales taxes might be well used to fund underfunded cyberdefense to protect not just the government, but citizens and individual businesses ...

    Live JoyFully!

    Judy
    • [ 1 ] Thanks
    • [1] reply
    • Zapseo - call it a hunch
      • [ 1 ] Thanks
      • [1] reply
  • Really appreciate your taking the time to update us here!
  • Looks like they just made an update to the status page:

    "We're happy to let you know that we've made huge progress in restoring our full mailing capacity. The current plan is to restore outbound email by tomorrow."

    One thing I'm thinking about is what's going to happen when GetResponse goes live again?

    They are queuing all the double-optin confirmation messages and broadcast messages. Many people are going to get a flood of emails.

    Personally, I cancelled any broadcast messages that have been sitting and don't plan on sending any for at least 72 hours after GetResponse comes back (I exported all of my most important lists/contacts out of GetResponse and temporarily plugged them into iContact since they don't require double confirmation and I was able to mail right away).

    This could be a big issue moving forward the next few days even after service has been restored.

    Many of the people I've seen on Facebook don't really understand how all this works and have been creating broadcast after broadcast message trying to "force" them through. One person even asked if they were allowed to send a "small, quick" message to their list because they thought only "large" messages weren't getting through and a "small, quick" message would be ok.

    I really hope there's some sort of guidance or plan GetResponse has to help customers fully understand what happens when all this goes live and has some clear-cut instructions on best practices or a lot of people are going to lose the majority of their list(s).
    • [3] replies
    • It's even better, brand new update:

      "Akamai and Windstream have finished provisioning our IP space through a fully routed always-on scrubbing service (clean pipe). We are now testing our IP configuration and working as fast as possible to resolve outbound mailing capability. Shouldn't take long now... "

      Cheers,

      Mick
      • [ 1 ] Thanks
    • Good point, I'm sure they are working on it, but I'll pass your suggestion along anyway.

      Cheers,

      Mick
      • [ 2 ] Thanks
    • Sounds like you were one set ahead of us.

      We are very close to being able to send emails again. If you have sent emails in the last few days this is your last opportunity to cancel them. If you have scheduled emails to go out this is your last opportunity to edit them.
  • The thing that concerns me is that hardly any online business could possibly take the steps that GetResponse has done. I don't have a 1/2 mil to invest in counter measures like this. Not many businesses do. Surely somewhere in all of this there is a trail leading to where the attacks are coming from and that where we need to be focusing.

    Sounds like a job for Seal Team 6 again.
    • [ 1 ] Thanks
    • [1] reply
    • More like for NSA to make themselves actually useful...
      • [ 3 ] Thanks
  • Banned
    On the positive side, this really portrays well for GetResponse brand overall; glad to see how you're doing what you can do help customers and sorry to hear about the DDOS attacks.

    Will very well consider moving to GetResponse or promoting your business instead (seeing the care you have for your clients) Will be in touch shortly
    • [ 1 ] Thanks
    • [1] reply
    • Thanks for your kind words humbledmarket.

      When you are ready to start promoting GetResponse let me know. I manage the affiliate program and love seeing affiliates succeed.

      Cheers,

      Mick
      • [1] reply
  • Do you have any intentions on crediting customers for every day they can't fully use your services?

    Can only imagine how much money was lost in paid advertising to landing pages with your non-functioning webform code on them. And now that those are working, the fact that autoresponder emails arent going out as well as broadcasts are not being able to be sent is costing companies money.

    Obviously I'm speaking for myself, but I'm sure I'm not the only one who's lost and is losing money as each day goes without your service being fully operational.
    • [3] replies
    • Hi Jasondinner,

      I'm not being glib, it's a good question, and I don't have a good answer...yet.

      Let me get back to you when I have a good answer.

      Cheers,

      Mick
      • [ 1 ] Thanks
    • I gotta say, Mick's comment is far more generous than I would be.

      This WAS NOT due to a shortcoming in GR's service, but in an external, unprecedented attack...

      ...the number of people who lost money on this...not just people who paid for traffic, but people who created products and launched them, sent out promotions and more...is undoubtedly substantial.

      ...This feels a tad like someone asking a for compensation from their utility company for spoiled food in their refrigerator after the utility company was bombed.

      That GR is even considering it (after having to outlay a 1/2 MILLION to get back online) ... is fairly impressive.

      Personally, I'd rather see them put effort & initiatives in terms of online security that would benefit us all as previous posters have at least alluded to.

      A bit of downside risk planning is something that more marketers should spend more time thinking about ...

      ....because I doubt this crapola is going to get any better, any time soon. Just a matter of what they choose to target NEXT...
      • [ 5 ] Thanks
      • [2] replies
    • Hi jasondinner,

      I checked into it, you can contact our customer success team at http://support.getresponse.com

      Cheers,

      Mick
      • [1] reply
  • Great to see communication, of course they had no choice but to communicate, it demonstrates just how serious bad internet traffic can disrupt your business, even end it if you do not have the resources to get advanced corporate resources.

    DDOS is hard to stop, really all you can do is slow it down.
  • Are other GR users NOT seeing their opt-in numbers increase on the dashboard? Just curious since mine aren't but the status page says opt-ins are working.

    -g
    • [1] reply
    • I'm posting all zeros as well. Talked to live chat about it and they said they are capturing the lead and will add all new optins once they have ability to mail autoresponder messages.

      I also tested some of my optin pages and you end up on the thank you page.

      Good thing I'm profitable right after I acquire the lead
      • [ 1 ] Thanks
  • I think you've been nothing but transparent about the whole thing. Well done! It sucks that these criminals also give the cryptocoin world a bad name as well! Not everything in the crypto world is evil so keep that in mind.
  • I love what you're doing. I use both aweber and getreponse but you guys have a top quality service. Keep up the good work! : )

    Joe
  • Thank you for the update, already saw it through official channels but I am glad to see that you are reaching out here as well.
  • Happy to see updates. Thank you Mick.
    • [1] reply
    • If you're a GetResponse customer and felt your business took a hit from the recent problems, ask them what they can do for you.

      I asked, and I'm happy with what they offered.

      Marcia Yudkin
      • [1] reply
  • I cannot begin to imagine the scope of effort it took to come back from this attack. Kudos to GR for not caving in and building back their business.
    • [ 1 ] Thanks
  • Thanks for reaching out on here Mick.

    Has full mailing services been resumed or are you still saying Friday?

    Cheers Z
    • [2] replies
    • Not sure if full services have been resumed but one of my newsletters got sent out a couple of hours back.

      I've been pleased with Get Responses handling of this problem who have kept us constantly updated.
      • [ 1 ] Thanks
    • Hi Zedsta,

      I like to under promise and over deliver.

      We have quite the backlog of emails to deliver so if you hit "send now" it won't be sent as quickly as you are used to.

      We'll get there.

      We're not sprinting yet, but we are definitely walking briskly.

      Thanks for your patience.

      Latest status update:

      "Update - We're currently delivering emails held in queue as well as add queued contacts to your target campaigns.
      May 1, 04:25 EDT


      Update - Great news: We have managed to resolve outbound mailing capability. All emails that had been scheduled in the past days will now be sent, however it may take several hours in order to provide the best possible deliverability.

      New scheduled emails may have a bit longer delivery time as we are dealing with a quite large volume of emails that were scheduled in the past few days.

      If you have any questions about your newsletters or account, please contact our Customer Service Department at support.getresponse.com (GetResponse 360 customers, please contact your Account Managers). "
      • [ 2 ] Thanks
  • Hi Mick,


    I am considering reliable auto responder company between Getresponse and Awaber.
    Understand that it is inevitable for such attacks happened every day, everywhere,
    I appreciate your effort to explain the turbulence happened there.

    I will go for GetResponse service as I believe you guys will do your best to provide good service.

    Thanks
    • [ 1 ] Thanks
    • [1] reply
    • Thanks SeoApprentice,

      If you can use someone's GetResponse affiliate link to sign up I'd appreciate it. I'm responsible for our affiliate program so I naturally like to see the sales take place there
  • Definitely appreciate the update. Hate that someone would do that, but it takes all kinds I guess.
  • Hi,

    Yeah I missed out on 2 days of emailing due to this little mishap.

    Glad you didn't back down to these people as you are right it will
    only encourage them to do it more.

    I suppose it's like the MODERN DAY PIRATES of the Internet we are
    seeing emerge here.
    • [ 3 ] Thanks
  • I am a get response customer and I am glad how professionally they handled the attack.
    • [ 1 ] Thanks
  • Although I'm not a GetResponse customer, I have worked directly with Simon in the past and have to say he's an upstanding professional who truly cares about his customers, and I have no doubt things will get back on track soon.
  • Down for me, too....
  • Sure is nice that they have provided updates here.

    But I sure picked a bad time to attempt to launch a new project.

    Guess I'll go fishing in the morning and hope for the best for later in the day tomorrow.
    • [1] reply
    • GetResponse completely down, yet again.

      Can someone track down whoever is doing this DDOS bullsh** and put a bullet in their head? Thanks.
  • very delicate subject
    if i had as much money as getresponse have, i would have paid the $750 to buy some times while i implement a solution to prevent that again.

    I had to install an sales funel yesterday for my new website so i choose aweber.

    Each times someone will talk me about a mailing list, i will never suggest getresponse because for whatever reason they can be under atack, down and make customers loose money.

    I know i have not a moral viewpoint on this but i can see how this energy is bad for getresponse customers so would be paying to buy peace (while i bullet proof my network)..
    • [4] replies
    • They apparently were shut down by their crappy datacenter provider service before they had a chance to pay ransom (although they probably wouldn't pay it anyways)

      Gleb

    • Saying this as nice as possible...Do you also send money to people in Africa that promise you 10 million dollars, but they need $500 first in order to wire the money?

      What if you had one of those viruses on your computer and it said "Pay $50" and you get your computer back. You really think you would just pay it and the hacker would be thinking "Oh okay, cool he paid me $50, no reason to bother him again"

      NO, these types of people LOVE people like you so they can hit them up multiple times before the person actually gets smart about it.
      • [ 1 ] Thanks
    • You're obviously not up to speed on "historic DDoS attacks...from just 2014!" ...aWeber was hit badly back in February ... have their fixes been better than those of GR ? Hard to say...

      I doubt there's enough about the specific details (for security reasons) to make an adequate assessment.

      Live JoyFully!

      Judy

      P.S. And I am not even a GR customer at this time...I have been with aweber for a LONG time, however.
      • [ 3 ] Thanks
    • It can happen to any company - and it has. This was not Getresponse's fault - I am not a customer and I still would stick up for GR - because in my opinion, they did everything right - not paying the thug, paying for mitigation, and communicating with their members throughout the ordeal. Outstanding customer service is priceless, in my opinion, and for all those people who are blaming GR - you don't know what you're talking about - and until it happens to you, you'll never understand.
      • [ 2 ] Thanks
      • [1] reply
  • I would pay 5 cents for sure as it's more what $750 is for getresponse

    Again, that's all relative and it also depend of the price to pay, like someone said, would you pay if someone have a knife on the troat of your baby ?

    My viewpoint is easy, do what is best for energy and don't bother overthinking about it.

    but he, i'm a crazy guy and i'm out of my mind for a long times lol

    i got 2 things that govern my life.

    1. Where do you want to go wand what price are you willing to pay ?
    2. Stand for something or fall for anything.
    • [1] reply
    • The knife to your child analogy is an awful one though...if that were really the case, obviously you would give it a shot a pay..BUT you wouldn't be letting your baby out of your sight until he/she was grown so there wouldn't be a chance for that person to kidnap your child again so quickly.


      A better analogy would be if someone stole your new 40,000 dollar car and said they would give it back to you if you paid them 1000. In that case I would not and I would let the proper people(police) handle it.


      There are no ifs ands or buts here...had getresponse paid the 750 then the hackers 100% would hit them again and want 1000 or 1500 and they would keep doing it until the company wised up.
      • [ 1 ] Thanks
      • [1] reply
  • Banned
    So does Windstream have a solution in hand ready to implement?

    Is a resolution to this problem imminent? I realize this is not an easy issue to deal with as I just recently mitigated a small scale attack on one of my sites.

    I am crossing my fingers that you guys have your finger on a fix that gets the job done very soon.

    Best of luck to all of us!

    Regards,

    -don
    • [1] reply
    • Mickk,
      Won't give you any credit on the Windstream front. They are crappy and everyone knows it. Want to stop those issues, get new pipe.

      As far as not paying extortion. Hats off. If GR had paid this would never end. Capitulating to extortion is the same thing as saying my integrity is 0 and I have no spine. I would be watching everyone who said to pay up and take that in account in any future dealings. It's a sign of bad character.

      Just on the integrity of their stand deserves a smidge of loyalty and a bit of respect. Windstream, a grazing tap to the junk, but integrity A+

      Seriously, Windstream??? SMH
      • [1] reply
  • Mick, I'm sorry you guys got hit. Thanks for your transparency. And for taking action. Good luck.
    • [ 1 ] Thanks
  • Thanks for the update. Not happy with it of course........ This is costing me money. Non of my emails can go out.......

    Hope you fix it soon.

    Good luck!
    • [ 1 ] Thanks
  • Thanks for the updates. I'm a brand new GR customer....hope to see the service back soon!
    • [ 1 ] Thanks
    • [1] reply
    • What's a quick solution not to lose the emails while this Getreponse DDOS issue is fixed? I have a bunch of sites sending sign ups to Getresponse API.

      I'm losing email sign ups!!!
      • [1] reply
  • I'm sorry this is happening to Getresponse, a so reliable service that helped me in last year and half to get my list, to grow it and much more. Right now hackers are trying to destroy all world, going to hit all the big services who earn money.

    By my side I've now put up a replacement autoresponder on my server, well protected and on a minor hosting (unknown to me as much as to hackers), where together with Mandrill I'm able to send out my email day by day.

    Hope everything will come back to normality, and that finally Getresponse will be able to protect itself from these bad guys, even if it will be a hard one to one battle.

    Even your choice of not paying the hacker is perfectly right, otherwise 10 more, 20 more will do the same with you and with other companies without solving the real problem behind this.

    Thanks for updating us, and see you soon!
    Alessandro
    • [1] reply
    • Well protected? If large companies have problems with and Denial of service attack then you can get problems too.

      Timo
  • Hey Guys,

    I am having serious issues with deliverability now. My open rates went extremely down after first outtage. So i checked with subscribers and my own test emails and it went to spam.

    I am using similar emails as before and even tested emails that should hit inbox anytime.
    I have chat with support a few times, but we can't really fix it.

    I would like to know if other people are facing the same problem?
    I also changed my domain email already.

    Thanks.
    • [3] replies
    • Zlatan,

      PM or Skype me your GetResponse Login.

      I will ask our deliverability team to look into it for you.

      Or you can always contact customer service.

      CHeers,

      Mick
      • [1] reply
    • I am having similar problems.
    • I checked with my delivery team, this is a temporary issue as a result of the DDoS attack. They are slaving away night and day to fix it.

      We apologize.

      Cheers,

      Mick
      • [1] reply
  • Sorry to hear about this. Hope you guys can withstand such things in the future.
    Great job you are doing.
    Best wishes and regards.
  • I am looking at getting into list building this weekend but now am at a quandry...aweber vs getresponse... is there no point in even considering the latter any more because of the DDOS?
    • [1] reply
    • Banned
      Hello jex1,

      It looks like they have things back under control as GetResponse has been working fine for me lately.

      Best of luck!

      -don
      • [ 1 ] Thanks
  • You can't look at that as a big long-term factor. Not without more info on their DDoS mitigation plans than most of us have.

    Aweber got hit. MailChimp got hit. Quite a few companies have gotten DDoS'd recently. It's just going to become more common and more extreme. No-one is immune.

    Until we get more info from the various companies involved about how they're handling the attacks, or see enough long-term impact to guess, the best strategy is to leave the DDoSs out of consideration and look at the other factors.


    Paul
    • [ 1 ] Thanks
  • Hello everyone:
    So glad Get Response handle the crisis with constant updates..... I use both Get Response and AWebber and both of them are really good and reliable... Their customer service in great..
    Things will go wrong sometimes..... How to handle a crisis is the key...
    Happy everything is back to normal.
    Gloria
  • Down again for me... This is becoming tedious
  • [DELETED]

Next Topics on Trending Feed

  • 161

    Dear GetResponse Customer, As you may know, GetResponse has had a pretty rough weekend… We suffered a major outage caused by a DDoS (distributed denial of service) attack on an unprecedented scale, which has flooded our network and our data center (Windstream) with malicious traffic.