Mar 9, 2011

Yesterday’s DoS Attack – Response from CEO

As you may already be aware, yesterday morning Codero was hit by a massive Denial of Service (“DoS”) attack that began at approximately 8:00 a.m. CST and impacted a number of our customers.  This was a deliberate coordinated attack from multiple origination points.  The attack appeared to have been politically motivated and three other hosting companies were impacted in addition to Codero.

Codero immediately initiated countermeasures and contacted our upstream bandwidth providers who are usually able to mitigate the impact of DoS attacks.  In this case, however, the scale of the attack was so large that traditional approaches were not effective.  After several attempts our engineers isolated and blocked the source of the attack and took immediate steps to restore service to our impacted customers.  Law enforcement was also notified.

Unfortunately, because the internet now plays an unprecedented and important role in political events worldwide, these kinds of attacks are becoming more numerous and frequent and the methods of their originators are becoming more and more sophisticated.  Similar attacks hit the free-speech platform WordPress® as recently as last week.

Codero stands by our commitment to continuously improve our services and to communicate with our customers when unforeseen events like this occur, using Twitter, blog, telephone, and email.  We will continue to work with law enforcement as appropriate and work to prevent and resolve future attacks where countermeasures are met with new methods of attack.  As our teams continue to gather more information we will continue to communicate about what we learn and new additional countermeasures we are implementing.

Share on FacebookTweet about this on TwitterShare on Google+Share on LinkedIn

Tags: ,

  • I first noticed my servers were unresponsive around 9:15am. I called Codero support, but couldn’t get through and figured it must be a pretty big problem, since they are always available, even for minor questions. I pulled up their official Twitter account @CoderNOC and learned about the DDOS. My experience with Codero, previously, has always been exemplary. I updated my own customers, watched the Twitter feed and sat back in boredom as I waited confident that the worker bees at Codero would fix the issue. Great work, the internet is a vast and complicated place. We just try to keep our small part trucking along.
    Sure enough, by 3pm everything was back. Kudos and virtual high fives.

    • Shelby Garlock

      Virtual High Five Back!!

  • Codero handled this exceptionally well. I’ve seen DDOS attacks last for days. They handled this problem exceptionally well and in a timely manner while consistently updating their customer!

    Dedicated Customer with a Dedicated Server!


    • Shelby Garlock

      Thanks Jason! It’s our dedicated customers that keep us in business. 🙂

  • David

    The issue was not the attack, it was how it was handled. For the first several hours, they were not answering their phones. When we did hear from somebody, they kept saying “it would be resolved in 20-30 minutes”. We heard this over and over again for hours and hours.

    We passed on this information to our clients, only to be embarrassed when the problems persisted. Again, lack of communication was the primary issue.

    • Shelby Garlock

      David, this was an attack of massive proportions which also affected our chat and phone lines. Because of that we maintained an open dialogue of communication via our Twitter accounts providing a continuous flow of information as we learned of it. Your satisfaction is very important to us, as such our Director of Support will be reaching out by phone shortly.

  • Mike

    Do you have the Cisco Guard DDoS Appliance in place? If so, was it effective?

    • Shelby Garlock

      Mike, we do not have that specific appliance in place. Please stay tuned to the blog for a more detailed technical update.

  • Mike

    I have a suggestion. I couldn’t contact anyone to find out what was going on and just found out about the Twitter posts today. Why don’t you tell people about the Twitter updates on the phone message? It would have saved you guys a ton of phone calls and your customers would know what is going on.

    • Shelby Garlock

      Mike, thank you for the suggestion.

  • When I saw that my server was down, I went straight to the Twitter account. Kudos for @Codero and the updates through that account. I only saw one “20-30 minute ETA” mentioned there, and although it was longer than that it was still less than an hour or two from that point; other than that, I saw them advising people to switch to emergency backup plans if they had them. I felt like the reports through Twitter were exceptionally honest and all I could hope for. Looking forward to reading more details about the attack as they come out.

    • Shelby Garlock

      Jon, thanks for the kudos. Our COO has posted more details about the attack here on our blog.

  • Taliesin

    I’m sympathetic to the downtime – there is only so much that can be done in the face of large scale attacks like this, and I’m sure that Codero were working excellently behind the scenes to restore service. My experience with Codero has been excellent so far and an incident like this wont change that.

    Two things would have assisted me as a customer however.
    One would be to update your phone messages asap to inform as to what was happening.
    The second would be a proactive notification of some sort. I was actually able to swap important services to another machine elsewhere which only took about sixty seconds. However, I didn’t learn of the outage for around forty minutes until two clients reported problems.

    The only final thing I would love to know, purely out of curiosity, is more details about the “political” nature of the attack. However, I understand that these may not be forthcoming.

    • Shelby Garlock

      Taliesin, thank you for your feedback. We have put together a task force and are gathering suggestions like yours to evaluate and put in action for future instances.

  • We use a monitoring service ( so we knew of the outage almost immediately.

    It was truly shocking to me that the outage lasted as long as it did. Understandable given the size of the attack. I don’t think any of us expected an attack of that magnitude on a bunch of little guys.

    I understand Codero not posting more details on what exactly happened but news articles did have more details, you can Google it.

    Suggestions from Codero on emergency backup plans would be welcome.

    • Shelby Garlock

      Rich, our COO just posted an update outlining more details about the attack. You can read it here.

  • I feel that Codero handled the situation exceptionally well. As Jason appropriately pointed out, I’ve also seen these outages last for days with other data centers under similar attacks. It’s easy to say what should have been done after the fact, but for me (what was done) the frequency and honesty of Twitter updates and speed of resolution was exceptional given all of your “limitations” at that time.

    • Shelby Garlock

      Thank you Scott.

  • Mike

    I have a question regarding the suggestion that we switch DNS to point to backups if we can. If we had updated the DNS records associated with domains that point to sites on Codero servers to point to some kind of backup instead, wouldn’t the new DNS information have taken 24 hours to propagate?

    • Shelby Garlock

      Mike, depending on the TTL, domain name propagation can take as little as minutes. It all depends on caching at the provider level, but in any case, some traffic would start making it to the new destination in a short amount of time, while full propagation may take longer.

  • Steve

    the twitter feed information was good but would like to know what you have in place to prevent the the DOS attacks in the first place???

    As there clearly seem to be solutions available which could be used.

    • Shelby Garlock

      Steve, our COO just made a follow up post explaining the size of this attack and how traditional countermeasures were ineffective.

  • Paul Guziel

    As inconvenient and damaging as the attack was to our business, I was very impressed by the level of service and commitment that Codero was able to provide, given the circumstances. The attack clearly impaired their ability to communicate with us as effectively as they do under normal conditions. Yet they were able to provide regular feedback via the Twitter account and our rep Jeremy called me during the attack to provide any additional information that he had at the time. I appreciate all the efforts of the Codero engineers in limiting the downtime and getting our servers back online as soon as possible.

    • Jeremy

      Thank you Paul! We understand as an infrastructure provider that our customers rely on our services to support vital internal and external applications and solutions. In the event an outage occurs, accurate and honest communication of events along with continuous status updates are extremely important, not only for you as an organization to stay informed but also to provide you with the information needed to keep your customers up-to-date. As part of your support team I will always to try to do what I can to maintain a successful partnership between our companies and I sincerely appreciate your feedback and continued business.

  • Jay

    very nice to see a hosting company come right out and say what the issue was rather than say nothing at all…refreshing and impressive!

    • Shelby Garlock

      Thanks Jay!

  • I was out of town when this happened but started receiving calls from my clients within an hour of it starting. I called the support line immediately and sat on hold for an hour only to be disconnected. I called in again, and after another hour of waiting was disconnected again. By then my battery was dead. It would have saved me a ton of time if there was a message that said what was going on.

    Also, now over a week later, I haven’t received any kind of notification or explanation email. I’ve always gotten an email after a major outage like this but not this time.

    While the outage itself may have been handled well, the communication about the event has been horrible from the very beginning.

    • Chris Branding

      Greg, it’s unfortunate that you experienced this frustration during the DoS attack. Our COO has posted a follow-up on the attack, which specifically addresses some of the communication issues we had and why they occurred. Read his follow-up post here. As he makes mention of, we are committed to the continuous improvement of our communications plan and will be reviewing our policies and practices in the coming weeks. I also invite you to follow us on Twitter®, where we provide status updates when events like this occur.