On March 8, 2011, shortly after 8:10 am Central Time, a very large scale Distributed Denial Of Service (DDoS) attack began that was focused on an IP administered by a DNS service provider, terminating in one of the networks of our Phoenix data center.
When the attack began, our network engineering team immediately implemented our DDoS recovery Standard Operating Procedures (SOP). These procedures include:
a) investigating the nature and type of the attack;
b) communicating with our upstream providers in an effort to thwart the attack; and
c) implementing network routing rules and other preventative measures within our infrastructure.
We also initiated our communications SOP’s by providing updates to customers through our @CoderoNOC Twitter account. Unfortunately, the Codero website was offline and we were therefore unable to update our network status page. In addition, our telephony system, which sits on the same network, was also affected by the attack. We were also unable to modify phone system greetings to assist in updating customers during the attack because the greetings server was also offline.
The volume of the DDoS attack exceeded 10 million packets per second into our core routing equipment. This is over 25 times higher than the maximum amount of aggregate traffic we receive during peak traffic times. While all hosting providers plan for significant traffic bursts, we simply did not have the capacity to handle a 25X increase over peak loads. As a result, we were not able to analyze the traffic internally, and we were forced to rely on our upstream providers for information.
Unfortunately, our upstream providers were unable to identify the targeted IP. So, our engineers redesigned our routing tables and started routing small blocks of the network one at a time in order to pinpoint the IP addresses in the range being attacked. We were able to narrow the issue down to the particular Class C (/24) of address space affected, disabled that /24, and restore service to customers outside of that /24. After service was restored to those customers, we were able to pinpoint the exact IP address targeted within the affected /24 and restore service to those customers that were not the focus of the attack. The IP is administered by a long-time Codero client who operates a large DNS service out of multiple data centers. The domain name being targeted by the attackers utilized our customer’s DNS service, and our customer experienced DDoS attacks at all 4 hosting companies hosting the DNS for the domain. Once the website’s record was removed from the customer’s DNS servers, the attacks ceased at all four locations early that afternoon.
DDoS attacks are more common and widespread, and they have negative effects on your business, and ours. As your hosting partner, we remain vigilant in preventing these types of incidents from occurring. When this one occurred, our first priority was to restore service to our impacted customers, and our entire company was mobilized toward that effort. We remain committed to clear and consistent communication with our customers, and did our best to communicate our progress through channels that were available to us. Still, we have heard from a few customers that feel that we did not communicate well enough. We understand your frustration, and as part of our commitment to continuous improvement, will be reviewing our policies and practices in the coming weeks and sharing our progress with our valued customers. We also continue to cooperate with Federal law enforcement officials, and would be happy to share our key learnings with other hosting companies to prevent future outages in their facilities.