PHX Power Outage – No Customer Impact

At approximately 7:55 PM CST there was a power outage in the City Phoenix due to a weather system that disrupted utility power to our Phoenix Data center. our back up generators kicked in as designed and planned with no impact to customers. We ran on generator power till 9:28PM CST, when utility power came back on. Everything is running normally now, however, our operations and engineering teams are on high alert and monitoring the situation closely till the weather system passes

Updates

  • There are currently no updates. Please subscribe for updates or check back soon.

04.13.2012-1 Investigating reports of service interruption

We are currently investigating reports of service interruption in the PHX datacenter. This does not appear to be related to yesterday’s issue. We will post more information as soon as it is available.

Updates

  • PatL says:

    The issue was resolved immediately and the failover in the networking systems functioned as designed and expected. The short traffic interruption is due to traffic in transit through the failed circuit dropping during the failover. Most customers may have not experienced any interruption. Only a few customers may have experienced a short instantaneous blip as the traffic was re-routed.

04.12.2012-1 Investigating network issues (Resolved)

We are currently investigating a network interruption for some customers in our PHX datacetenter over the last 15-30 minutes. We have switched off one of our redundant upstream providers that was having issues and are currently following up to ensure everything is stable.

Updates

  • PatL says:

    This issue is currently resolved and will be providing more information on the cause shortly.

03.09.2012-1 Temporary network degradation

This afternoon some servers in our Phoenix datacenter saw a connectivity degradation related to a DDoS attack that affected traffic coming from one of our upstream providers at 1:10 – 1:15 PM CST. Codero network engineers temporarily shut down the upstream provider to work around the problem.

Between 1:55 and 2:00pm CST Codero engineers restarted our DDoS mitigation system and re-enabled the upstream provider which resolved all issues, but may have caused an additional short interruption of service during this window. Total impact time for affected customers was approximately ten minutes.

Updates

  • There are currently no updates. Please subscribe for updates or check back soon.

03.08.2012-2 : DG-1 Intermittent Issues

We are experiencing intermittent issues with DG-1 in PHX.  Network engineers are addressing the issue.  Update soon.

Updates

  • Codero NOC says:

    DG-1 performance has stabilized, though we continue to monitor it closely for any degradation in service.

  • Codero NOC says:

    DG-1 has been stable for over an hour now. Engineers continue to analyze the root-cause of the problem with the core routing equipment to ensure our fix is complete. We apologize for the intermittent issues this afternoon and assure you we are committed to providing the reliability our customers deserve and will address any more issue.

03.08.12 – DG1 Network Impact

At 12:19 pm Central, a network equipment issue caused latency to customers in PHX DG-1 distribution group.  Issue was immediately addressed by networking team with complete resolution by 12:24 pm.

Updates

  • There are currently no updates. Please subscribe for updates or check back soon.

PHX Network Issue under investigation

We are currently investigating a network issue with one of our upstream providers which is affecting the ability of some clients in PHX from accessing their servers.  We have shut the provider down and are monitoring the situation.  Updates to follow.

Updates

  • Codero NOC says:

    We have shutdown our Highwinds circuit in PHX. Traffic issues are resolved.

  • Codero NOC says:

    Problem should be resolved. Engineering was on it ASAP. This was a network issue & we switched providers immediately.

Network Issue Under Investigation

An outbound network attack is affecting some clients on our network.  Currently, engineers have blocked the affected site and are shutting down attacking servers — which seem to have a commonality of running Parallels Plesk on a Windows server.  If your server is unreachable now, and does not have Plesk and Windows, please contact support.  Our Above.net traffic routing has been ceased to improve the issue for other customers.

 

Updates

  • Codero NOC says:

    A handful of servers remain compromised, with outbound bandwidth maxing out, restricting access to the compromised servers. Our data center hands are consoling affected servers to resolve remaining issues.

Phoenix Network Issue Being Investigated

Our network engineering team is currently investigating some intermittent network issues within our PHX data center.  We will update this status page and our @CoderoNOC twitter account with information as it becomes available.

Updates

  • Codero NOC says:

    Issues affecting some clients within our DC was actually backbone issues with AT&T and XO Communications, not an issue within Codero network.

Brief DDoS Attack

Today April the 1st 2011, from approximately 11:37 am CDT till 11:57 am CDT, Codero’s Phoenix facility experienced a brief DDoS attack. Network operations did not completely cease however high network latency was experienced. This issue has been resolved. If you are still experiencing issues, please contact customer support by submitting a ticket via https://serverportal.com/.

Updates

  • There are currently no updates. Please subscribe for updates or check back soon.