Summary of the December 17th event in the South America Region (SA-EAST-1)
December 20, 2013
We want to give you some additional insight into the event that impacted a single Availability Zone in the South America Region (SA-EAST-1). On December 17th at 10:05PM PST, the impacted Availability Zone lost utility power due to a fault that happened at the substation of the local utility provider. The impacted Availability Zone automatically switched over to run on generator power when utility power was lost. Availability Zones are built with multiple layers of redundancy, and are designed to continue to operate even when multiple components fail at the same time. In this particular case when we experienced a loss in utility power, the load switched over to our backup generators as designed. During that failover a breaker in front of one of the generators opened, rendering that generator unavailable. Shortly thereafter, a second generator independently failed due to a mechanical issue. The loss of utility power combined with the unavailability of two additional generators meant that there was more load in the facility than the remaining healthy generators could handle. With more load on them than they could support, the remaining healthy generators also shut down. Our facilities team immediately began working to bring the failed generators back online. This facility uses an automated control system which allows it to aggregate power from multiple generators together. The team experienced several additional setbacks when trying to bring the power infrastructure back online, and eventually identified that the automated control system wasn’t functioning properly. Once the team identified the issue, they bypassed the automated control system and began the slow process of manually bringing the generators online. Once there was sufficient generator capacity to fully support the facility, all of the impacted instances were recovered. While we have not completed forensics on the breaker opening and the generator mechanical failure, we find the double failure to be extremely unusual, and are deeply reviewing the operational records of the failed components.
Instances in the second Availability Zone in the Region did not experience any power related issues, however instances in both Availability Zones did experience a total of 20 minutes of degraded network connectivity due to an error that was made in bringing our network back online once power was restored. As part of the recovery process, a network technician brought a network device up manually in the power-impacted Availability Zone and introduced a bad configuration. That misconfiguration led to the device advertising an invalid network route when it came back online, which resulted in degraded Internet connectivity for both SA-EAST-1 Availability Zones. Once we understood the issue, we took the device out of service and full connectivity to the Region was restored. After power and networking were fully restored to the facility, all of our services were brought back online and full customer access was restored.
We apologize for any difficulty this event may have caused you. We appreciate how critical our services are to our customers, and will take steps to ensure this Availability Zone in Brazil is better able to withstand a similar power failure in the future.
Sincerely, The AWS Team