Beginning on December 15 at 0045 UTC and continuing until 0330 UTC, customers in our EU region experienced a reduction in availability for running apps. Some DNS lookups from apps failed, leading to intermittent failures of connections originating from apps. In addition, dyno cycling and restarts may have been delayed. We accept full responsibility for downtime arising from this issue, and apologize for the significant negative effects our customers experienced.
Here is some additional detail about what happened and steps we are taking to mitigate future outages of a similar nature.
Who was affected?
Most EU customers experienced intermittent failures of DNS lookups originating on running apps.
What Happened?
The EC2 DNS resolution service intermittently returned errors to processes on EC2 instances that were performing name lookups. This happened mostly in one availability zone (AZ). Because most applications have retries, and not every lookup attempt received an error, this looked like connection latency on the application level. Our orchestration system was also affected, resulting in a backlog of restarts and scaling requests. The orchestration system caught up over the course twelve hours after the EC2 DNS service had returned to normal.
What did we do to prevent wider impact?
Our automatic platform maintenance tools began responding to the issue by terminating affected instances. Our engineers worked in parallel to launch new instances in unaffected AZs. Finally, our orchestration system automatically moved the dynos which were affected by instance terminations. These factors together had the effect of moving the shared platform away from the most-impacted AZ so that customers could see recovery more quickly.
What will we do to mitigate problems like this in the future?
This was a difficult problem for us to diagnose because DNS is so fundamental to the operation of distributed systems that we were inundated with alerts from our monitoring. The volume of alerts suggested to us that it was a network latency issue, leading to confusion and delay in our response. The alert volume also obscured the most meaningful alerts. We’re working on a way to consolidate or deduplicate alerts in order to highlight the most significant or unique ones.
An addition, we’ll be consulting with AWS on the way we evacuate an availability zone and what a safe rate for that is.