Beginning on September 20 around 09:13 UTC until 18:21 UTC, and then again on September 23 around 13:00 UTC until 18:19 UTC, the Heroku platform experienced an outage. The incident affected deploys, scaling, and our API. Running apps experienced an increase in the rate of H99 errors and increased latency.
Heroku users rely on the availability of our services to run their apps and, in many cases, their businesses. We take both availability and uptime very seriously. We sincerely apologize for the impact these outages had on our customers and users.
A great deal of thought and care goes into which providers and technologies we use to build Heroku. As such, any instability or unavailability due to issues with those providers or technologies are a consequence of our choices.
Many of the problems during this incident were caused by the AWS DynamoDB outage. The DynamoDB outage, in turn, caused issues with services we rely on in the AWS infrastructure. It also caused problems with Heroku services that directly rely on DynamoDB. Below we provide details on which services were affected and why they were affected. We also outline the steps we’re taking to mitigate the impact of future outages of a similar nature.
What was affected, how was it affected, and why?
Deploys and builds were unavailable in both the US and EU regions for nearly the entire duration of both incidents. The services powering these features have a direct dependency on DynamoDB, and were not functioning properly for the duration of the DynamoDB outage.
Dyno management, including scaling, provisioning, and un-idling was unavailable in both the US and EU regions for nearly the entire duration of both incidents. Heroku’s dyno manager, the internal service that manages dynos, has a direct dependency on DynamoDB, and was not able to properly provision new dynos while DynamoDB was unavailable. In addition we were not able to provision new EC2 instances due to the impact on Amazon’s own services. This limited our ability to add additional capacity, even if we had been able to restore service to Heroku’s dyno manager.
Apps that tried to restart (due to heroku restart, a deploy, dyno cycling, etc.) dynos would start to receive H99 errors. This was a result of the dyno management systems not being able to provision the new dynos. To prevent users from mistakenly putting their applications into this state we placed the Heroku API into maintenance mode, which allows running applications to continue serving traffic but prevents users from managing those applications.
As things started to recover we were able to pull the API out of maintenance mode. During the first incident, the sudden influx of requests on the platform for creating new dynos, deploys, and un-idling dynos put a huge strain on the Heroku router. The added strain caused apps to experience decreased availability and increased latency from about 14:00 UTC to about 17:15 UTC. At that time we were able to add additional capacity to our routing fleet to keep up with the increased demand. We were able to place the Heroku API into maintenance mode quickly enough during the second outage to mitigate this added strain during recovery.
Some databases failed to provision quickly during the incident. This affected customers who attempted to provision databases before the Heroku API was put into maintenance mode. The delayed provisions were caused by AWS not being able to create EBS disks because the EBS API was unavailable (because it also relies on DynamoDB). Once the EBS API was available, databases provisioned successfully.
What will we do to mitigate problems like this in the future?
We’ve been engaged in extensive discussion with AWS and the DynamoDB team about the nature and scope of the incidents and what we can do to make our systems respond more gracefully in the future. We’re re-evaluating our timeout and retry logic for access AWS APIs based on specific recommendations from AWS, and we will be implementing more circuit breaker patterns so that our systems recover more quickly from AWS failures.
Our internal retrospective revealed that disabling builds, dyno cycling, and the Heroku API earlier would have dramatically reduced the time to recovery (by helping to prevent a backlog of queued events). This understanding allowed us to manage the second incident more smoothly, preventing the significant latency and H99s that occurred during the first incident. In addition, we found that some of the timeouts in our orchestration system weren’t aggressive enough, which contributed to the event backlog and increased our time to recovery. We are revisiting these timeouts, which will help our systems process backlogs of this type more quickly.
We also discovered that our playbooks didn’t have enough diagnostic criteria in them- it was too hard for our engineers to decide which playbook applied to this event. This was partly because our monitoring and logging configuration had gaps that left our engineers with an incomplete view of what might be wrong. Based on this, we’re adding log lines, improving our monitoring coverage, and rewriting playbooks to provide clearer guidance on when and how to put our API into maintenance mode when we have large scale AWS outages.