On August 31, 2019, between 12:48 and 21:22 UTC, customers using the US Common Runtime and Virginia Private Spaces region were negatively impacted by a widespread disruption to the Heroku platform. We sincerely apologize for the negative effects our customers experienced.
Who was affected?
All applications running in the US Common Runtime were potentially affected, with 7.5% of dynos failing. Web requests against applications with affected dynos were impacted, varying from degraded performance to downtime. In addition, a small number of data services were unavailable during this time. Apps in Virginia Private Spaces saw minimal impact, which was resolved by our automated systems or by actions taken by our service provider.
What happened?
At 12:48 UTC, engineers were alerted by monitoring to a widespread event affecting our data services. At approximately the same time, we were also alerted to a widespread event affecting our US Common Runtime and Virginia Private Spaces. Dyno management (starting, restarting, stopping, etc.) was intermittently failing. In the case of dynos failing to start, restarting those dynos was advised. We recommended against restarting running dynos as this may have caused them to not boot successfully. Additional teams responded to alerts for other Heroku services, including the teams responsible for the platform API and build services. These services were minimally impacted until the incident was resolved.
At 13:22 UTC, an incident was confirmed by our service provider. We engaged our service provider to confirm that an evacuation of affected infrastructure was the best course of action. At 13:45 UTC, we began evacuation of affected data infrastructure. At 14:35 UTC, we began removing affected routing and logging nodes to reduce customer impact and replaced them with new nodes. At 15:21 UTC, we disabled dyno cycling (24 hour restarting of dynos) to avoid increasing impact since restarting the dynos was causing them not to boot. Most data services were restored by 15:51 UTC, with a small number taking additional time proportional to the size of their data set.
At 15:57 UTC, we tried adding more capacity on new infrastructure with the goal of evacuating customer apps from the affected infrastructure, but we were unsuccessful. Unfortunately, two of our internal systems that we use to spin up additional capacity were running on the affected infrastructure. These services were replaced and were available by 17:40 UTC.
At 17:52 UTC, our monitoring confirmed we were able to increase the available capacity and began to do so. Over the next 80 minutes, we increased our capacity using new infrastructure until we reached sufficient capacity levels.
At 19:11 UTC, we re-enabled dyno cycling and informed our customers that dynos that crashed or were otherwise restarted during this incident would automatically restart soon or they could be manually restarted by the customer. Full recovery of all affected data services was completed at 18:39 UTC. All dynos had recovered or had been appropriately cycled as of 21:22 UTC.
What did we do to prevent wider impact?
From 15:21 to 19:11 UTC, we disabled dyno cycling to avoid increasing impact since these restarted dynos may not successfully boot. This also disabled manually restarting of dynos, since that may have had the same effect and resulted in customer downtime.
What will we do to mitigate problems like this in the future?
We understand the seriousness of this outage and how it impacted our customers. We failed to meet our customer’s expectations with respect to the resiliency of the platform and the time it took for us to recover from this failure. Overall, our ability to recover relied heavily on manual actions and critical decision making of subject matter experts, resulting in a slower response time. Less engineering intervention and more automation is key to reducing downtime.
To better prepare us to handle similar incidents in the future, we will be creating playbooks that will allow us to handle this type of incident with less downtime for our customers. As part of this exercise, we will test these playbooks in a test environment to identify gaps in our automation and internal systems and invest in making those better. Lastly, we will be running simulations of unanticipated and widespread failures in our test environment to identify gaps in our playbooks and tooling.
Additionally, some of our internal services were not architected in such a way to be resilient to failure, and thus needed to be replaced during the incident. While these services are not in the critical path of running customer applications under normal operations, they were in the critical path to being able to react to this failure. We will be ensuring these systems are highly available in order to be able to recover more quickly in the future. We are also reviewing other internal systems to ensure they are resilient to this type of failure.