On Tuesday, February 3rd, Heroku’s Support team responded to customer reports of elevated error rates around application builds and execution beginning at 16:30 UTC. Many customers continued to experience a high error rate for the following hour. At this time, Heroku notified customers that the API had been placed into maintenance mode to allow engineers to address the ongoing incident safely. By 17:40 UTC, there was a notable improvement in error rates, but the incident was still under investigation. Heroku engineers continued to work actively to resolve the problem until 19:00 UTC.
###Who was affected?
This incident affected all customers in the US and EU regions and particularly impacted those running applications in the multi-tenant cloud in the US.
We know that our customers depend on our platform’s stability and reliability. During this incident, many users were unable to deploy their applications. We are deeply sorry for the disruption and accept full responsibility for any resulting downtime or negative effects.
###What happened during the incident and what did we do to prevent wider impact?
In response to a security threat over the weekend, Heroku’s Security Team requested credential rotations for services that Heroku engineers rely upon for paging and alerts. During this process, some alerts were effectively disabled.
On Tuesday morning, a kernel bug had impacted a significant percentage of Heroku’s multi-tenant fleet, which led to a precipitous drop in the number of instances available to run dynos. Unfortunately, engineers had not received any alerts regarding the loss of healthy instances. The result was that customers attempting to build or start their applications received error messages and found that they were unable to boot dynos.
Engineers were tasked with removing instances in a bad state from the platform as well as adding new instances to increase capacity. Once we were able to bring new instances online and remove existing unhealthy instances, the error rate began to fall, enabling customers to resume work on the platform.
###What will we do to mitigate problems like this in the future?
We are working to remediate the issues that gave rise to this incident and taking steps to make our monitoring system more robust. In particular, we are currently evaluating safer procedures for credential rotation. Please feel free to contact us if you have any additional questions or concerns.