On October 31, 2017, between 05:15 UTC and 14:10 UTC, our customers experienced errors while trying to perform actions on US applications. All dyno-related actions such as scaling, restarting, and running one-off dynos failed during this time. Application builds in the US and EU also failed. We accept full responsibility for downtime arising from this issue and apologize for the negative effects our customers experienced.
Who was affected?
All applications operating on the US Common Runtime had issues performing any dyno creation-related activity. Those activities included:
- Scaling and restarts, including scaling initiated by autoscaling
- One-off dynos, including those used by Heroku Scheduler
- Free dynos were unable to sleep or wake from sleep
- Dynos that crashed may not have been restarted
Additionally:
- Builds in both the US and EU failed to execute
- Some applications in the US Common Runtime experienced H99 errors
- When using the CLI or API to manage applications in the US Common Runtime, customers encountered timeouts or 503 errors.
During a small portion of the incident, all API, CLI and Dashboard access to Heroku in all regions was impacted. We quickly recovered from this additional load and we were able to restore management access to the EU region and Private Spaces.
What happened?
As part of follow-up actions for prior incidents, we have been scaling our routing infrastructure to increase routing capacity and reduce request latencies for our customers on the Common Runtime. These fixes had unintended consequences that did not become apparent until near the end of the month.
Near the end of each month, the number of accounts that have exhausted their free dyno quota steadily increases and hits its peak on the last day. The increase in the size of the Common Runtime meant that more components were sending messages through our infrastructure to manage free dynos. Due to a bug in the way these messages were delivered to an API gateway, that gateway became completely overloaded. The overload meant that it was unable to accept and route messages to downstream services. In turn, orchestration requests such as deploys, scaling, and restart requests were unable to be delivered.
Our monitoring alerted engineers to the increased error rate, but it did not provide insight into the source of those errors. This led our engineers down several incorrect paths of investigation. After several dead-ends, engineers from multiple teams were added to the incident response. These teams worked to find ways to restore partial functionality to customers. Unfortunately, due to limitations in our service lockdown procedures, restoring functionality to some impacted systems like builds was unattainable until engineers had resolved the incident.
Ultimately, engineers were able to track down the source of the messages causing the overload. Once identified, engineers took steps to block the traffic. This allowed full functionality to be restored. When the systems had recovered, additional steps were taken to scale the gateway API service to an appropriate level to handle the rate of incoming free dyno management messages.
What did we do to prevent wider impact?
As soon as we detected the issue, our engineers locked down the US region. This prevented job queues from backing up further and ensured that other regions would remain functional.
Several teams responded to this incident and were involved in different mitigation paths. These teams worked to restore services and find ways to route requests to new regions.
What will we do to mitigate problems like this in the future?
We’re investigating the following avenues for improvement:
- Improve alerting for free dyno management failures to make identifying a root cause of a similar issue much easier in the future.
- Ensure we have more fine-grained mechanisms in place to effectively pause certain operations, thus allowing us to quickly prevent wider impact and restore functionality to systems that are not impacted by the incident.
- Investigate the feasibility of running region-independent build infrastructure to further isolate a US outage from impacting the EU region.
- Investigate ways to reduce the request overhead associated with managing free dynos and our increased fleet size.