On September 4th Heroku experienced an outage that affected both production applications and access to our APIs.
We’d like to apologize for this disruption in service – we know that many of you rely on Heroku for your production applications, and are aware we fell short of delivering a reliable service this time.
Below we’ll explain the impact, root cause, and remediation work being implemented to prevent this vector of failure from happening in the Heroku platform again.
Timeline
Starting at 05:30 UTC our queue responsible for propagating messages to different components in our distributed architecture (namely the router and the dyno manager) started to grow, leaving some of the apps modified during that period in an unroutable state. Due to a misconfigured monitor our on-call engineers were not alerted at this time, and the rate of H99s in the platform started to climb slowly.
At 07:13 UTC this incline reached a threshold and triggered another level of alerting, which put our engineers on call to immediately investigate the issue. The faulty monitoring delayed us tracking the issue to the growing queue until 09:18 UTC, when we added more capacity to handle the growing queue and brought the platform back to normal by 09:57 UTC. 0.07% of running apps were affected during this window. At this point we believed the queue was under heavy load because of a bug with how our API handles rate limiting, which turned out to be a red herring. We fixed the rate limiting in the API and fixed the monitoring on the affected queue, considering the issue resolved.
At 13:15 UTC the queue started to grow again and we were unable to contain it by adding more workers. At this point we put our API in read-only mode to avoid making the problem worse, and divided our response team in two fronts: to repair applications experiencing H99 errors, and to investigate and fix the root cause.
At 15:45 UTC we had a working script available to run in production that fixed affected apps and we started to see a decline in H99 errors. 0.25% of running apps were affected during this window, and by 15:57 UTC all of them were working properly again. The API was kept in read-only mode though; as much as we don’t like to prevent users from deploying and scaling their apps, we’ll continue to make these operations unavailable when they can compromise application availability, which was the case at this point.
On the root cause investigation front we identified at 14:25 UTC that our Postgres database was performing significantly below normal levels and that there was a lot of contention on internal operations (like running VACUUM). At 17:58 UTC we determined that a query running in a follower database used for business analytics was causing feedback into our primary database. Killing that query immediately fixed database performance and stability issues with the API. At 18:04 UTC the incident was considered fully resolved.
Root cause
This issue was caused by degraded database performance, triggered by long-running transactions in a follower which caused feedback and affected our primary database.
We have a safeguard in place to limit the effects of such a problem, in the form of a Postgres statement timeout. This should ensure that any such queries can only run for a limited amount of time. However, in this case it not function as intended. We’ve attempted to reproduce this failure, but in all of our testing the timeout has engaged as expected, and the safeguard is working properly in production. We will continue to monitor this system so that we can identify and correct the malfunction in the future.
Our incident response was delayed by a bug in our queue monitoring, and by a bug in the code used to resync application state across our distributed architecture.
Remediation
- We’ve fixed the monitoring on our queues.
- We’ve fixed the code path used to fix unroutable apps during outages.
- We’re researching alternatives on how to monitor this database contention.
- We’re committed to writing continuous testing to the code path used to fix the app state during outages to make sure we don’t introduce regressions in the future.