Issue reported (2022-08-15 22:34 UTC)
Beginning at approximately 21:54 UTC Aug 15th, routing availability for US Common Runtime-hosted apps dropped below acceptable levels. Engineering was notified and immediately began scaling up capacity to address demand needs.
Update (2022-08-15 23:09 UTC)
Some customers apps hosted in the US Common Runtime region were experiencing increased latencies and increased error rates while attempting to route requests to dynos.
Engineers have rolled back a change that they believe to have caused the issue and are monitoring for stability.
Resolved (2022-08-15 23:16 UTC)
The issue was resolved at 22:42 UTC Aug 15th.
Follow-up (2022-09-23 20:22 UTC)
On Aug 15th, between 21:54 UTC and 22:42 UTC, our customers experienced increased latencies and increased error rates while attempting to route requests to dynos in the US Common Runtime region. We sincerely apologize for the negative effects our customers experienced.
Who was affected?
Dynos in a specific runtime with an overwhelmed router layer experienced request failures.
What happened?
A change to the core application that manages the underlying infrastructure for the Common Runtime included a dependency upgrade that caused a timing lock issue that greatly reduced the throughput of our task workers.
This dependency change, coupled with a failure to appropriately scale up due to increased workload scheduling, caused the application’s work queue to build up.
Contributing to the issue, the team was not alerted immediately that new router instances were not being initialized correctly on startup largely because of incorrectly configured alerts.
These router instances were serving live traffic already but were shown to be in the wrong boot state, and they were deleted via our normal processes due to failing readiness checks. The deletion caused a degradation of the associated runtime cluster while the autoscaling group was creating new instances.
This reduced pool of router instances caused requests to fail as more requests were coming in faster than the limited number of routers could handle. This is when customers started noticing issues with the service.
What did we do to prevent a wider impact?
To mitigate the incident, our engineering team rolled back the change that caused the issue, which allowed the system to stabilize again. Our team also manually scaled up the number of public-facing routers to handle the existing load on the system that it was unable to handle in the degraded state.
What will we do to mitigate problems like this in the future?
To prevent this issue from occurring again, our engineering team has implemented a stricter deployment process that more accurately tests real-world usage and uses new preventative checks before rolling out changes.
Our team has also adjusted the default scaling parameters to ensure that our infrastructure stays above what is required to handle incoming workloads, even under degraded conditions.
Finally, our team has increased the number of alertable metrics to help track down potential issues before they land in production or as soon as initial issues arise so related changes can be reverted more quickly.