On March 3, 2020, between 17:15 and 20:10 UTC, customers of the Common Runtime experienced issues running dynos in the EU region. We sincerely apologize for the negative effects our customers experienced.
Who was affected?
A subset of all dynos in the EU region were stopped and failed to restart. Customers running formations with only a few dynos may have seen an entire application outage as a result.
What happened?
On March 3, 2020 at 17:01 UTC, a deployment of a service central to dyno execution was started, targeting the EU-based Common Runtime fleet.
At 17:28 UTC, our monitoring system alerted us to service degradation in the EU-based Common Runtime fleet. Investigation quickly revealed that a subset of the fleet was failing to run dynos.
At 17:41 UTC, the active deployment to our Common Runtime fleet was halted, and rolled back to its prior release. Following the rollback, no additional dynos were stopped. Dynos stopped during the incident began restarting.
At 18:45 UTC, engineers reconfigured the Common Runtime scheduler to increase the rate at which it restarted the collection of stopped dynos.
At 20:11 UTC, all dynos that stopped during the incident were confirmed to be running.
What will we do to mitigate problems like this in the future?
We are taking preventative measures to avoid future occurrences of this issue by both improving the gating mechanisms in our release pipelines, and reviewing our internal playbooks.
- Engineers will be revisiting the deployment pipeline for this service to improve the safety of our release process.
- Internal playbooks will be reviewed to clarify response actions so that customer impact can be communicated and mitigated quicker.
- Review environmental defaults and known capacity limits to ensure fastest recovery.