On July 20, 2021, between 15:10 and 17:30 UTC, the Heroku Runtime team became aware of an issue that would fail to start new Performance-M and Performance-L class dynos in a subset of our fleet. We sincerely apologize for the negative effects our customers experienced.
Who was affected?
Customers using Performance-M or Performance-L class dynos were impacted by this outage. This includes web and work dynos; scheduled jobs; one-off dynos; and Heroku builds that use Performance-M or Performance-L dyno types. 737 customers using Performance-M and Performance-L type dynos were affected. The affected customers received network errors when starting new dynos during the incident.
What happened?
On July 20, 2021 at 15:10 UTC we started a deployment of our container execution engine, which included a feature pertaining to new improved shared dyno resources. This deployment occurs gradually across multiple infrastructure fleets. At 16:45 UTC alerting thresholds were triggered, notifying our Engineering teams of an irregular number of failures starting new dynos across multiple fleets. Upon further investigation, the team identified a misconfiguration within a subset of our global fleet. Specifically, our container execution engine was applying an unavailable feature to Performance-class dynos, resulting in startup failures. This configuration inconsistency was remediated, and the issue resolved.
What did we do to prevent wider impact?
To prevent further issues, we temporarily disabled aspects of dyno cycling (e.g. 24 hour restarts) to avoid running Performance dynos from stopping and failing to start.
What will we do to mitigate problems like this in the future?
We have made improvements to our configuration management process to reduce the likelihood of a skewed value across our fleets. We additionally began reviewing our rollout strategy for this component to identify opportunities to bring increased safety to, and decrease the potential impact of, future changes.