What happened
Between 20:52:27 UTC on 2023-03-16 and 02:30:00 UTC on 2023-03-17, the Heroku API was unavailable. This incident also impacted most Heroku CLI and Dashboard operations for Common Runtime, Private Spaces, Salesforce Functions, and Heroku Data. Customers could not log in, access dashboard metrics, perform releases, add log drains, or start, restart, or stop dynos. In addition, customers could not provision new add-ons, de-provision old ones, or use tools such as heroku pg:sql which depend on the Heroku API for authorization. Customer applications and add-ons already running during this time were not impacted.
Who was affected?
No Heroku customer could perform any action that depended on the Heroku API for approximately five hours starting on March 16, 2023 20:52 UTC, ending on March 17, 2023 at 02:30 UTC.
What happened?
Much of Heroku runs in Heroku Private Spaces, including the services that back the Heroku API. Two weeks before the incident started, Heroku Engineers changed the networking configuration of a service hosted in the same Private Space as the Heroku API. The configuration change contained an error that was not detected at the time. There was no immediate impact, as the Private Spaces Router gracefully degraded by continuing to serve traffic using the previous (valid) configuration.
Impact began on 2023-03-16 when an unrelated, routine infrastructure upgrade replaced the Router instances in the Private Space. The new Router instances failed to start up, as they could not parse the malformed configuration and no longer had a previous configuration to roll which to roll back. With no healthy Router instances left, all applications in the Private Space became unavailable. Heroku Engineers were alerted immediately, and the infrastructure upgrade was quickly rolled back. Unfortunately, rolling back the infrastructure upgrade did not resolve the impact because the malformed configuration was still present.
Once engineers identified the malformed configuration as the cause of the failing Router instances, they quickly attempted to revert to the previous known good configuration. Engineers manually corrected the configuration, allowing the Router instances to start successfully. This made the Heroku API available again.
What will we do to mitigate problems like this in the future?
In response to this incident, the following remediation work will occur:
- Improve tooling and playbooks used to respond to Heroku API outages.
- Increase resilience in the network router to handle incorrect configurations.
- Improve automation that generates router configurations to prevent incorrect configurations.
- Improve the monitoring of non-fatal errors from the router.