Issues with Heroku API, Dashboard, and CLI
Incident Updates
Issue identified — June 8, 2023, 15:57 UTC Our engineers are investigating an issue identified with Authentication with global impact. Impacted customers will receive an “Application error” message with a 500 HTTP response on authentication requests from the Heroku Dashboard, Heroku CLI and Heroku API.
Update — June 8, 2023, 16:13 UTC Our engineers have identified the root cause of the issue and actively working on remediation.
Update — June 8, 2023, 17:26 UTC The Heroku Platform API is unavailable. Customers using the API, the CLI, and the Heroku Dashboard will be unable to make changes/releases or queries of their applications. Engineers are investigating, we’ll update as we have more details.
Resolved — June 8, 2023, 18:57 UTC The Heroku API, Dashboard, and CLI are working as expected. The incident is now resolved. There was no impact on running Heroku applications during this incident that are not using the Heroku API. The impact start time is 15:08 UTC and the end time is 18:24 UTC.
Follow-up (Postmortem) — June 15, 2023
On June 8, 2023 between 15:06 UTC and 18:57 UTC, our customers were unable to interact with the Heroku API, CLI, and Dashboard. Deployed applications continued to run as expected on the Common Runtime and in Private Spaces. Management of those applications, including new deploys, would result in errors. We sincerely apologize for the impact to our customers.
Who was affected?
For 1 hour and 45 minutes, all Heroku customers were unable to login or deploy new releases to their applications. Starting at 16:50 UTC and lasting an additional 2 hours and 3 minutes, the outage expanded to a full API outage, where all Heroku customers were unable to log into the Dashboard or CLI, promote slugs, interact with webhooks, or perform other application management tasks. For the entire duration of the incident, running applications were unaffected.
What happened?
At 15:05 UTC on June 8, 2023, a database error occurred where a foreign key used a smaller data type than the primary key that it referenced. This error caused an overflow when the primary key exceeded the allowable value, resulting in an inability to create new authorizations within Heroku. This error also prevented customers from creating new deployments.
Our on-call engineers responded by performing a migration to update that foreign key to the appropriate type. The migration was successful and new authorizations could be created, but altering the type caused the internal Postgres statistics to be cleared and query times increased significantly. This new impact cascaded into a full outage for the Heroku API.
The API was then put into a read-only mode so that engineers could perform maintenance activities. At 18:24 UTC, our engineers rectified the underlying statistics issue and query times returned to normal. The API was switched back to its normal read/write mode. The database was monitored to ensure the change was successful, and at 18:57 UTC the incident was closed.
What will we do to mitigate problems like this in the future?
We are implementing new tooling to better monitor our database schemas to avoid similar column type mismatches. We will also be improving our monitoring to watch for any columns or foreign keys that are approaching the limits of their data type. Finally, we will be updating our database migration playbooks to ensure that table statistics are updated appropriately when schemas evolve.