On March 25th, 2021, between 13:34 UTC and 17:34 PM UTC, our customers experienced errors using the Heroku CLI, the Heroku Dashboard, and the Heroku Platform API. We sincerely apologize for the negative effects our customers experienced.
Who was affected?
All Heroku customers were unable to access the Platform API or use services which rely on the Platform API for approximately two hours, and were only able to access the Platform API intermittently for the remainder of the incident.
What happened?
During routine engineering work, we identified a legacy domain name for an internal component and determined that it was not in use. For operational hygiene reasons, we removed this domain name. The api.heroku.com domain name, which provides access to the Heroku Platform API, had been left configured as an alias for the legacy domain name, and when that domain name was dropped, clients were no longer able to resolve api.heroku.com or access the Heroku Platform API.
What will we do to mitigate problems like this in the future?
We have added automated validation to DNS changes, to discover DNS aliases that may become dangling before those changes go live. This complements our existing review process, and will improve our ability to make substantial DNS changes with confidence that they will not affect customers.
We are revisiting how we review DNS changes to encourage better collaboration between engineering teams and more complete understanding of the intent of these changes.