On February 10, 2021 between 15:05 UTC and 18:05 UTC, our customers experienced failures using the Heroku Dashboard, Heroku CLI, and Heroku API. We apologize for the negative effects our customers experienced.
Who was affected?
Any Heroku user using services depending on the Heroku API, including the Dashboard and CLI, would have been impacted.
What happened?
On February 10, 2021, at 15:39 UTC, we detected a sharp increase in failures to communicate between Heroku Platform internal apps. Additionally our Platform API Gateway experienced a massive increase in request volume, which exceeded its capacity to forward the traffic to our Platform API.
As a protective measure, our engineers began throttling requests and scaled up the Platform API Gateway fleet to handle the additional load. However, the throttling impacted performance of Heroku Platform internal apps running in Private Spaces. Shortly thereafter, we disabled this throttling for those customers while leaving it enabled for the Common Runtime.
On February 10, 2021, at 18:42 UTC, the impacted internal teams reported recovery. Dashboard logins were succeeding and our incident commander called the incident closed.
What will we do to mitigate problems like this in the future?
As a temporary mitigation, we are maintaining our Platform API Gateway fleet at a larger size to increase capacity and de-risk recurrence of this or similar incidents.
Additionally, we will investigate and implement means to further reduce load on our Platform API Gateway to prevent recurrence of request saturation.
To reduce our time to detect issues in our Platform API Gateway, we will augment the Platform API Gateway to emit additional metrics to provide even more accurate indications of its health and performance.