On September 1, 2021, between 00:47:00 UTC and 08:51 UTC, our customers experienced delays for release phase, Heroku CI, and app setups in all regions. We sincerely apologize for the negative effects our customers experienced.
Who was affected?
During the incident, all Heroku customers using release phase, Heroku CI, and app setups would have seen increased delays and timeouts resulting in failures for these services.
What happened?
At 2021-09-01 01:50:00 UTC, automated checks alerted our engineers because components reading from Heroku’s internal eventbus saw increased latencies. They observed a dramatic increase in requests to api.heroku.com, which saturated the system responsible for publishing events and prevented events from being processed in a timely manner. Engineers were able to mitigate the problem by pausing the collection of affected events. This caused latencies to begin dropping starting at 2021-09-01 06:39:00 UTC. However, due to the large backlog, latencies did not return to normal levels until 2021-09-01 08:17:00 UTC.
What will we do to mitigate problems like this in the future?
In addition to adding new instrumentation to dashboards to more quickly identify similar issues in the future, engineers realized that they were collecting more events than actually needed. Stopping the collection of those events has reduced pressure on the event publishing system. Work is also planned to improve the performance of the publishing system so that it can be resilient to increased event volume.