On Wednesday, April 22, 2020, between 19:13 UTC and 19:44 UTC, our customers experienced build, pipeline, CLI, Dashboard, and API failures across all Heroku regions. We sincerely apologize for the negative effects our customers experienced.
Who was affected?
During this time, the API returned intermittent errors, which caused tools that access information or make modifications to the platform to return errors. All Heroku regions were affected. Additionally, we were unable to process support tickets during the incident.
What happened?
On April 21, 2020, at 23:34 UTC, our engineers received a page indicating issues with a Redis database used to gather analytics data. On April 22 at 06:34 UTC, our engineers received a page for a related failing automated test. These pages automatically cleared as the issue was resolved without manual intervention.
On April 22 at 19:13 UTC, our support team noticed errors when accessing their tooling. A few minutes later, multiple engineering teams were paged as key metrics indicated failures were occurring in their respective services.
Due to our monitoring, we were able to determine the cause quickly. Most requests to our Platform API write to our analytics Redis database. Normally, this Redis database is drained to process the data stored within it. However, at this time the draining processes were not able to keep up with the traffic on the Platform API. This led to the Redis database becoming filled to capacity and resulted in the Platform API returning errors when it could not write to the filled Redis database.
Additionally, our Support system was unable to process tickets during this time, including by email. When the Platform API is unavailable, a code branch is activated which uses a backup system to allow incoming tickets to continue to be processed. This branch failed to operate as intended during the incident.
At 19:44 UTC, we disabled writes to the Redis database from the Platform API to mitigate the errors. This allowed Platform API requests to operate as intended and quickly brought the Heroku Platform back to normal operations.
What will we do to mitigate problems like this in the future?
Since the data that we collect in this Redis is not critical to the operation of the Heroku Platform, we will make a code change to prevent the Platform API from experiencing failures if and when the Redis database again reaches capacity. We will also create additional automated alerts and modify our operational playbooks for our operators to prevent the Redis database and the analytics system from entering a critical state in the first place.
We have fixed the issue which caused our support system to fail while utilizing the backup system. We have also created an independent backup system we can use should our support system be entirely unavailable.