Between May 31, 2018 at 21:52 UTC, and June 1, 2018 at 01:49 UTC, some customers experienced loss of access to their Heroku Postgres databases in the US Common Runtime and Virginia Private Spaces. We sincerely apologize for the negative effects our customers experienced.
Here is some additional detail about what happened and steps we are taking to mitigate future outages of a similar nature.
Who was affected?
Approximately 3% of databases in the region, across both the US Common Runtime and the Virginia Private Spaces region, were impacted by the incident. Approximately 90% of Premium databases recovered within 1 hour, either by failing over to their HA standby or recovery of their original service.
Standard databases, and a small number of Premium databases, were recovered from their Continuous Protection backups. The Premium databases that were not failed over did not meet their failover conditions as designed. These Premium databases, as well as Standard databases, were recovered as quickly as possible depending on their size (as larger databases take longer to restore), or when their original instance was recovered.
Additionally, approximately 50% of Private Spaces in the Virginia region were impacted as an internal database was unavailable. As a result, apps utilizing the release phase of builds were unable to finish and thus could not release.
What happened?
A power disruption occurred at our service provider resulting in a number of instances going offline. Heroku databases running on these instances were impacted. Our monitoring quickly detected these failures and paged our on-call engineers. Due to the number of failures that occurred simultaneously, a circuit breaker was triggered that stopped automatic recovery of all databases so that the engineering team could assess the situation. As a result of the wide impact, critical internal databases had to be recovered before recovery of customer databases could begin. The process of recovering impacted databases onto new instances began as soon as the situation was understood.
What will we do to mitigate problems like this in the future?
During the incident, we discovered a number of playbooks that could use improving and opportunities to create additional procedures, scripts, and playbooks. This will help us improve our response time and more rapidly recover databases should this happen again.