Starting on September 27th at 07:16:00 until September 27th at 09:27:00 Universal Coordinated Time (UTC), Heroku customers using the Private Spaces product in the Virginia region experienced intermittent network connectivity failures from their dynos for the duration of the incident. We sincerely apologize for the negative effects our customers experienced.
Who was affected?
There was intermittent impact to all customers for the following features: Review Apps Creation and Deletion of apps Builds Release Phase CI Private Spaces
What happened?
At 07:16 UTC we received a number of alerts regarding unavailable Heroku Data Services, including Heroku Postgres services. We were alerted that some hosts were in the ‘restarting’ state. These databases were operating within a geographic zone that was experiencing network connectivity disruption.
At 08:20 UTC in an attempt to lessen the impact to our customers, we began temporarily disabling this geographic zone for any customers that had Private Spaces dynos residing within the zone. We then begin to move the impacted services to the unimpacted regions.
At 09:27 UTC, we confirmed that all databases had been successfully relocated and the original impacted geographic zone was no longer experiencing any service disruptions.
What did we do to prevent wider impact?
We disabled this geographic zone for all customers to prevent the creation of new databases in the geographic zone and relocated databases from the geographic zone to geographic zones that were not experiencing any issues.
What will we do to mitigate problems like this in the future?
We have begun distributing our services across a larger number of geographic zones to further reduce the impacted customers in zonal failures. We are identifying ways to make our systems more resilient to zonal failures.