On November 2, 2017, between 06:48 UTC and 09:35 UTC, we experienced a lack of connectivity between the Heroku Data control plane and databases running in the Tokyo region. We sincerely apologize for any downtime arising from this incident and for the negative effects our customers experienced.
Who was affected?
Apps outside the Tokyo region saw intermittent connectivity issues to Postgres, Kafka, or Redis data services in Tokyo, even though services were still available within the region. Our data services control plane had intermittent contact to databases inside the Tokyo region, which meant we could not provision new databases, failover, or automatically resolve faults that occurred during the incident.
What Happened?
Our upstream provider experienced network saturation on regional links, which resulted in increased packet loss and network timeouts between Tokyo and outside regions. This effectively partitioned the Tokyo region from the other outside regions, and cut the servers in the region off from the data services control plane.
Our monitoring detected a loss of connectivity to a large number of Postgres, Kafka, and Redis databases. Our monitoring and automated responses have threshold limits to sanity check and prevent an overreaction by our control planes. These performed as designed, allowing responding engineers to isolate and respond appropriately to a network connectivity issue rather than to a high volume of server issues. Those alerts that were identified as actual faults were manually remediated as quickly as possible.
Once normal network operation was restored to Tokyo by our upstream provider, all data service operations were restored.
What will we do to mitigate problems like this in the future?
We are working with our upstream provider and refining internal monitoring processes to improve our detection and response to regional network events.
We’ll also update our disaster recovery program to address regional network health issues, and explore possible methods to manage the impact of network service degradation.