Global: Cloud Networking faced severe packet loss

Summary

Google Cloud Networking experienced intermittent packet loss affecting traffic between multiple cloud regions for approximately 20 minutes. The disruption began at 13:47 and was automatically mitigated by 14:07 US/Pacific on May 20, 2022.

Root Cause

A fiber path component failure at a central US gateway campus reduced available network bandwidth. The network topology was undergoing augmentation, forcing traffic rerouting through a third-best path rather than the intended second-best path, causing extended traffic migration and greater disruption severity than anticipated.

Remediation

Automated repair mechanisms detected the bandwidth decrease and rerouted traffic through alternate links, completing recovery by 14:06 US/Pacific. Google committed to completing network augmentation by July 11, 2022, and implementing automatic analysis to ensure fast convergence during future augmentations.

Affected Services

Cloud Networking, Cloud VPN, Cloud Router, Cloud Interconnect, Google Cloud Storage, Cloud SQL, and Messages.

Customer Impact

Affected customers would have observed packet loss for ingress and egress traffic to and from US Central Cloud regions, alongside elevated latency, HTTP errors, and API failures across services.