Google Cloud Storage is experiencing elevated latency in multiple regions
Incident duration: July 14, 2022 19:30 US/Pacific to July 15, 2022 15:02 US/Pacific.
ISSUE SUMMARY
Google Cloud Networking experienced reduced capacity for lower priority traffic such as batch, streaming and transfer operations from 19:30 US/Pacific on Thursday, 14 July 2022, through 15:02 US/Pacific on Friday, 15 July 2022. High-priority user-facing traffic was not affected.
SYMPTOMS/DETAILED DESCRIPTION
Google Cloud Storage impact: Between Thursday, 14 July 2022 21:57 US/Pacific and Friday, 15 July 12:40 US/Pacific, GCS customers may have experienced elevated latency, delays, issues importing, exporting or querying data from GCS buckets, and HTTP 500 errors. This disruption affected 0.007% of GCS requests and impacted customers reading/writing data and metadata. Affected regions included us-east1, southamerica-west1, us-central1 and us-central2.
Google Cloud Networking impact: Reduced capacity was observed across multiple regions, with egress packet loss between 20% and 30% reported in the southamerica zones for lower-priority traffic classes.
Google BigQuery impact: Customers using the Storage Read and Write APIs experienced elevated latency.
ROOT CAUSE
The root cause was identified as an issue with a new control plane configuration rollout, causing low-priority classified traffic capacity reduction in Google’s internal backbone network.
REMEDIATION AND PREVENTION
Detection improvements:
- Improve signals to detect significant changes in traffic for upgraded domains in the backbone network.
- Improve debugging dashboards used during incident response.
Prevention measures:
- Improve the global safety systems that globally halt elective rollouts on Google’s wide area (WAN) networks when global capacity is reduced.
- Improve automated handling of disconnected network controllers.
Mitigation improvements:
- Provide better APIs for pushing configuration changes safely.
- Improve validation protocols applied before configuration rollouts.