We are experiencing Networking issues
Incident began at 2021-11-12 00:30 US/Pacific and ended at 2021-11-12 02:14 US/Pacific (1 hour 44 minutes).
Google’s Front End load balancing service experienced failures resulting in impact to several downstream Google Cloud services, primarily in Europe.
Root cause
From preliminary analysis, the root cause of the issue was caused by a new infrastructure feature triggering a latent issue within internal network load balancer code.
Customer impact
Multiple services were affected, including Cloud Developer Tools, Cloud Endpoints, Cloud Eventarc, Google App Engine, Google Cloud Bigtable, Google Cloud Console, Google Cloud Infrastructure Components, Google Cloud Networking, Google Cloud SQL, Cloud Load Balancing, Container Registry, Cloud Spanner, and Operations.
Specific customer-visible symptoms included:
- Google Cloud Console loading failures and timeouts, primarily for customers in Europe
- Container Registry API connectivity issues in europe-west1
- Cloud Bigtable unavailable errors and elevated latency for both data and admin operations
- Cloud Spanner unavailable errors and elevated latency
- Cloud Endpoints unavailable in europe-west1 and europe-west4
- Firebase Cloud Messaging notification delivery issues in asia-southeast1
- Cloud SQL instance/probe creation failures in multiple European regions
- Eventarc trigger creation failures in europe-west1
- Google App Engine elevated errors in europe-west1
Remediation and resolution
The error was caught within 4 minutes by automated safety systems. The issue was fully mitigated approximately 1 hour 44 minutes later, when the engineering team completed a rollout to disable the vulnerable code path. A rollout implementing a permanent fix for the root cause was expected to complete by 2021-11-12 21:00 US/Pacific.