Google Cloud Status Incident Report #16007
Incident Title: Connectivity issues in all regions
Date/Times:
- Incident began: April 11, 2016 at 18:25 US/Pacific
- Incident ended: April 11, 2016 at 19:27 US/Pacific
- Cloud VPN asia-east1 issue: Started 18:14, ended 19:27 US/Pacific
Affected Products:
- Google Compute Engine (all regions)
- Cloud VPN (particularly asia-east1)
- L3 network load balancers
Unaffected Services:
- Google App Engine
- Google Cloud Storage
- Other Google Cloud Platform products
- Internal connectivity between GCE services
- HTTP/HTTPS (L7) load balancers
- Outbound internet traffic
Complete Description
On April 11, 2016, inbound traffic to Compute Engine instances experienced routing failures lasting 18 minutes. Engineers removed an unused IP block from network configuration at 14:50 Pacific Time. A timing inconsistency triggered a software bug in the configuration management system, which instead of retaining the previous known good configuration, the management software instead removed all GCE IP blocks.
Safety mechanisms existed but failed—the canary step detected the unsafe configuration, yet a second software bug in the management software did not propagate the canary step’s conclusion back to the push process. As the faulty configuration rolled out progressively across sites, GCE IP block announcements ceased, causing traffic loss exceeding 95% by 19:09.
Engineers responded by reverting recent configuration changes, resolving the outage in 18 minutes. Google offered service credits exceeding their standard SLA obligations and implemented 14 engineering changes for prevention and detection improvements.