New VMs are experiencing connectivity issues

Incident: Google Compute Engine Incident #17003

On January 30, 2017, newly created Google Compute Engine instances, Cloud VPNs, and network load balancers experienced unavailability for approximately 2 hours and 8 minutes. The incident impacted resources created or live migrated between 10:36 and 12:42 PDT and prevented public IP connectivity and outbound traffic.

Root Cause

The issue originated in shared layer 2 load balancers responsible for inbound networking. A large set of updates applied to a rarely used configuration exposed an inefficient code path which resulted in the canary timing out. This created a backlog preventing all subsequent public address changes from processing.

Remediation

Engineers restarted the network load balancer programming jobs. The problematic changes processed as a batch and bypassed the inefficient code path, restoring normal traffic between 11:36 and 12:42.

Prevention Measures

Short-term: Increased canary timeout thresholds. Long-term: code path optimization, expanded test coverage, accelerated transition to decentralized routing, and enhanced monitoring/alerting systems to enable faster issue identification.