Google Cloud Incident #17007 - Full Report

Incident Title: 502 errors for HTTP(S) Load Balancers

Dates/Times:

• Incident occurred: April 5, 2017, 01:13–01:35 PDT (22 minutes)
• Report published: April 12, 2017, 14:11

Affected Product: Google Compute Engine - HTTP(S) Load Balancer

Complete Description

Impact Summary: During the 22-minute window, the service experienced widespread failures. Requests to the Google Cloud HTTP(S) Load Balancer experienced a 25% error rate. Recently modified load balancers were hit harder, with error rates of 100%. Configuration changes were frozen for approximately 3 hours 41 minutes afterward, which caused deployments of App Engine Flexible apps to fail.

Root Cause: A configuration management flaw triggered the outage. A master server replica lost access to Google’s distributed file system and was unable to read recent configuration files. Mastership transferred to this compromised server. During the subsequent configuration update test, the system reverted load balancers to the latest configuration that the master could read, which was substantially out-of-date.

The outdated configuration triggered excessive garbage collection on frontend servers, consuming significant CPU resources and causing failed health checks and eventual restart of affected servers. Restarting servers returning 502 errors to clients.

Resolution & Prevention: Engineers were alerted at 01:22 PDT and switched to an alternate master server at 01:34, resolving most issues within one minute. Google implemented five preventive measures, including configuration age limits, improved testing protocols, and fixes addressing filesystem access and garbage collection performance issues.