Google Cloud Storage Incident #19002

Elevated error rate with Google Cloud Storage

Duration: March 12, 2019, 18:40 to 22:50 US/Pacific (4 hours, 10 minutes)

ISSUE SUMMARY

On Tuesday 12 March 2019, Google’s internal blob storage service experienced a service disruption for a duration of 4 hours and 10 minutes. We apologize to our customers whose services or businesses were impacted during this incident, and we are taking immediate steps to improve our availability and prevent outages of this type from recurring.

DETAILED DESCRIPTION OF IMPACT

The internal blob storage service saw elevated error rates, averaging 20% error rates with a short peak of 31% errors. Google Cloud Storage specifically experienced elevated long tail latency and an average error rate of 4.8%. All bucket locations and storage classes were impacted.

Additional affected services included:

• Stackdriver Monitoring: up to 5% errors retrieving historical data
• App Engine Blobstore API: error rates peaked at 21% for blob fetching and 90% for deployments
• User-visible services (Gmail, Photos, Google Drive) were also impacted but the impact was mitigated by built-in caching

ROOT CAUSE

On Monday 11 March 2019, Google SREs were alerted to a significant increase in storage resources for metadata used by the internal blob service. On Tuesday 12 March, to reduce resource usage, SREs made a configuration change which had a side effect of overloading a key part of the system for looking up the location of blob data.

REMEDIATION AND PREVENTION

SREs were alerted to the service disruption at 18:56 PDT and immediately stopped the job that was making the problematic configuration change, and manually reduced traffic to allow the service to recover.

Google committed to the following long-term improvements:

• Enhanced regional isolation
• Faster resource provisioning
• Configuration safeguards to prevent this class of change from being applied unsafely
• Graceful degradation of the system under overload