Multiple Cloud products experiencing elevated error rates, latencies or service unavailability in europe-west2

Incident began at 2022-07-19 06:33 and ended at 2022-07-20 21:20 (all times are US/Pacific). Previously affected location: London (europe-west2).

Affected products included Cloud Memorystore, Google Cloud SQL, Google Cloud Storage, Google BigQuery, Google App Engine, Google Cloud Functions, Google Cloud Dataflow, Persistent Disk, API Gateway, Cloud Spanner, Google Cloud Tasks, Google Compute Engine, Vertex AI Online Prediction, Virtual Private Cloud (VPC), Cloud Firestore, Google Cloud Datastore, Google Cloud Composer, Cloud Data Fusion, Managed Service for Microsoft Active Directory (AD), Google Cloud Dataproc, Google Cloud Bigtable, Datastream, Google Kubernetes Engine, Cloud Filestore, Memorystore for Memcached, and Memorystore for Redis.

Incident Report (29 Jul 2022, 14:00 PDT)

Summary

On Tuesday, 19 July 2022 at 06:33 US/Pacific, a simultaneous failure of multiple, redundant cooling systems in one of the data centers that hosts the zone europe-west2-a impacted multiple Google Cloud services. This resulted in some customers experiencing service unavailability for impacted products.

To our customers whose businesses were impacted during this outage, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps (detailed in the Remediation & Prevention section below) to improve the region’s resilience.

Regional Impact

A number of regional Google Cloud services experienced impact during this incident, despite regional services being designed to survive the failure of a single zone. Upon investigation we have found two key contributing factors which led to these regional impacts:

• At the start of the incident, we inadvertently modified traffic routing for internal services to avoid all three zones in the europe-west2 region, rather than just the impacted europe-west2-a zone. We corrected this on 19 July 2022 at 12:35 US/Pacific.
• Our regional storage services, including GCS and BigQuery, replicate customer data across multiple zones. Due to the regional traffic routing change, they were unable to access any replica for a number of storage objects. This prevented customers from reading these objects until the traffic routing was corrected, at which point access was immediately restored.

Cooling System Impact Duration

Google engineers powered down the data center that hosted a portion of the impacted zone europe-west2-a on Tuesday, 19 July 2022 at 10:05 US/Pacific while the cooling system was repaired. The cooling system was repaired at 14:13 US/Pacific. The total duration of the cooling system failure impact was 4 hours, 8 minutes.

Cloud Service Restoration Duration

Google engineers began service restoration once the cooling system was repaired on Tuesday, 19 July 2022 at 14:13 US/Pacific. Cloud services were restored to operation by Wednesday, 20 July 2022 at 04:28 US/Pacific.

The duration of impact on Cloud services spans the window from 10:05 US/Pacific on Tuesday, 19 July 2022 (when a portion of europe-west2-a was powered down), to 04:28 US/Pacific on Wednesday, 20 July 2022, when Cloud services were restored; a total of 18 hours, 23 minutes.

Long Tail Duration

After the initial restoration of service to the zone, a small number of Google Compute Engine instances required additional work by our engineers to restore them to normal operations. This manifested as unavailable instances in GCE, and unavailable instances in products and services that rely on Google Compute Engine, such as Cloud SQL. This was fully mitigated on Wednesday, 20 July 2022 at 21:20 US/Pacific and the incident was closed with all services restored.

For the instances in this long tail, impact duration spans the window from the initial power down (at 10:05 US/Pacific on Tuesday, 19 July 2022) to the eventual full mitigation (at 21:20 US/Pacific on Wednesday, 20 July 2022); a total of 35 hours, 15 minutes.

Google is conducting a detailed analysis of the systems and processes involved in both the cooling failure and the service recovery, with specific followup AIs identified below.

Root Cause

One of the data centers that hosts zone europe-west2-a could not maintain a safe operating temperature due to a simultaneous failure of multiple, redundant cooling systems combined with the extraordinarily high outside temperatures. We powered down this part of the zone to prevent an even longer outage or damage to machines. This caused a partial failure of capacity in that zone, leading to instance terminations, service degradation, and networking issues for a subset of customers.

Remediation and Prevention

Google engineers were alerted to an issue affecting two cooling systems in one of the data centers that hosts europe-west2-a on Tuesday, 19 July 2022 at 06:33 US/Pacific and began an investigation. Engineers were engaged at 07:02 and began assessing viable mitigations. At 10:05, our engineers decided to power down servers in the impacted data center within europe-west2-a to prevent an even longer outage and further impact to infrastructure in the zone.

The cooling system was repaired at 14:13, and we restored our services by Wednesday, 20 July 2022, at 04:28 US/Pacific. A small subset of customers experienced residual effects which were fully mitigated by 21:20.

Google is committed to preventing a future recurrence and improving recovery times by taking the following actions:

• A small number of services experienced problems in zonal failover automation. We will repair and carefully re-test our failover automation to ensure stronger resilience in our failover protocols during large scale events such as this one.
• We will investigate and develop more advanced methods to progressively decrease the thermal load within a single data center space, reducing the probability that a full shutdown is required.
• Our initial recovery of impacted services once cooling was restored was 14 hours, 15 minutes. Additionally, the recovery of the long tail of impacted Google Compute Engine instances and related services was an additional 16 hours, 52 minutes. We are examining our procedures, tooling, and automated recovery systems for gaps to substantially improve our recovery times in the future.
• Google engineers are actively conducting a detailed analysis of the cooling system failure that triggered this incident.
• Google engineers will be conducting an audit of cooling system equipment and standards across the data centers that house Google Cloud globally.

Detailed Description of Impact

On Tuesday, 19 July 2022 from 06:33 to Wednesday, 20 July 2022 21:20 US/Pacific, some customers may have experienced high latency or errors in multiple Google Cloud services in the impacted location as detailed below:

Google Compute Engine: As data center temperatures started to increase, on Tuesday, 19 July 2022 at 08:06 US/Pacific Compute Engine terminated 42% of the Preemptible VMs (PVMs) across the europe-west2 region to reduce thermal load in zone europe-west2-a and ensure space for zonal failover activities in zones europe-west2-b and europe-west2-c.

When we proceeded to power down the impacted data center to mitigate the cooling overload on Tuesday, 19 July 2022 at 10:07 US/Pacific, Compute Engine terminated all VMs in the impacted data center, representing approximately 35% of the VMs in the europe-west2-a zone. We were able to re-enable PVM launches in the running europe-west2-b and europe-west2-c zones at 13:50 US/Pacific.

Power was restored on 19 July 2022 at 14:13 US/Pacific, at which point we began the recovery process for Compute Engine. To ensure a safe restoration, the team carefully sequenced the startup of the services powering Compute Engine. This process completed and the majority of VMs came back online starting at 20:18 US/Pacific. We enabled PVMs once the majority of the other VMs were running at 21:27 US/Pacific. The total impact duration was 12 hours, 12 minutes.

A small number of VMs (approximately 0.6% of the europe-west2-a zone) encountered conflicts in our control plane state, requiring a manual reconciliation process. We completed this reconciliation for the “long tail” of all VMs on Thursday, 21 July 2022 at 02:32 US/Pacific. A small number of control plane requests to delete missing VMs during the incident required manual resolution, which was completed at 15:50 US/Pacific.

Persistent Disk (PD): Approximately 38% of Persistent Disk volumes in zone europe-west2-a were unavailable from Tuesday, 19 July 2022 09:13 US/Pacific. Affected customers would observe unresponsive disks or I/O errors. In most cases, the GCE instances using these volumes were terminated shortly afterward, but about 1% of the unavailable Persistent Disk volumes in zone europe-west2-a were attached to instances that remained online throughout the incident. Approximately 96% of Persistent Disk volumes recovered automatically by 20:30 US/Pacific, but the remainder required additional work to recover, which completed on Wednesday, 20 July 2022 03:10 US/Pacific.

Additionally, about 11% of Regional Persistent Disk volumes in the europe-west2 region experienced high disk latency at the beginning of the incident. Most of the Regional Persistent Disk volumes successfully detected the fault in zone europe-west2-a and switched to unreplicated mode, but about 8% of Regional Persistent Disk volumes in the europe-west2 region were unable to correctly detect the fault until the Persistent Disk team forced them into unreplicated mode, which completed around 11:50 US/Pacific. About 17% of Regional Persistent Disk customers in the europe-west2 region experienced errors performing control plane operations, including creating, deleting, attaching, detaching, and snapshotting Regional Persistent Disk volumes from 19 July 2022 07:18 US/Pacific to 13:40 US/Pacific as an unintended side effect of mitigation efforts.

As a result of mitigation efforts, 38% of customers were unable to create new Persistent Disk volumes in zone europe-west2-a from Tuesday, 19 July 2022 11:16 US/Pacific to 20 July 2022 02:56 US/Pacific. Additionally, 48% of customers were unable to create new Persistent Disk volumes in zones europe-west2-b and europe-west2-c from Tuesday, 19 July 2022 10:23 US/Pacific to 11:36 US/Pacific.

The Persistent Disk snapshot service was unavailable for 38% of customers in zone europe-west2-a from Tuesday, 19 July 2022 from 08:11 US/Pacific to 21:29 US/Pacific. During this time, affected customers could not create new Persistent Disk snapshots from disks located in zone europe-west2-a nor restore snapshots and disk images to disks in zone europe-west2-a. The total impact duration was 12 hours, 16 minutes.

Google Cloud Storage: On Tuesday, 19 July 2022 between 07:18 and 08:54 US/Pacific, ReadObject availability for buckets located in europe-west2 dropped to approximately 86%, and the entire region’s availability to 96%. This impacted 24% of customer projects within this region. Customers would have received HTTP 500s when reading previously written data. All other operations, as well as read for new ingress, or data that was outside of the affected zones were not impacted. The total impact duration was 96 minutes.

Google Cloud Storage (GCS) stores replicas in at least 2 independent colossus clusters. As a regional (not zonal) product, GCS leverages placement algorithms to ensure locations, network, and data center diversity when selecting colossus clusters. There was a legacy placement algorithm isolated to just the europe-west2 region allowing for three colossus clusters in the region to be impacted by the same underlying issue relating to a single data center. In a small number of cases GCS had replicated regional data residing in two offline clusters, and subsequently some customers were unable to access and read some of their existing regional scoped data when the regional clusters were taken offline. Writes and other operations were not impacted.

API Gateway: ~47% of projects with traffic experienced an elevated number of spikes in 5xx responses in europe-west2 on Tuesday, 19 July 2022 from 07:20 to 14:00 US/Pacific. Affected customers observed 5xx responses from their API Gateways. The total impact duration was 6 hours, 40 minutes.

Cloud Bigtable: ~70% of unreplicated Bigtable instances in europe-west2-a experienced 100% data plane unavailability from Tuesday, 19 July 2022 07:05 to Wednesday, 20 July 2022 02:20 US/Pacific. We observed failed control plane operations for 11% of instances that contained a replica in the europe-west2-a zone. Customers with replicated bigtables in europe-west2-a/b/c using Multi-Cluster routing may have seen increased latencies due to traffic failing over to other regions. The total impact duration was 19 hours, 15 minutes.

Cloud Composer: Cloud Composer environments (data-plane side) and operations (control-plane side) experienced degraded performance in europe-west2 on Tuesday, 19 July 2022 from 09:00 to 22:30 US/Pacific. Control plane operations experienced high failure rates (failure rate reached 100% at the peak). The number of Composer environments reporting a normal, operational state dropped by ~37%. Total capacity of all environments in the region (measured by the total number of tasks being executed in the whole region) was significantly reduced and the drop reached ~50% at the peak. The total impact duration was 13 hours, 30 minutes.

Cloud Data Fusion: ~20% of Data Fusion Instances in europe-west2 experienced service degradation, ranging from logs and metrics not being updated to complete loss of availability of their instance on Tuesday, 19 July 2022 10:15 to 21:30 US/Pacific. ~5% of instances were usable during the impact duration. The total impact duration was 11 hours, 15 minutes.

Cloud Datastore: Customers may have experienced timeouts and degraded service on approximately 1% of writes in the europe-west2 Datastore instance. The total impact duration was 9 hours, 25 minutes, on Tuesday, 19 July 2022 from approximately 12:30 to 19:30 US/Pacific. Approximately 40% of customers experienced significant latency increase on query operations from Tuesday, 19 July 2022 09:45 to 12:45 US/Pacific.

Cloud Dataproc: On Tuesday, 19 July 2022, from 08:00 US/Pacific to 22:00 US/Pacific, customers experienced increased error rates in creating and scaling up clusters in europe-west2. About 24% of CREATE operations and 13% of UPDATE operations were affected. Due to a backlog of queued requests requiring manual attention, error rates for old CREATE/DELETE requests remained elevated until Thursday, 21 July 2022 16:45 US/Pacific (this did not impact availability of already-running clusters). Some Dataproc clusters that were allocated in the impacted zone were unavailable during power down. Most of these came up by 19 July 2022 22:00 US/Pacific.

Cloud Firestore: Firestore streaming (listen, write) requests via Webchannel were 100% unavailable in the europe-west2 instance on Tuesday, 19 July 2022 from 07:15 to 09:55 US/Pacific. The total impact duration was 2 hours, 40 minutes.

Cloud Secret Manager: Cloud Secret Manager experienced an outage for secrets that were exclusively stored in europe-west2 from Tuesday, 19 July 2022 07:29 to 08:46 US/Pacific. Secret Manager is a global service that lets users define in which regions to store a given secret. The regional service instances for europe-west2 were deployed in the impacted data center. The total impact duration was 1 hour, 17 minutes.

Cloud Spanner: A more comprehensive investigation into Cloud Spanner logs indicated no evidence of any customer impact.

Cloud SQL: Customers experienced downtime on Tuesday, 19 July 2022 starting at 09:25 US/Pacific. 36% of zonal (non-HA) instances in europe-west2-a were affected. Additionally, 31% of regional (HA) instances whose primaries were located in europe-west2-a experienced extended downtime because they were unable to successfully fail over to another zone. Finally, customers experienced some failures during the incident for backup, instance creation, update, delete, restart, export, and Database Migration Service operations. The total impact duration was 17 hours, 30 minutes.

Dataflow: Approximately 8% of Dataflow streaming jobs running in europe-west2 were stuck from Tuesday, 19 July 2022 07:11 to 12:38 US/Pacific. There was limited impact to batch Dataflow jobs. Some new Dataflow jobs could not be initiated. The total impact duration was 5 hours, 27 minutes.

Datastream: Datastream streams experienced errors and processing lag in europe-west2-a as a result of impact on the data-plane infrastructure and services from Tuesday, 19 July 2022 09:05 to Wednesday, 20 July 2022 03:00 US/Pacific. Customers were advised to run their streams in another region. The total impact duration was 17 hours, 55 minutes.

Google App Engine, Cloud Functions, and Cloud Run: Customers may have experienced high error rates for Google App Engine, Cloud Functions and Cloud Run in europe-west2. Customers with a multi-region architecture could failover to another region. Traffic for ~72% of projects from Cloud Tasks, Cloud Scheduler, Eventarc, Cloud Pubsub to App Engine and Cloud Functions experienced up to 18% dropped requests/events on Tuesday, 19 July 2022 07:18 to 10:05 US/Pacific.