Google Cloud 是不是挂了？

## Incident Report ### **Summary** On Monday, 19 Oct 2026 (Tuesday, 20 Oct BST), a data center facility that hosts europe-west2-b, experienced an external electrical power disruption. This resulted in service unavailability for workloads located in this zone. As traffic was redirected to alternative cloud regions or unaffected zones within europe-west2 cloud region, these workloads experienced transient latency and elevated errors affecting services across europe-west2.  The primary impact was mitigated after 6 hours 8 minutes. ### **Root Cause** **Facility architecture and background:** The data center facility in the UK region hosts Cloud clusters serving the europe-west2-b zone, which form part of the multi-zonal europe-west2 Cloud region. The site relies on a 2N utility power architecture—meaning the facility has complete duplicate power paths for every critical load—backed by a central high-voltage generator bank managed by a dual-redundant Programmable Logic Controller (PLC). Primary utility power is supplied to the facility via two independent feeders (A and B), supplemented by a standby utility connection configured for manual campus-wide transfer during maintenance operations or emergency events. The two high-voltage primary feeders terminate at separate primary intake switchboards capable of closed-loop operation if either board is taken offline. Switching between the primary and standby utility providers utilizes open transitions and key interlocks to prevent parallel connections across sources. Static Uninterruptible Power Supply (UPS) battery systems provide 5 minutes of autonomous runtime. This bridge power ensures uninterrupted operation while the PLC detects utility undervoltage events, initiates diesel generator startup, synchronizes the generators, and transfers the site load. Prior to the utility disruption, the facility had been placed in a temporary single-feed (1N) bridged maintenance posture on the primary utility. Circuit breaker feeder B was physically opened and key-interlocked (Locked Out Tagged Out) to allow for corrective repairs following issues identified during a quarterly generator load test on 14 Oct 2026. Consequently, primary utility feed A was supporting the total critical load while repairs were conducted on the B feeder pathway. **Root Cause:** On Monday, 19 Oct 2026 at 22:53 US/Pacific (Tuesday, 20 Oct 06:53 BST), the primary utility provider experienced an outage on primary utility feed A, caused by an external underground cable fault on the utility network outside the data center facility.   When the primary utility feed A was lost, the facility’s generator plant failed to receive a signal to automatically start and take critical load due to a design voltage-sensing and misconfiguration in the PLCs. The facility's static UPS battery banks carried the critical load as designed until their battery reserves depleted resulting in a complete power loss across data halls. The investigation of the PLC configuration identified the root cause as a misconfiguration of the PLC logic that did not verify both an undervoltage condition and physical breaker position for both utility feeders. Instead, the PLC undervoltage sensing inputs were wired upstream (line side) of the isolated utility feed B breaker. The PLC was programmed to verify whether voltage was present on the feeder before commanding a generator start. As the utility feed B remained energized upstream, both PLCs sensed live upstream voltage on the disconnected path. As a result, the PLC logic indicated that a healthy utility source was available, even though the primary and secondary distribution feeds were de-energized, and did not issue a start command to the generator plant.; The onsite facilities team attempted to execute a remote load transfer via the PLC; however, the system inhibited the action. Recognizing that the UPS battery reserves would be exhausted before further remote operations could be resolved, the team transferred the facility load to the standby utility source, restoring the critical load that was lost once batteries depleted during this process. Once utility power was restored via the standby utility source, and UPS batteries started recharging, the onsite facility teams and OEM vendors reconfigured and validated PLC functionality on both undervoltage and breaker status. After the power loss was initially detected, Google engineers began work to mitigate failover failures and prepare service recovery. Upon restoration of the utility source, UPS and generator capabilities, they began the process to re-enable services in europe-west2-b. ### Remediation and Prevention The sequence of events leading to customer impact was as follows: * **19 Oct 22:53 US/Pacific (20 Oct BST 06:53 BST) -** An electrical power disruption occurred at the data center facility hosting availability zone europe-west2-b, and a PLC (Programmable Logic Controller) startup fault prevented facility emergency generators from assuming the electrical load. Facility load continued to be supported by UPS battery banks until the batteries were depleted. * **19 Oct 23:00 US/Pacific (20 Oct 07:00 BST)** **Impact start -** As battery backups were exhausted, physical host servers, local storage infrastructure, and network aggregation hardware in availability zone europe-west2-b lost power, causing instances, database replicas, and worker nodes in that zone to become immediately unreachable. * **19 Oct 23:04 US/Pacific (20 Oct 07:04 BST) -** Google engineers were alerted to the disruption via automated monitoring systems and probers, which detected elevated API error ratios, job submission timeouts, and loss of infrastructure reachability in zone europe-west2-b. Engineering teams immediately mobilized to isolate the affected infrastructure and commence recovery. * **19 Oct 23:09 US/Pacific (20 Oct 07:09 BST) -** Regional traffic-routing mechanisms and client applications initiated automated failover toward available infrastructure in adjacent availability zones within the region. * Cross-zone remote procedure calls from regional service control planes directed to metadata and catalog components in the offline zone encountered timeouts, causing initial job submission stalls and API latency spikes. * The sudden influx of redirected query executions, high-availability database failovers, and compute workloads saturated available compute slots and processing capacity in other availability zones, creating elevated admission queues, delayed database failover completions, and intermittent 500 and 503 API errors across the region. * **Traffic Isolation:** Google engineers isolated the offline availability zone (europe-west2-b) by adjusting regional service routing away from the affected infrastructure, preventing cross-zone client requests from timing out against offline endpoints. * **Workload Reprioritization and Resource Safeguards:** Engineers adjusted scheduler allocations and capacity limits across unimpacted infrastructure to prioritize active customer reservations, reduce resource contention on execution slots, and relieve queue backlogs for regional workloads. * **19 Oct 23:54 US/Pacific (20 Oct 07:54 BST) Utility Power Restoration** **\-** Power supply from the standby utility source was restored and power restoration to the data hall began. * **20 Oct 00:20 US/Pacific (08:20 BST) Datahall Power Restoration** **\-** Utility power was re-established to the datahalls. Mains utility power normalized; power feed to data hall verified. Following this, the site operations and engineering teams focused on and completed a reset and tested the emergency backup generator automation controllers to restore facility power. * **20 Oct 00:40 US/Pacific (08:40 BST)** **\-** Clusters began to recover, cluster manager systems started rescheduling compute jobs.  * **20 Oct 00:45 US/Pacific (08:45 BST) \-** Network devices were back online.  * **20 Oct 00:50 US/Pacific (08:50 BST) \-** Due to the immediacy of power loss, storage devices with unclean shutdowns began recovery. Responders recognized this and chose not to restore services that would otherwise make this worse. Because of the large volume of storage devices involved, this process required a substantial period to finish prior to restoring service to customers. * **20 Oct 03:13 US/Pacific (11:06 BST)** **\-** Power redundancy was verified and the storage infrastructure was sufficiently restored to prepare the initiation of controlled traffic re-admission. The remaining storage systems continued recovery operations. * **20 Oct 03:45 US/Pacific (11:45 BST) Controlled Traffic Re-admission** **\-** Once system health and storage integrity were verified, engineers executed gradual, controlled traffic re-admission back to zone europe-west2-b allowing accumulated job backlogs to clear smoothly. * **20 Oct 04:11 US/Pacific (12:11 BST)** **\-** Google Cloud Platform services and APIs were made fully available to customers. Some Google Compute Engine virtual machines were still not yet available. * **20 Oct 04:29 US/Pacific (12:29 BST) \-** Regional services and backend compute management systems were fully undrained, marking the end of core platform-level disruption. * **20 Oct 05:08 US/Pacific (13:08 BST) Majority Workload Recovery** **\-** GCE virtual machines were 99.5% operational and healthy, and creating new VMs was fully supported. Service availability was restored for most customer workloads, with resolution confirmed for the majority of users across Google Cloud Platform services in the region. Following the recovery of a small tail of remaining isolated workloads, the closure notification for the incident was posted by 07:03 US/Pacific (15:03 BST). Google is committed to preventing a repeat of this issue in the future and is completing the following actions: **Data Center Action Items:** * Restoration of normal utility configuration (both A & B feeds) * Utility provider to locate the damaged cable fault, using low-frequency radar penetration testing procedures. **\[Completed 26 Oct\]** * Utility provider to send on-site engineers to repair the cable fault and restore the impacted primary utility feed A source. **\[Estimated Completion Date 30 Oct\]** * Onsite Operations teams with a controlled method of procedure (MOP) to transfer the facility to its primary utility source (feeds A and B) from standby utility feed through coordinated switching with the utility provider **\[Estimated Completion Date 31 Oct\]** * Generator confidence testing * Full generator load transfer tests with utility power cutover of the active utility feed. Test automatic startup of generators through PLC signals within the backup battery timer and detailed review of backup procedures **\[Completed on 20 Oct and 26 Oct (full critical load transfer test including updated EOPs)\]** * Full generator load transfer tests in coordination with the primary utility source feeds (A and B) after restoration work completed by the primary utility provider. **\[Estimated Completion Date 31 Oct\]** * PLC configuration fix  * Reconfigured and validated the PLC after the outage on 20 Oct and then again on 26 Oct **\[Completed 26 Oct\]** * Google engineers to work in collaboration with the facility vendor teams to perform a comprehensive PLC code review and system resiliency to multiple failure scenarios. **\[Estimated Completion Date (prior to generator transfer test) estimated 31 Oct\]** * Validate PLC telemetry once primary utility feed A and B lineup is restored. **\[Estimated Completion Date 31 Oct\]** * Validate global PLC configurations at all global sites with similar topologies and PLC manufacturers, to ensure confidence in automatic generator starts. **\[Estimated Completion Date 15 Nov\]** * Emergency Operating Procedures (EOP) documentation update: * Comprehensive update to the EOP for site’s response to unexpected loss of active utility source. Update includes \- manual steps to attempt to restore active utility feed, manual restart of generators and manual transfer to other utility sources within the UPS battery run time. **\[Completed 26 Oct\]** * Ensure updated EOP is practiced twice per shift by on-site facility team. **\[Completed 24 Oct and checked with Google DC Operations daily\]** * Strengthen change management:  * Freeze planned maintenance activities at other sites with similar topologies, and implement elevated approvals and reviews for data center configuration changes which are outside their standard designs. **\[Estimated Completion Date 30 Oct\]** **Cloud Service Action Items:** * **Communications Enhancement:** Bolster notification speed by developing an automated alert-to-communication mechanism for location-impacting incidents. * **Platform Testing:** Expand platform-wide testing outcomes for infrastructure-level events. ### **Detailed Description of Impact** Between Monday, 19 Oct 2026, 23:00 US/Pacific (Tuesday, 20 Oct 2026, 07:00 BST) and Tuesday, 20 Oct 2026, 05:08 US/Pacific (13:08 BST), customers utilizing Google Cloud Platform services in the europe-west2 region experienced service disruptions across two consecutive phases, lasting a total duration of 6 hours, 08 minutes. During the initial phase from 23:00 to 04:29 US/Pacific (07:00 to 12:29 BST) (duration 5 hours, 29 minutes), customers experienced unavailability of zone-scoped resources in products, including: * **Compute Engine** \- VM instances, zonal MIGs, unmanaged instance groups, zonal NEGs, sole-tenant node groups, zonal reservations, target instances, zonal API operations.  * **Persistent Disk / Hyperdisk** \- Zonal PD, zonal Hyperdisk, Hyperdisk storage pools, instant snapshots. * **GKE \-** Zonal clusters (control plane), node pool nodes in \-b, PVs on zonal disks. * **Cloud SQL \-** Non-HA instances, read replicas in \-b. * **AlloyDB \-** Basic (non-HA) primaries in \-b. * **Bigtable \-** Single-cluster instances in \-b. * **Other products with affected zonal resources:** Cloud Batch, Cloud Build, Cloud Storage, Cloud Workstations, Composer, Dataproc, Filestore, Managed Lustre, Migrate to VMs, NetApp Volumes, Pub/Sub Lite, Vertex AI Workbench, VMware Engine. The unavailability of those resources scoped to a zone that undergoes disruption is working within the design of a zonally-scoped system. Requests made to these resulted either in a timeout or error code during the disruption. Any workloads and resources hosted entirely within europe-west2-b became unreachable due to this event. Google Compute Engine's zonal resources such as VMs require customers to provision failover or cross-zone recovery resources if needed, and to set up a recovery mechanism that matches their needs. Some failovers were unsuccessful due to capacity constraints and difficulty in replicating the underlying storage mechanism.  Following platform service restoration, some customer workloads running on Compute Engine instances as well as products that rely on Compute Engine experienced a longer recovery. By 04:46 US/Pacific (12:46 BST) new Virtual Machine creation resumed and by 05:08 US/Pacific (13:08 BST) over 99% of Compute Engine Virtual Machines were in a healthy state. Any remaining customers that were experiencing Virtual Machine problems were directed to file a customer support case as per normal procedure. While the physical power outage was isolated strictly to data center facilities in zone europe-west2-b, some customers running workloads in unimpacted zones (europe-west2-a and europe-west2-c) or across regional architectures also experienced unexpected impact. The primary causes of impact outside of zone europe-west2-b were: 1\) Instantaneous demand surges on neighboring zones (a and c) and 2\) Control-plane dependencies across zonal boundaries Affected products included: AlloyDB, Apigee Infra and Apigee Edge, BigQuery, Cloud Dataflow, Cloud Pub/Sub, Cloud SQL, Google Compute Engine, Google Kubernetes Engine, Looker, Regional Persistent Disk, Vertex AI Prediction, and Cloud Speech-to-Text. These services experienced unexpected degradation and failover issues driven by sudden compute capacity exhaustion from surging demand, and control-plane API timeouts. Customers would have experienced the inability to create, update, or delete some regional resources during this period. This was not a full regional control-plane outage, but certainly a degraded state. Multi-zonal and regional services that spanned across the europe-west2 region experienced elevated latency, transient API errors (such as HTTP 500 and 503 errors), and task admission queuing as failover traffic concentrated onto unimpacted regional resources and cross-zone calls to offline components timed out.  Storage write operations in the region experienced a transient error rate during the initial failover period, and regional query admission queues temporarily peaked before rapidly subsiding as capacity re-balanced. We did not identify any impact of the outage outside of the europe-west2 region. The following Google Cloud Platform services experienced impact within europe-west2: * AlloyDB for PostgreSQL * Apigee * BigQuery Data Transfer Service * Cloud Data Fusion * Cloud Filestore * Cloud Firestore * Cloud Load Balancing * Cloud Memorystore * Cloud NAT * Cloud Run * Cloud Spanner * Cloud Workstations * Datastream * Dialogflow CX * Google App Engine * Google BigQuery * Google Cloud Bigtable * Google Cloud Dataflow * Google Cloud Dataproc * Google Cloud NetApp Volumes * Google Cloud Pub/Sub * Google Cloud SQL * Google Cloud Storage (GCS) * Google Compute Engine (GCE) * Google Kubernetes Engine (GKE) * Google SecOps (Security Information and Event Management \[SIEM\] and Security Orchestration, Automation, and Response \[SOAR\]) * Looker (Google Cloud core) * Looker Studio * Managed Airflow (Google Cloud Composer) * Managed Service for Apache Kafka * Memorystore for Memcached * Memorystore for Redis * Memorystore for Redis Cluster * Memorystore for Valkey * Network Connectivity Center * Persistent Disk * Private Service Connect * Speech-to-Text * Virtual Private Cloud (VPC) * VMWare engine ----------

## Incident Report ## Date/Time of the Issue (All time US/Pacific) **Incident First Occurrence** Incident Start: 08 October 2026 04:00 Incident End: 08 October 2026 16:30 Duration: 12 hours, 30 minutes **Incident Recurrence** Recurrence Start: 09 October 2026 04:05 Recurrence End: 09 October 2026 09:46 Duration: 5 hours, 41 minutes **Total Duration:** 18 hours, 11 minutes ## Summary On Thursday, 08 October 2026, Google Cloud Storage experienced elevated write latency and write error rates while performing upload or write operations in the us-central1 region for a duration of 12 hours, 30 minutes. This was followed by a recurrence on Friday, 09 October 2026 in the us-central1-b zone for a duration of 5 hours, 41 minutes. We are taking immediate steps to ensure this doesn’t happen again. ## Root Cause Google Cloud Storage relies on a highly distributed backend architecture to process and store incoming data. During the incident window, specific storage backend nodes within the us-central1 region experienced a localized capacity shortfall due to increased traffic. The sequence of events leading to customer impact was as follows: * A traffic sorting optimization configuration update to the Google Cloud Storage load balancing layer intended to address localized capacity constraints caused severe resource exhaustion on specific storage backend nodes in the affected region by inadvertently bypassing the standard capacity-aware routing logic, resulting in a severe load imbalance. * Consequently, a small fraction of storage backend nodes in the us-central1 region received an overwhelming proportion of incoming write traffic despite having low available serving capacity. This concentration of traffic led to resource exhaustion on the affected backend servers causing backend services handling object data ingestion to begin timing out. * Because the underlying instances could not process the high volume of requests quickly enough, the incoming upload and write requests routed to these nodes experienced elevated latency and failed to complete, resulting in errors surfaced to customers. * Concurrently, a separate issue with the underlying database service contributed to additional metadata processing errors during a portion of the incident window. * The recurrence of the issue on October 9 was caused by an incomplete rollback of an upload planner feature, which broke the load balancing mechanism and overloaded a few smaller storage backend nodes. ## Remediation and Prevention The issue was initially detected on 08 October 2026 at 04:50 by Google engineering teams actively monitoring system telemetry early in the incident window.  To mitigate the issue, Google engineers executed the following step-by-step strategy: * Engineers initially attempted to dynamically allocate additional emergency capacity to the affected region, but this was blocked by systemic regional capacity limits. * To relieve immediate pressure, engineers selectively drained traffic from the most overloaded storage backend nodes. * A configuration change was deployed to disable the traffic sorting optimization, significantly reducing resource overhead on the storage backend. * For the concurrent metadata issue, engineers rolled back a recent update to the underlying metadata database service. * While these steps provided temporary relief, a recurrence of the load imbalance required further traffic redirection and draining of affected storage components. The incident was permanently resolved when engineering fully reverted the problematic load balancing configuration change, restoring the previous capacity-aware routing behavior and allowing traffic to distribute evenly across the region. Once traffic levels stabilized and resource utilization returned to normal baseline levels, the previously drained nodes were restored to active service, successfully restoring normal operations. **Preventative Actions:** To ensure this issue does not recur, we are implementing the following improvements: - **Safe Configuration Rollouts:** We are updating our deployment protocols to ensure that load balancing logic changes are rigorously validated in alternative, smaller regions before being rolled out to larger, highly utilized regions. - **Intelligent Operational Controls:** We are enhancing the storage load balancing architecture to strictly enforce capacity constraints, ensuring that traffic distribution logic cannot bypass pre-spilling safeguards and route traffic to clusters lacking sufficient serving resources. - **Resilient Scaling Logic:** We are implementing targeted improvements to accelerate internal data movement within highly utilized storage cells, enabling the system to process data more efficiently and handle temporary load spikes with greater resilience. ## Description of Impact On 08 October 2026, from 04:00 to 16:30 in us-central1 region and 09 October 2026, from 04:05 to 09:46 in us-central1-b zone, customers experienced elevated latency and error rates during Cloud Storage upload and write operations. * **Upload & Write Failures:** Customers primarily observed HTTP 503 UNAVAILABLE errors on WriteObject operations and client-side timeouts. * **Increased Write Latency:** Write operations that did succeed experienced significantly higher latencies than normal. * Cloud Storage operations in all other regions, as well as all other Google Cloud services, remained unaffected by this specific event.

## Addendum to Incident Report This addendum extends the Detailed Description of Impact from the previous message. Though only two clusters in a datacenter in us-central1-b and us-central1-f experienced network isolation, a few regional products with a dependency on the affected datacenter were also affected. Between 07:41 and 11:52 US/Pacific on Tuesday, September 1, 2026, the following products were affected: - **Compute Engine VMs**: VMs running in the affected data center lost network connectivity to resources located outside of their datacenter. From a Compute Engine perspective, this incident only affected a subset of VMs in us-central1-b and us-central1-f. - **Zonal and Regional Products Based on Compute Engine** Products with dependencies on VMs in the affected datacenter experienced symptoms like connection timeouts or service unavailability because the underlying VMs lost network connectivity. Examples of products that are based on Compute Engine VMs include: Apigee, Cloud SQL, AlloyDB for PostgreSQL, Dataflow, Cloud Filestore, and Looker. - **Google Kubernetes Engine (GKE)**: GKE clusters whose control plane had dependencies on Compute Engine VMs in the affected clusters experienced errors and timeouts when administrators or nodes attempted to connect to the Kubernetes API server. Consequently, deployments might have been delayed, Kubernetes API operations might have failed, and new nodes might not have registered with the control plane (causing node pool operational delays). - **Cloud Run and App Engine:** Workloads in the us-central1 region experienced elevated serving latency, 5xx aborted request errors, and outbound connectivity degradation. To prevent severe request failures, traffic was diverted to other clusters. Diverting requests to other clusters temporarily caused elevated latency or errors in those clusters.Once the impacted cluster was restored, workload demand for these new backends created a 'thundering herd' of concurrent instance initializations. This sudden surge bypassed warm caches and overwhelmed internal file-serving tiers, extending symptoms until emergency capacity was provisioned. - **Other Google APIs and Services**, in us-central1 or a multi-region that includes us-central1, were affected if they had dependencies on software tasks in the affected clusters and weren't able to use software tasks in an unaffected cluster. Symptoms included connection timeouts or service unavailability. Examples of other Google APIs and services are: Google Cloud Bigtable, BigQuery, Google SecOps SOAR, Cloud Spanner, and Cloud Firestore. For some services, symptoms were limited to temporary latency spikes and service unavailability until workloads shifted to unaffected clusters. - **Private Service Connect**: Clients (in any region) experienced connectivity issues involving PSC endpoints with dependencies in us-central1, in specific circumstances: - PSC endpoints for Google APIs were affected in the same way as noted in the Google APIs and services section. - Producer/consumer PSC endpoints were affected if the producer had load balancers or backend VMs in the affected clusters of us-central1. - **Hybrid Connectivity**: Some Cloud Router, Cloud VPN, and Cloud Interconnect VLAN attachments in us-central1 were affected in specific circumstances: - Cloud Router BGP software tasks for some Cloud Routers in us-central1 experienced a Cloud Router maintenance event if their BGP software tasks were running in the affected clusters. - Cloud VPN tunnel software tasks running in the affected clusters lost network connectivity. This caused some Cloud VPN tunnels in us-central1 to go down until a replacement tunnel task in a different cluster was assigned. - Resources (in any region) sending packets through Cloud Interconnect VLAN attachments in us-central1 might have experienced intermittent connectivity issues in certain situations where network flows weren’t already programmed. - **Envoy-based load balancers and other products**: Load balancers and other products that depended on managed Envoy tasks of a proxy-only subnet in us-central1 experienced increased latency or connectivity issues if enough Envoy tasks were in the affected clusters. The following products were affected: - Regional external application load balancers in us-central1 - Regional internal application load balancers in us-central1 - Cross-region internal application load balancers with proxies in us-central1 - Regional external proxy network load balancers in us-central1 - Regional internal proxy network load balancers in us-central1 - Cross-region internal proxy network load balancers with proxies in us-central1 - Secure Web Proxy in us-central1

# Incident Report ## Summary On Thursday, 20 August 2026, from 08:00 to 10:22 US/Pacific (15:00 to 17:22 UTC), multiple Google Cloud services in the us-west1 region experienced elevated latency, provisioning failures, increased error rates, and widespread service degradations for a total duration of 2 hours and 22 minutes. The incident impacted a wide range of core services — including Persistent Disk, Google Kubernetes Engine, Google Compute Engine, Cloud Run, Google Cloud Bigtable, Cloud Storage, and Identity and Access Management — across both control plane operations and data plane requests. We sincerely apologize for the disruption this incident caused to your business operations and critical workloads. We recognize the vital role Google Cloud plays in supporting your organization, and we deeply regret the impact on your business operations and critical workloads. Engineering and infrastructure teams are actively implementing measures to address the root causes and strengthen network resiliency to prevent recurrences in the future. Specifically, teams are monitoring recovery progress, refining safety checks for planned optical maintenance, and optimizing regional traffic routing mechanisms to safeguard against unexpected capacity constraints. ## Root Cause The disruption originated during scheduled fiber optic maintenance, which unexpectedly compromised network capacity between data centers within the us-west1 region. Automated rerouting mechanisms failed to properly redistribute traffic to alternate capacity, resulting in network congestion as volumes exceeded available bandwidth in the affected area. This underlying network degradation subsequently impacted higher-level service components through severe packet loss, request throttling, and increased latency across inter-campus dependencies. Core infrastructure services, including Spanner Paxos consensus and the Unified Metadata Server (UMS), experienced significant latency spikes, which cascaded into timeouts and elevated error rates for downstream dependent products such as Cloud Storage, Cloud IAM, Persistent Disk, and Google Kubernetes Engine. Consequently, both control plane operations and data plane requests failed to execute successfully across multiple Google Cloud services in us-west1 throughout the duration of the incident. ## Remediation and Prevention Internal monitoring systems initially detected widespread service anomalies and alerted Google engineers, who promptly confirmed that multiple core services operating across the us-west1 region were severely impacted. In response to the immediate operational risks, engineering teams promptly executed emergency traffic draining protocols to reroute active service workloads away from the compromised inter-campus network infrastructure and minimize further customer disruption. Once teams restored inter-campus fiber network capacity, engineers systematically reintroduced production traffic back to us-west1 through controlled validation stages, ultimately confirming full service normalization across all impacted platforms. Longer term engineering remediations and architectural enhancements are actively being finalized and assigned to owning teams. Key focus areas include reforming scheduled maintenance protocols, establishing stricter safety checks and circuit redundancy requirements prior to routine maintenance, and refining automated regional traffic failover mechanisms to automatically handle sudden capacity degradation without incurring severe network congestion. ## Detailed Description of Impact On Thursday, August 20, between 08:00 and 10:22 US/Pacific, Google Cloud customers in the us-west1 region encountered elevated latency, provisioning failures, and increased error rates. * **Impact / Error Rates:** The reduced capacity caused request throttling, latency spikes, and increased retry volume. * **Scope Exclusion:** Services and workloads operating in regions other than us-west1 remained fully operational and unaffected. **Affected Services and Features** The following services experienced elevated latencies and/or increased error rates, across their respective data planes and control planes: * AlloyDB * Apache Kafka * Apigee Edge Public Cloud * Apigee X * Artifact Registry * BigQuery & BigQuery Data Transfer Service * Cloud Build * Cloud Data Fusion * Cloud Dataflow * Cloud Filestore * Cloud Key Management Service (KMS) * Cloud Monitoring * Cloud Run * Cloud SQL * Contact Center AI Platform * Dataproc Metastore * Google App Engine * Google Cloud Bigtable * Google Cloud Pub/Sub * Google Cloud Storage (GCS) * Google Compute Engine (GCE) * Google Kubernetes Engine (GKE) * Identity and Access Management (IAM) * Managed Airflow (Cloud Composer) * Managed Service for Apache Spark (Dataproc) * Persistent Disk The regional incident in us-west1 followed a structured three-phase recovery dictated by platform dependency layers: While core platform connectivity and live request serving were restored by 10:22 US/Pacific, certain services took additional time to fully recover due to asynchronous backlog processing and localized control-plane state reconciliation for a very small set of customers. For event-driven and pipeline services, inbound error rates dropped to 0 immediately, but some operations experienced elevated latency while workers processed through backlogs accumulated during the outage, clearing later for Cloud Pub/Sub, Cloud Build & Deploy, Cloud Dataflow, and Cloud Storage lifecycle deletions. Concurrently, while primary read/write traffic was healthy across the region, specific long-running lifecycle workflows required extra time to clear locks and reconcile distributed state machines. Compute Engine VM provisioning in zone us-west1-c and Cloud Filestore control plane instance allocation locks and resource validation checks took longer to normalize across regional storage backends.

## \# Incident Report ## \#\# Summary On Wednesday, 15 July 2026, Google Cloud VMware Engine (GCVE), Bare Metal Solution (BMS), and Google Cloud NetApp Volumes (GCNV) experienced service interruptions for a total duration of 14 hours, 55 minutes. The root cause for this outage is a 3ms voltage drop in the power feed from the utility provider and subsequent utility breaker protective action. This incident was mitigated by the data center provider rectifying the failed systems followed by the Google engineering team restoring the services. ## \#\# Root Cause A regional data center hosting services in europe-west4-a experienced an upstream voltage transient affecting both utility power feeds A and B. During this event, the utility breakers on both A and B feeds tripped and initiated the transfer to the back-up power source DRUPS (Diesel Rotary Uninterruptible Power Supply). The transition of side B to DRUPS system was successful without any power interruption. The DRUPS back-up power system for side A failed to take over the facility load due to electrical component failures. The DRUPS failure to take over load initiated automatic transfer of the affected 3 rows from feed A to redundant power feed B. Rows 1 and 2 transferred to power feed B successfully. Row 3 failed to transfer to power feed B and experienced complete power loss due to an overload protection breaker trip. The root cause of the failed transfer on the affected row was identified as load deployment discrepancy which is under further investigation by Google engineering team. This resulted in loss of both redundant power feeds to the single row. During the voltage transient event, the server data hall experienced an increased temperature due to a cooling system failure. The chiller controller dropped offline during the voltage transient event, failing to signal the chilled water distribution pumps to restart and ultimately causing the chiller system A to shut down. The redundant source was not available due to known ongoing construction work at the facility. This resulted in the data hall temperatures reaching 44°C and subsequent shutdown of the affected data hall machines. Google engineering teams initiated machine shutdown procedures for the remainder of reachable devices as part of the cooling emergency shutdown process. A timeline of events during the incident is provided below. ### DataCenter Events | Timestamp(PST) | Event Description | | :---- | :---- | | 07-15-2026 16:24 | An electrical fault occurred on the utility grid upstream of the data center, disrupting the electrical distribution. | | 07-15-2026 16:24 | DRUPS back-up power system for side A failed to take over the data hall load. | | 07-15-2026 16:24 | The chiller controller dropped offline during the voltage transient event causing distribution pumps to stop and unable to restart. | | 07-15-2026 16:37 | Rows 1 and 2 transferred to power feed B successfully. Row 3 failed to transfer to power feed B and lost power. | | 07-15-2026 18:29 | Data hall temperatures reached 44C and crossed the safe machine operating threshold. | | 07-15-2026 19:55 | Machine shutdown procedures implemented. | | 07-15-2026 21:05 | Cooling system fully recovered and temperatures in the data hall returned to normal operating range. | | 07-15-2026 21:24 | Notification about cooling recovery sent by provider | | 07-15-2026 21:46 | Notification about power recovery sent by provider | ### GCVE Timelines | Timestamp(PST) | Event Description | | :---- | :---- | | 07-15-2026 16:30 | GCVE service power monitoring alert received | | 07-15-2026 16:41 | First symptom detected, servers reporting redundancy power feed failure. | | 07-15-2026 17:05 | Switch temperature \>60°C alerts triggered. | | 07-15-2026 17:09 | Start of user impact, first prober alert failure for north south traffic for customers. | | 07-15-2026 19:00 | Server / Network devices shutdown initiated. | | 07-15-2026 19:55 | All reachable server/network devices were shut down. | | 07-15-2026 21:24 | Cooling system fully recovered and temperatures in the datahall were returning to normal operating range. | | 07-15-2026 21:56 | Network restoration started with reachable hydra devices. | | 07-15-2026 22:40 | Onsite technician arrived, to recover the console servers. | | 07-15-2026 23:10 | Console server reboot/recovery complete. | | 07-16-2026 00:33 | Network recovery for all Placement Groups complete. | | 07-16-2026 02:36 | Incident mitigated, after recovering all the customer PCs are fully healthy. | This outage impacted 24 private clouds of 20 distinct customers in europe-west4-a region. ### Bare Metal Solutions (BMS) Timelines | Timestamp(PST) | Event Description | | :---- | :---- | | 07-15-2026 16:41 | First symptom detected, servers and storage reporting redundancy power feed failure | | 07-15-2026 17:52 | Automated monitoring detected rising temperatures | | 07-15-2026 18:37 | Four Netapp SAN storage nodes failed and were shut down due to overheating, resulting in storage availability issue | | 07-15-2026 18:37 | Customer impact started, first server shutdown detected | | 07-15-2026 18:59 | Reserve and buffer servers started to get powered off | | 07-15-2026 20:43 | Drop in the temperature has been detected | | 07-15-2026 20:52 | Four Netapp SAN storage nodes recovered, mitigating storage availability issue | | 07-15-2026 21:05 | The cooling system fully recovered and temperatures in the datahall were returning to normal operating range. | | 07-15-2026 23:23 | First customer server reboot started | | 07-16-2026 06:32 | Last customer server reboot started | | 07-16-2026 08:42 | Incident mitigated. All impacted customer servers were rebooted and confirmed in healthy state. | This outage impacted 9 distinct BMS customers in europe-west4 ### NetApp Timelines | Timestamp(PST) | Event Description | | :---- | :---- | | 07-15-2026 16:32 | Netapp facility remote monitoring alert on high temp and network switches switches failure received | | 07-15-2026 17:32 | All 6 clusters failed and were shut down automatically due to high temperature | | 07-15-2026 21:05 | Cooling capacity restored | | 07-15-2026 22:08 | Cluster recovery process initiated for all 6 clusters. 1st cluster recovered | | 07-15-2026 23:55 | All 6 Netapp Clusters and all customers recovered and validated as healthy & operational | ## \#\# Remediation and Prevention Google engineers were alerted to the infrastructure power loss on Wednesday, 15 July 2026 at 16:30 US/Pacific and subsequent increase in data hall temperatures at 17:44 US/Pacific via our automated monitoring and telemetry, and immediately began investigating. During the voltage transient event, the chillers maintained power but shut down because the distribution pumps failed to restart, resulting in no water flow. The pumps were manually switched from auto to hand mode to restore circulation and cooling. The facility team deployed an interim portable UPS to support the chiller controllers and prevent localized controller power loss. Upon confirmation of utility power restoration, the facility returned to utility power via automatic switching with the exception of two Remote Power Panels (RPPs), which required manual verification before restoration of power to affected server row. To provide full resiliency the facility team aligned a swing DRUPS unit to restore the redundant configuration on the A feed and is working on repairing the faulty DRUPS. To remediate the power loss at Row 3, once the electrical fault was confirmed to be fully isolated, teams reset the tripped breakers, successfully restoring both redundant power to the impacted server racks. Google is committed to preventing a repeat of this issue in the future and is completing the following actions: * Detailed investigation by the engineering team of the sequence of events that prevented transfer of load to DRUPS system and identified system improvements **\[ETA Aug 2026\].** * Determine and resolve the cause of overload conditions that caused power loss to Row 3 **\[ETA Aug 2026\].** * Perform investigation of chiller pump control system redundancy setup to ensure cooling system resiliency **\[ETA Sep 2026\].** * Implement improvements to facility system monitoring alerts for power events and configure early alert thresholds for cooling excursion detection. **\[ETA Aug 2026\].** * The GCVE engineering team is working to improve the automated triggering of shutdown during quickly evolving thermal runoff situations **\[ETA Oct 2026\]**. * The BMS engineering team is working with partner teams to update the incident classification and SLO definitions to improve personnel availability during major datacenter events **\[ETA Sep 2026\]**. * The GCNV engineering team is collaborate with the data center team to review incident handling SLA and runbooks, identify potential future occurrences and create playbooks to address/mitigate them, establish monthly joint emergency drill, and review if the Netapp cluster recovery process can be expedited **\[ETA Aug 2026\].** ## \#\# Detailed Description of Impact On 15 July 2026 from 16:39 to 16 July 2026 07:34 US/Pacific, customers experienced service disruptions across several products in europe-west4-a zone: * **Google Cloud VMware Engine**: Customers lost connectivity to their private clouds and workloads as foundational hosts and network switches were powered off for thermal protection. * **Bare Metal Solution**: Customers experienced connectivity loss to their database servers and storage appliances as the underlying hardware was powered down. * **Google Cloud NetApp Volumes**: Customers in the STANDARD, PREMIUM, and EXTREME service levels were unable to access their storage volumes. Control plane operations, including the creation of new storage pools, volumes, and backups, experienced failures in the affected region.