Major data center power failure (again): Cloudflare Code Orange tested
Authors: Matthew Prince, John Graham-Cumming, Jeremy Hartman
Overview
Cloudflare experienced a second major power outage at its Portland, Oregon data center facility within five months. However, the company’s response differed dramatically from the first incident due to significant infrastructure improvements implemented under “Code Orange,” an internal crisis management initiative.
The Incident
On March 26, 2024, at 14:58 UTC, the PDX01 facility lost complete power. Unlike the November 2023 outage, Cloudflare’s systems responded automatically. By 15:05 UTC — just seven minutes later — APIs and dashboards were operating normally “all without human intervention.”
The facility experienced a “total loss of power to Cloudflare’s physical infrastructure following a reportedly simultaneous failure of four Flexential-owned and operated switchboards.”
Code Orange Initiative
Following the November 2023 outage, Cloudflare declared Code Orange, borrowing from Google’s crisis management model. The initiative empowered leadership to redirect engineering resources toward critical resilience priorities. This represented a company-wide commitment to ensure service continuity despite catastrophic facility failures.
Results of the March 26 Incident
Control Plane Recovery
Services that experienced six hours or more of downtime in November — including API, Dashboard, Zero Trust, Magic Transit, SSL, Workers, and others — recovered within minutes in March. The control plane’s rapid recovery resulted from migrating configuration databases to highly available topologies with pre-provisioned capacity.
Analytics Impact
The Analytics platform remained partially impacted, though this was expected. The platform still depends heavily on PDX01. Recovery work continues, with completion expected “in the near future.”
Data Center Cold Start
Restarting the affected facility took approximately 10 hours in March, compared to 72 hours in November — a significant improvement in operational procedures.
Technical Improvements
Database Migration: More than 100 databases across over 20 clusters automatically failed over from the affected facility. Weekly failover tests validated this capability.
Logpush Infrastructure: Cloudflare made Logpush highly available in Portland and created an active failover option in Amsterdam, enabling continuous log delivery during the outage.
Product Resilience: Stream and Zero Trust products implemented improvements allowing seamless failover to Amsterdam and distribution across 300+ global data centers, respectively.
Root Cause Analysis
Investigation by Flexential, the facility operator, identified incorrectly configured breaker coordination settings in Circuit Switch Boards (CSBs) as a contributing factor. “Trip settings which are too restrictive can result in overly sensitive overcurrent protection and the potential nuisance tripping of devices.” When breakers tripped, cascading failures disabled all power serving Cloudflare’s infrastructure.
The facility’s team reset the CSBs and adjusted settings to correct values, enabling staged server restart.
Next Steps
Cloudflare’s priorities include:
- Completing Analytics platform resilience work
- Eliminating manual intervention requirements for service recovery
- Conducting production cut tests to validate improvements
- Collaborating with Flexential on operational review procedures across all critical facilities
Conclusion
The company acknowledged that “our work over the past four months has yielded the results that we expected.” The March incident served as validation that infrastructure improvements implemented under Code Orange provided the resilience customers require, though additional work remains to eliminate remaining vulnerabilities.