Cloudflare Dashboard and API Outage on April 15, 2020
Author: John Graham-Cumming
Overview
“Starting at 1531 UTC and lasting until 1952 UTC, the Cloudflare Dashboard and API were unavailable because of the disconnection of multiple, redundant fibre connections from one of our two core data centers.”
The incident was not caused by DDoS attacks, COVID-19 traffic surges, software malfunctions, hardware failures, or misconfiguration.
What Happened
During planned maintenance at a core data center, technicians were instructed to remove equipment from a cabinet containing inactive hardware scheduled for retirement. However, this same cabinet housed “a patch panel (switchboard of cables) providing all external connectivity to other Cloudflare data centers.” Over three minutes, the technician disconnected cables from this patch panel while decommissioning the unused hardware.
Since this data center houses Cloudflare’s main control plane and database, connectivity loss immediately rendered the Dashboard and API unavailable. Customer-facing services — including the CDN network, Magic Transit, Cloudflare Access, and Cloudflare Spectrum — continued operating normally.
Services Impacted
Unavailable during the outage:
- Dashboard login and API access
- Configuration changes (DNS records, etc.)
- Cache purging
- Load Balancing health checks
- Argo Tunnel connections
- Cloudflare Workers creation/updates
- Domain transfers via Cloudflare Registrar
- Logs and Analytics access
- Cloudflare Stream video encoding
No configuration data was lost; all customer data remained intact.
Response
Two virtual war rooms coordinated the response: one focused on restoring connectivity, the other on disaster recovery failover. Engineers made failover decisions every 20 minutes, weighing complexity of failback procedures against ongoing outage duration.
Restoration Timeline
- 1944 UTC: First backup link (10Gbps) restored
- 1951 UTC: First major Internet link restored
- 1952 UTC: Dashboard and API became available
- 2016 UTC: Second major link restored
- 2019 UTC: Third major link restored
- 2031 UTC: Full redundant connectivity restored
Moving Forward
Cloudflare identified three areas for improvement:
Design: External connections routed through a single patch panel created a single point of failure. Connectivity should be distributed across multiple facility locations.
Documentation: Cable and panel labeling would expedite identification and restoration during future incidents.
Process: Hardware retirement instructions should explicitly identify critical cabling to avoid accidental disconnection.
A full internal post-mortem was planned to address root causes and prevent recurrence.
Closing Statement: “We are very sorry for the disruption.”