Cloudflare incident on February 6, 2025
Authors: Matt Silverlock and Javier Castro
Overview
On Thursday, February 6, 2025, Cloudflare experienced a significant outage affecting its R2 object storage service and multiple dependent products. The disruption lasted 59 minutes and resulted from human error during abuse remediation processes. Importantly, the incident caused no data loss or corruption within R2’s storage infrastructure.
As stated in the incident report: “This caused all operations against R2 to fail for the duration of the incident, and caused a number of other Cloudflare services that depend on R2…to suffer significant failures.”
What Was Impacted
The outage affected multiple services with varying degrees of severity:
R2 Object Storage experienced 100% failure rates during the primary incident window (08:14 to 09:13 UTC), with no data loss in the storage subsystem itself.
Dependent Services suffered cascading failures:
- Stream: 100% operations failure (uploads and streaming)
- Images: 100% operations failure (uploads and downloads)
- Cache Reserve: Increased origin requests as cached assets became unavailable
- Vectorize: 75% query failures; 100% insert/upsert/delete operation failures
- Log Delivery: Up to 13.6% data loss for R2 delivery jobs
- Durable Objects: 0.09% error rate increase during recovery phase
- Workers and Pages: 0.002% deployment failures for services with R2 bindings
Root Cause Analysis
The incident stemmed from a critical failure in abuse remediation workflows. During routine processing of a phishing complaint, an operator inadvertently disabled the entire R2 Gateway service rather than the specific endpoint associated with the report.
The fundamental issue involved insufficient validation safeguards: “A key system-level control that led to this incident was in how we identify (or ‘tag’) internal accounts used by our teams.”
The system lacked explicit protections to prevent disablement actions against production infrastructure accounts, relying instead on operator training and manual tagging — both insufficient controls for mission-critical systems.
Incident Timeline
| Time | Event |
|---|---|
| 08:12 UTC | R2 Gateway service inadvertently disabled during abuse report response |
| 08:14 UTC | IMPACT BEGINS |
| 08:15 UTC | Service metrics show degradation |
| 08:17 UTC | Critical R2 alerts fire due to failed health checks |
| 08:18 UTC | R2 on-call team engaged |
| 08:23 UTC | Sales engineering escalates customer impact reports |
| 08:25 UTC | Internal incident declared |
| 08:33 UTC | Root cause investigation escalated |
| 08:42 UTC | Root cause identified in deployment history review |
| 08:46 UTC | Initial recovery attempt fails (tooling relies on R2) |
| 08:49 UTC | Escalation to operations team with lower-level system access |
| 08:57 UTC | Operations team begins R2 Gateway re-enablement |
| 09:09 UTC | R2 team triggers forced redeployment |
| 09:10 UTC | R2 begins recovery; clients reconnect |
| 09:13 UTC | IMPACT ENDS; R2 returns to SLO; Durable Objects sees 0.09% error spike |
| 09:36 UTC | Durable Objects error rate normalizes |
| 10:29 UTC | Incident closed |
Technical Details
R2’s architecture separates concerns across multiple components: the Gateway service (authentication and API serving), metadata store (built on Durable Objects), intermediate caches, and distributed storage. This separation proved crucial during recovery — all other components remained operational while only the Gateway was disabled.
When the Gateway service stopped, all in-flight requests terminated. However, any requests that received HTTP 200 responses had already succeeded with data persisted to storage and metadata layers. This architectural design prevented consistency violations upon service recovery.
The secondary incident phase (09:13-09:36 UTC) involved a “stampeding horde” effect as reconnecting R2 clients overwhelmed the metadata layer’s Durable Objects infrastructure, causing the modest 0.09% error rate increase.
Remediation Actions
Immediately Implemented:
- Additional guardrails in the Admin API preventing product disablement of services in internal accounts
- Suspension of product disablement actions in the abuse review UI pending stronger safeguards
In-Progress Initiatives:
- Restructuring internal account creation to ensure proper organizational provisioning
- Restricting product disablement access to senior operators only
- Implementing two-party approval requirements for ad-hoc disablement actions
- Expanding abuse checks to prevent disablement of products associated with internal Cloudflare accounts
- Migrating internal accounts to the Organizations model with protective controls
Conclusion
Cloudflare acknowledged the severity: “We understand this was a serious incident, and we are painfully aware of — and extremely sorry for — the impact it caused to customers and teams building and running their businesses on Cloudflare.”
The company committed to preventing recurrence through systemic controls rather than process reliance, emphasizing “removing the need to rely on training or process, and instead ensuring that our systems have the right guardrails and controls built-in to prevent operator errors.”