Cloudflare incident on March 21, 2025
Author: Phillip Jones
Overview
Multiple Cloudflare services experienced elevated error rates on March 21, 2025, between 21:38 UTC and 22:45 UTC — a duration of 1 hour and 7 minutes. The primary service affected was R2 object storage, though the incident cascaded to impact Cache Reserve, Images, Log Delivery, Stream, and Vectorize.
The fundamental problem stemmed from a credential rotation process gone awry. During the key rotation, engineers inadvertently deployed new storage credentials to a development Worker instead of the production Worker due to an omitted environment parameter in their deployment commands.
Impact Details
R2 Service:
- Write operations failed completely (100% error rate)
- Read operations failed at approximately 35% globally
- No data loss or corruption occurred
- Metadata-only operations remained unaffected
Dependent Services:
- Billing: Customers experienced errors downloading past invoices
- Cache Reserve: Increased origin requests as R2 reads failed
- Images: 100% of uploads failed; successful delivery dropped to 25%
- Key Transparency Auditor: 100% of operations failed
- Log Delivery: Delays up to 70 minutes in log processing
- Stream: 100% of uploads failed; video segment delivery dropped to 94%
- Vectorize: Increased error rates for read queries; 100% failure for insert/upsert operations
Incident Timeline
| Time (UTC) | Event |
|---|---|
| 19:49 | R2 team initiated credential rotation process |
| 20:19 | Deployed updated credentials using wrangler deploy (with omitted --env parameter) |
| 20:20 | Updated Worker assigned to default environment (incorrect) |
| 20:37 | Old credentials removed from storage infrastructure |
| 21:38 | IMPACT BEGINS – Service degradation visible in metrics |
| 21:45 | Global availability alerts triggered |
| 21:50 | Internal incident declared |
| 22:05 | Public incident status published |
| 22:15 | R2 team created new credentials attempting force re-propagation |
| 22:30 | Deployed another credential set (still to wrong Worker) |
| 22:36 | ROOT CAUSE IDENTIFIED – Reviewed production Worker release history |
| 22:45 | IMPACT ENDS – Deployed credentials to correct production Worker |
| 22:54 | Incident resolved |
Root Cause Analysis
R2’s architecture comprises three components: the production gateway Worker, metadata service, and storage infrastructure. The gateway Worker uses credential pairs (ID and key) to authenticate with distributed storage.
The standard key rotation process involves five steps:
- Create new credentials while maintaining old ones
- Deploy new secret via
wrangler secret put - Deploy new credential ID via
wrangler deploy - Delete old credentials from storage
- Validate through operational monitoring
The critical failure occurred in step 3. The wrangler secret put and wrangler deploy commands default to the “default” environment when the --env parameter is omitted. The team deployed to the default environment instead of the production environment, meaning the production Worker still held outdated credentials.
When step 4 removed the old credentials, the production gateway Worker lost authentication access, triggering widespread failures. The gradual error propagation delayed discovery — the team should have verified which credentials the gateway Worker actually used before deleting legacy credentials.
Resolution and Prevention
The immediate fix involved deploying correct credentials to the production Worker, restoring availability immediately.
To prevent recurrence, Cloudflare implemented several measures:
Completed Actions:
- Added logging tags displaying credential ID suffixes for explicit verification
- Updated processes requiring confirmation that new token IDs match storage infrastructure logs before deleting old tokens
- Mandated credential rotation through hotfix release tooling rather than manual command entry
- Explicitly required two-human validation in standard operating procedures
In Progress:
- Extending health check systems to test new keys and automate status reporting
- Updating observability platforms to display upstream success rates bypassing cache for clearer diagnostics
The organization emphasized that “this incident happened because of human error” and acknowledged that proper visibility into credential usage would have accelerated resolution significantly.