Cloudflare Incident on August 21, 2025
Authors: David Tuber, Emily Music, Bryton Herdes
Overview
On August 21, 2025, a traffic surge directed toward customers hosted in AWS us-east-1 created severe congestion on connections between Cloudflare and that AWS region. This regional issue impacted users connecting to origins in us-east-1 through elevated latency, packet loss, and connection failures. The incident began at 16:27 UTC and was substantially resolved by 19:38 UTC, with intermittent issues continuing until 20:18 UTC.
According to the post, “This was a regional problem between Cloudflare and AWS us-east-1, and global Cloudflare services were not affected.”
Background
Cloudflare operates as a reverse proxy service, sitting between visitors and origin servers. When visitors request content, Cloudflare either serves cached material from its global network or fetches it from the origin server. The company’s internal network capacity is intentionally oversized to handle redundancy failures, traffic rerouting, and demand spikes.
However, during this incident, “some edge router links to an AWS peering switch had insufficient capacity to handle this particular surge.”
What Happened
Beginning at approximately 16:27 UTC, a single customer initiated a massive volume of requests from AWS us-east-1 for cached objects. The response traffic saturated all available direct peering connections between Cloudflare and AWS. AWS attempted to mitigate congestion by withdrawing BGP advertisements, which rerouted traffic to additional peering links that subsequently became saturated as well.
Two pre-existing vulnerabilities exacerbated the problem: one direct peering link operated at half-capacity due to prior failure, and the Data Center Interconnect connecting edge routers to the offsite switch was scheduled for capacity upgrades but hadn’t yet been implemented.
Timeline
| Time | Description |
|---|---|
| 2025-08-21 16:27 UTC | Traffic surge begins, doubling total traffic to AWS (IMPACT START) |
| 2025-08-21 16:37 UTC | AWS withdraws prefixes on congested PNI BGP sessions |
| 2025-08-21 16:44 UTC | Network team alerted to Ashburn internal congestion |
| 2025-08-21 16:45 UTC | Team evaluates response options; AWS withdrawals block uncongested paths |
| 2025-08-21 17:22 UTC | BGP withdrawals increase dropped traffic (IMPACT INCREASE) |
| 2025-08-21 17:45 UTC | Incident formally raised for Ashburn customer impact |
| 2025-08-21 19:05 UTC | Rate limiting decreases congestion from surge customer |
| 2025-08-21 19:27 UTC | Additional traffic engineering fully resolves congestion (IMPACT DECREASE) |
| 2025-08-21 19:45 UTC | AWS begins reverting BGP withdrawals |
| 2025-08-21 20:07 UTC | AWS finishes normalizing BGP announcements |
| 2025-08-21 20:18 UTC | IMPACT END |
Impact Analysis
The congestion caused router network queues to grow substantially, resulting in packet dropping. Edge routers consistently dropped high-priority packets during the outage. Customer impact manifested as elevated latency, timeouts, and reduced throughput.
Cloudflare monitors Service Level Objectives through latency measurements simulating customer requests to origins. During the incident, these metrics fell below acceptable thresholds in alignment with packet drop periods.
Remediations and Follow-Up Steps
Cloudflare identified the core issue: “one customer’s usage patterns cannot negatively affect the broader ecosystem.” The company is implementing a multi-phase strategy:
Short and medium-term actions:
- Developing selective traffic deprioritization mechanisms for customers causing network congestion
- Expediting Data Center Interconnect capacity upgrades
- Coordinating with AWS to prevent BGP traffic engineering conflicts
Long-term solution: The company plans to build an enhanced traffic management system allocating per-customer network resource budgets. Exceeding these allocations will prevent individual customers from degrading service for others. This system will also automate many manual mitigation actions currently required.
Conclusion
The August 21 incident resulted from insufficient congestion management during an unusual high-traffic event from a single customer. Cloudflare acknowledges the disruption and states it is “actively making these improvements to ensure improved stability moving forward and to prevent this problem from happening again.”