A Post Mortem on this Morning’s Incident

Author: Jérôme Fleury

Overview

Cloudflare experienced significant internet outages on June 17 and June 20, both involving packet loss through Telia Carrier’s backbone network. The company detailed the incidents and outlined steps to prevent future occurrences.

June 17 Incident

On June 17 at 08:32 UTC, Cloudflare systems detected “significant packet loss between multiple destinations on one of our major transit provider backbone networks, Telia Carrier.” The loss became intermittent and eventually ceased during the analysis period.

June 20 Incident

At 12:10 UTC on June 20, massive packet loss recurred on the same provider. Cloudflare noted that “transit providers are very reliable and transport all of our packets from one point of the globe to the other without loss.” When this assumption failed, the infrastructure struggled to manage the volume of dropped packets.

The company explained that the Internet Protocol relies on TCP for retransmission of lost packets, but “when the loss is too significant, as was the case this morning, your browser can’t do much.” At 12:30 UTC, Cloudflare took down its ports with Telia and rerouted traffic through other providers.

Customer Impact

The incidents caused a spike in 522 HTTP errors, which indicate the inability to reach customer origin servers.

Communication Challenges

Cloudflare acknowledged deficiencies in its incident communication, noting that “the scope of the incident was incorrectly identified in Europe only, and our response time was not adequate.” The company committed to improving status page updates and implementing automated detection systems.

Network Resilience Plans

The post outlined infrastructure improvements:

• Multiple transit providers: Using different providers reduces single-point-of-failure risks
• BGP augmentation: Developing mechanisms to detect packet loss beyond basic routing protocol capabilities
• Automated failover: Currently active for smaller data centers; planned expansion within two weeks to all points of presence (PoPs)
• Traffic diversity: Maintaining buffer capacity and spreading traffic across redundant providers and peering relationships

The Johannesburg PoP example demonstrated automatic failover capability when connection issues were detected.

Conclusion

Cloudflare emphasized its commitment to “completely automated systems to deal with this type of incident,” aiming to minimize disruptions regardless of failure origin.