How Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline Today
Author: Tom Strickx
Massive route leak impacts major parts of the Internet, including Cloudflare
What happened?
At 10:30 UTC on June 24, 2019, a significant internet disruption occurred when a small Pennsylvania company’s routes became prioritized through Verizon (AS701). The incident resulted in widespread unavailability of websites hosted on Cloudflare and other providers across large portions of the internet.
The core problem involved a “BGP Optimizer” product from Noction that fragmentizes IP prefixes into smaller, more-specific routes. For example, Cloudflare’s IPv4 route 104.20.0.0/20 was split into 104.20.0.0/21 and 104.20.8.0/21. As the article notes, “By splitting these major IP blocks into smaller parts, a network has a mechanism to steer traffic within their network but that split should never have been announced to the world at large.”
The internet operates through interconnected Autonomous Systems (AS) that communicate via Border Gateway Protocol (BGP). Specific routes take precedence over general ones in this system. DQE Communications (AS33154), an ISP in Pennsylvania using the BGP optimizer, announced these more-specific routes to their customer, Allegheny Technologies Inc. (AS396531). This information then propagated to Verizon, which broadcast it globally — a critical error that violated routing best practices.
The incident caused approximately 15% loss of Cloudflare’s global traffic at its peak. Major services including Amazon, Linode, and Cloudflare experienced disruptions because these networks were suddenly handling massive traffic volumes they weren’t equipped to manage.
How could this leak have been prevented?
Three primary prevention mechanisms existed:
Prefix Limits: BGP sessions can implement hard caps on received prefixes. Verizon could have configured such limits to automatically shut down sessions exceeding thresholds. This represents standard industry practice with minimal implementation cost.
IRR-Based Filtering: The Internet Routing Registry allows operators to create filtering rules based on established records. IRR filtering “has been around (and well documented) for over 24 years” according to Cloudflare, yet Verizon apparently implemented none with Allegheny Technologies despite the absence of cost or service limitations.
RPKI Deployment: Cloudflare implemented Resource Public Key Infrastructure globally the previous year. This framework signs prefixes with maximum size specifications — Cloudflare’s routes specified a maximum of /20. “If Verizon had used RPKI, they would have seen that the advertised routes were not valid, and the routes could have been automatically dropped by the router.” Providers like AT&T have already successfully enabled BGP Origin Validation.
These measures align with MANRS (Mutually Agreed Norms for Routing Security).
How it was resolved
Cloudflare’s network team contacted DQE Communications and Allegheny Technologies. After initial difficulty reaching contacts, a Cloudflare engineer successfully connected with DQE, who worked directly to cease advertising the optimized routes. The internet stabilized once this occurred.
Notably, despite simultaneous email and phone contact attempts, Verizon did not respond within eight hours of the incident’s conclusion, nor did the company appear to implement corrective measures independently.
Conclusion
The article emphasizes that such incidents reflect inadequate internet infrastructure safeguards. Cloudflare advocates for industry-wide RPKI adoption and applauds companies like Amazon and AT&T for validating routes. The post expresses regret for service disruption while acknowledging that engineering teams across four continents responded within minutes of identifying the problem.