What Happened?
Mozilla’s infrastructure experienced a critical incident on January 13, 2022. The company maintains various servers handling internal services including updates, telemetry, and crash reporting across multiple cloud providers. On the specified date, Google Cloud Platform deployed an unannounced change making HTTP/3 the default protocol for load balancers that had been set to “Automatic (default)” settings.
Firefox’s networking code prioritizes HTTP/3 when available, so connections to Mozilla’s services shifted from HTTP/2 to HTTP/3. Shortly after this change, Mozilla observed a dramatic spike in crash reports, eventually reaching approximately 300,000 unprocessed reports, and received multiple accounts of browser hangs worldwide.
During incident response, engineers discovered that clients were hanging during network requests to internal services but couldn’t immediately identify the trigger. Mozilla’s infrastructure hadn’t deployed updates or configuration changes, and HTTP/3 had been supported since Firefox 88. The investigation revealed that load balancers were suddenly serving HTTP/3 connections. Mozilla disabled HTTP/3 explicitly on GCP at 09:12 UTC, restoring functionality, though the root cause remained unclear.
A Special Mix of Ingredients
Investigation revealed the issue required a specific combination of circumstances. Standard testing with various tools and remote services couldn’t reproduce the problem, but Firefox itself successfully triggered the issue with the staging server.
The critical factor involved Rust components accessing the network through “viaduct,” an intermediate library, rather than directly through Necko, Firefox’s networking stack. For HTTP/3 upload requests, Necko’s higher-level APIs automatically add the Content-Length header if absent. However, viaduct lowercases all headers before passing them to Necko. While Necko’s API checks are case-insensitive, the lower-level HTTP/3 code performs case-sensitive lookups. This mismatch meant headers added by viaduct would pass initial checks but remain undetectable to HTTP/3 code.
Telemetry emerged as currently the only Rust-based component in Firefox Desktop that uses the network stack and adds a Content-Length header. Users disabling Telemetry experienced resolution, though the issue wasn’t Telemetry-specific.
The Infinite Loop
The final ingredient existed deep within Necko’s HTTP/3 implementation. When the code performed a case-sensitive search for the header that viaduct had lowercased, it failed to locate it. Without finding the header, the code determined the request was complete, leaving the actual body unsent.
Critically, the code would only terminate when there was no additional content to send. This unexpected state caused the code to loop indefinitely rather than returning an error. Since all network requests operate through a single socket thread, this infinite loop blocked further network communication entirely, rendering Firefox unresponsive and unable to load web content.
Lessons Learned
Mozilla identified several contributing factors:
-
GCP’s unannounced HTTP/3 deployment lacked communication. While announcement alone might not prevent incidents entirely, it would likely have prompted controlled experimentation and deliberate deployment procedures.
-
Setting load balancers to “Automatic (default)” rather than explicit configuration allowed automatic deployment. Mozilla committed to reviewing all service configurations to prevent similar occurrences.
-
The combination of HTTP/3 with viaduct on Firefox Desktop wasn’t covered by continuous integration testing. Though comprehensive testing of every configuration combination isn’t feasible, HTTP version selection represents a major change warranting systematic testing alongside additional networking layers. Current HTTP/3 tests focus on low-level protocol behavior and Necko usage by web content; Mozilla plans to expand testing across HTTP versions.
Mozilla indicated ongoing investigation into making the browser more resilient and accelerating incident response, emphasizing that comprehensive learning from incidents strengthens product quality.