Earlier today, July 6, 2016, the npm registry experienced a read outage for 0.5% of all package tarballs for all network regions. Not all packages and versions were affected, but the ones that were affected were completely unavailable during the outage for any region of our CDN.
The unavailable tarballs were offline for about 16 hours, from mid-afternoon PDT on July 5 to early morning July 6. All tarballs should now be available for read.
Outage Timeline
- Evening of 2016 July 1: Code implementing changes to etag generation for new tarball publications was deployed.
- Midday 2016 July 6: A script updating all existing tarballs with new etag schemes commenced across servers over several hours.
- 10pm PDT 2016 July 5: Initial reports of 502 errors. During investigation, a known process leak was identified and addressed.
- 6am PDT 2016 July 6: Additional 502 reports prompted opening a formal status incident.
- 8am PDT: Root cause identified as negative file modification times on affected tarballs. A corrective script was executed immediately.
Within an hour, 502 error rates returned to normal levels.
Prevention Measures
Enhanced alerting on all 500-class status codes will be implemented, not just 503s. The operational playbook will emphasize more frequent CDN log examination and improved tools for analyzing error patterns across points of presence.
Technical Root Cause: Nginx and Etags
The underlying issue involved file modification time, nginx etag generation, and cache headers interaction. Nginx generates etags using the formula approximately mtime + '-' + file size in bytes.
The team implemented a caching improvement by setting file modification times based on the first 32-bit integer from md5 hashes. Testing in staging showed this produced consistent etags. However, the production script failed to prevent negative numbers, creating timestamps before the Unix epoch.
Negative mtimes triggered an nginx bug: while serving the initial request with a negative etag, subsequent requests with that etag in the if-none-match header caused nginx to attempt a 304 response that never completed, resulting in bad gateway errors.
Since affected tarballs excluded those in smoketest monitoring, servers remained active. The system alerted on 503s but not 502s. The fix involved resetting file timestamps to valid values, busting the CDN cache and circumventing the nginx behavior.
Conclusion
CDN logs proved invaluable for identifying actual user experience issues that internal monitoring missed. Logs revealed service quality problems invisible to standard monitoring systems.