2023-03-08 Incident: Infrastructure Connectivity Issue Postmortem

by Alexis Lê-Quôc — Published May 16, 2023

Summary

On March 8, 2023, Datadog experienced a multi-region outage affecting US1, EU1, US3, US4, and US5 regions. The incident rendered the platform inaccessible and disabled monitoring alerts for approximately 27 hours until full resolution on March 10.

Root Cause

The outage stemmed from an automatic security update to systemd that created an unexpected interaction with container networking. When systemd-networkd restarted following the update, it deleted routing rules managed by Cilium (the Container Network Interface plugin), taking tens of thousands of nodes offline simultaneously across all five regions between 06:00-07:00 UTC.

The update applied universally because a legacy OS security channel remained enabled on base images. The timing window defaulted to 06:00-07:00 UTC, causing synchronous failures across geographically independent regions.

Impact Timeline

• 06:00 UTC: Automatic systemd security update triggered
• 06:03 UTC: Internal monitoring detected the issue
• 06:31 UTC: First public status update issued
• 09:13 UTC: Web access began restoring
• March 9, 08:58 UTC: All services declared operational
• March 10, 06:25 UTC: Full resolution with historical data backfilled

Response Efforts

Datadog activated an emergency operations center with 10 senior engineering leaders, approximately 70 incident commanders, and 450-750 responders across four shifts. Communications teams published roughly 200 status updates and engaged hundreds of support engineers in continuous customer communication.

Key Recovery Decisions

The team prioritized restoring real-time data processing before historical access, recognizing that live telemetry and active alerts held greater value than historical information. They simultaneously recovered regional control planes before Kubernetes cluster repairs.

Preventive Measures Implemented

• Disabled the legacy security update channel across affected regions
• Reconfigured systemd-networkd to preserve routing tables on restart
• Audited infrastructure for similar legacy update channels
• Committed to chaos engineering improvements using actual disruption scales