Postmortem: May 21, 2021 - Delay in starting Docker Jobs. Machine & remote Docker environments blocked

Posted by JonathanC503 — June 3, 2021

Summary

The incident affected Docker, Machine executor (Windows, Mac, Arm, GPU), and remote Docker jobs from May 21 at 13:30 UTC through May 22 at 01:18 UTC. CircleCI acknowledged the disruption and committed to preventing similar outages.

Root Cause

A RabbitMQ upgrade from version 3.8.9 to 3.8.16 on May 20 introduced unexpected behavior. The changelog indicated a 15-minute acknowledgment timeout would affect only quorum queues, but this change impacts all queue types. This caused message consumers to disconnect, eliminating the queue responsible for VM cleanup in a specific region.

Key Events

• 02:31 May 21: VM destroyer service queue lost all consumers
• 10:11: Queue backup alert triggered but misconfigured; no page sent
• 13:00: Engineers paged about job startup delays
• 14:13: Linux machine jobs blocked from work queue
• 18:05: Database burst balance exhausted; connections timed out
• 21:57: Distribution system revised to block remote-docker jobs
• 01:18 May 22: Incident fully resolved

Prevention Measures

CircleCI planned improvements including enhanced VM provisioning resilience, independent provisioning systems, better RabbitMQ observability, GCP rate limit batching, and improved incident response protocols.