Postmortem: March 26 - April 10 Workflow Delay Incidents
Posted by Sara — April 29, 2019
CircleCI experienced platform instability between March 26 and April 10, 2019, primarily stemming from issues with the datastore backing our builds queue. The company has now maintained stability for over two weeks and is sharing details to enhance customer understanding.
Timeline Overview
On March 26, 2019, workflow processing delays emerged. The investigation revealed slow timings across multiple MongoDB replica sets. The team scaled services and MongoDB capacity while also updating to a new JVM minor version intended for security improvements.
The JVM upgrade inadvertently reduced thread and connection pool sizes through newly-enabled docker-awareness features, constraining system throughput. By the following day, CircleCI disabled this docker detection and manually tuned thread pools, achieving a 40% increase in the number of threads per service instance by week’s end.
Subsequent incidents on April 2-4 revealed MongoDB replica set degradation and Out of Memory errors in API containers. Investigation identified that creating indexes in MongoDB is idempotent, yet the company’s practice of redeclaring indexes during service startup consumed database-level locks, starving WiredTiger storage backend tickets and causing widespread contention.
By April 10, memory pressure and cache eviction failures plagued the primary MongoDB instance. After one hour of investigation, the team failed over to a secondary instance, which instantly began processing twice as many database operations.
Key Learnings
MongoDB support recommended TCMalloc tuning parameters: setting tcmallocEnableMarkThreadIdle to false and TCMALLOC_AGGRESSIVE_DECOMMIT to “y”. The team also identified the tcmalloc.spinlock_total_delay_ns metric as a leading degradation indicator.
Next Steps
CircleCI implemented automated failover protocols, improved monitoring through Datadog and Honeycomb, and committed to application-level data access optimizations and MongoDB version upgrades.