Website Intermittent Timeouts

Timeline of Updates

January 12, 2017 - 08:11 PST (Investigating)

We’re currently investigating intermittent timeouts with our Control Panel, API, SMTP and Tracking links at the moment.

January 12, 2017 - 11:49 PST (Update)

We’re seeing improved performance and stability from our core API & SMTP services at this time.

January 12, 2017 - 12:51 PST (Monitoring)

Engineers made changes improving performance and were monitoring results.

January 12, 2017 - 15:10 PST (Resolved)

The incident was marked as resolved.

Postmortem Summary

On January 12 at 14:09 UTC, MongoDB secondary servers experienced heavy load following configuration changes made the previous day. An attempted rollback at 15:15 UTC inadvertently caused API and SMTP services to shift to primary servers, overwhelming them by 15:55 UTC. Between 16:00-19:00 UTC, the team resized servers and implemented connection limits, achieving roughly 75% throughput. At 20:28 UTC, connection limits and read preference changes were deployed, stabilizing operations by 20:35 UTC.

Root Cause

Logs showed broad cluster degradation without identifying specific problematic queries.

Corrective Actions Completed

• Redundant tracking infrastructure across two regions
• Connection limits across all database-dependent services
• Increased primary MongoDB cluster capacity