Postmortem: Delayed Start Compute Operations
Date: May 16 & 19, 2025
Authors: Em Sharnoff, Mihai Bojin
Published: May 21, 2025
Summary
Neon customers in AWS/us-east-1 experienced two separate incidents preventing database creation or activation of inactive databases, totaling 5.5 hours of downtime. Active databases remained unaffected. The root cause involved IP address assignment failures for new instances. The team implemented preventative measures and plans longer-term architectural changes.
Details
Friday, May 16, 2025 (14:13 UTC):
A query execution plan change in the control plane database caused severe performance degradation—“increasing query execution times by 1-2 orders of magnitude (10x-100x).” This prevented suspension of inactive databases, creating approximately 8,000 running pods versus the planned 6,000 capacity. This exhausted available IP addresses in two of three subnets, blocking new database launches.
Mitigation steps included:
- Reconfiguring AWS CNI with modified IP targeting parameters
- Upscaling the control plane database
- Implementing rate limiting at the Neon Proxy layer
- Adjusting warm IP and ENI targets to zero, which improved allocation
Full recovery occurred around 16:50 UTC, though rate-limiting caused temporary connection errors.
Monday, May 19, 2025 (13:17 UTC):
During post-incident configuration restoration, resetting parameters “unexpectedly triggered a new outage.” Despite available subnet IPs, allocation failed. The team immediately reverted changes but experienced extended recovery delays. They doubled prewarmed compute pool sizes and investigated the underlying behavior.
Closing
The team continues investigating root causes and post-incident action items, including AWS CNI code review and VPC subnet expansion. They are developing architectural improvements through a project called “Cells” to address scaling limitations in their Kubernetes infrastructure. Further updates will follow.