Delayed Start Compute Operations – Triggering Event
Date: May 30, 2025
Authors: Mihai Bojin, Matt Sherman
Category: Engineering
Summary
The Neon Control Plane service runs on a Postgres database. A scheduled job called the Activity Monitor identifies Computes ready for suspension. A query executed by this job altered its execution plan, selecting a suboptimal index that dramatically degraded performance. This caused database CPU saturation, preventing the Activity Monitor from suspending Compute VMs. Consequently, concurrent Computes exceeded cluster capacity, resulting in IP allocation failures.
Architecture
The Neon Control Plane is a regional Golang service orchestrating Neon Postgres VMs, called “Computes.” The serverless architecture requires starting Computes when customers execute SQL statements and suspending them during inactivity periods—a function called scale-to-zero.
The Activity Monitor, a scheduled job within the Control Plane, evaluates running computes and identifies candidates for suspension.
Chain of Events
The Activity Monitor relies on a Postgres SQL query involving multiple joins, including one against the computes table, which stores records for every started or idle Compute.
The triggering event involved a Postgres execution plan change for Activity Monitor queries. This caused a broad scan on the computes table, saturating CPU on the control plane’s backing Postgres database.
The computes table contains tens of millions of rows representing the complete history of start and suspend events. Yet only a small subset represents actively running Computes.
When proper indexes are used, queries execute in milliseconds. However, incorrect index selection causes performance degradation, scanning millions of pages to retrieve thousands of rows.
This plan change occurred region-specifically, as Control Plane databases are region-isolated. The us-east-1 region experienced this issue first.
Response Measures
During the incident:
- The team executed
ANALYZE computesto refresh statistics and restore the execution plan - Engineers refactored
fetchEndpointWithComputeby wrapping the expensive sub-query in a materialized CTE, which acts as an optimization fence, ensuring the planner evaluates it once using appropriate indexes
Future Improvements
- Implementing more aggressive garbage collection to reduce table rows from tens of millions to millions
- Moving historical data to an OLAP system separate from the production OLTP database
- Investigating exact conditions triggering execution plan changes across regions