Incident Review: Pageserver outage in us-east-1
By John Spray – Engineering Manager Storage August 21, 2024
On 2024-08-14, about 0.4% of customer projects in us-east-1 experienced an outage for up to 2 hours, after an EC2 instance hosting one of our services failed.
Background
Neon’s database architecture relies on safekeeper and pageserver services. Pageservers ingest data from Postgres WAL stored by safekeepers and cache it at 8KiB page granularity. They function as a local disk cache using S3 for durable storage. When a pageserver fails, projects hosted there become unavailable until reassigned to another pageserver, though no data is lost.
Sequence of Events
- 17:28 EC2 instance pageserver-7.us-east-1.aws.neon.tech became unresponsive
- 17:38 Alerts fired, notifying the team and paging on-call engineers
- 17:41 Incident declared
- 17:45 Team identified the failed node and diagnosed it as unrecoverable
- 17:52 Decision made to migrate 17,037 projects to other pageservers
- 17:58 Migration began
- 18:40 99.5% of migrations complete; approximately 20 projects required manual intervention
- 19:41 All projects restored; alerts cleared
What Took So Long?
The recovery involved three delays:
- Approximately 30 minutes elapsed between node failure and migration decision due to human-in-the-loop processes and multi-minute alert latency
- The semi-manual migration script required roughly 40 minutes to relocate all 17,037 projects
- Some projects with in-flight operations required additional manual intervention, extending the outage to approximately two hours for some customers
How Are We Fixing It?
Neon created the Storage Controller, a new service designed for anti-fragility and fault tolerance. Unlike the original control plane’s CRUD API model, the Storage Controller uses a reconciliation-loop approach to autonomously reschedule projects when failures occur. It maintains independent heartbeat mechanisms separate from monitoring systems, enabling faster response without human intervention. When a Storage Controller-managed pageserver fails, migration occurs within seconds rather than hours.
What’s Next?
The Storage Controller has operated in production since May 2024. Currently managing all high-capacity projects (above 64GiB), the team is accelerating migration of remaining smaller projects. Following this incident, Neon plans to transition all paying customers to Storage Controller management.