Summary
GoCardless experienced an outage affecting their API and Dashboard for 1 hour and 50 minutes on 10 October 2017. The incident stemmed from hardware failure on the primary database node combined with unusual circumstances preventing the database cluster automation from promoting a replica as the new primary.
Our Database Setup
GoCardless stores critical data in Postgres within a cluster of three nodes: one primary, one synchronous replica, and one asynchronous replica. The company runs Pacemaker software on each node to manage promotion of new primary nodes during machine failures. Clients connect to the primary via a virtual IP address (VIP) managed by Pacemaker.
Incident Timeline
15:09: Monitoring detected total outage; engineers began investigating.
15:11: Evidence indicated disk array failure on the primary; unclear why cluster failed to promote the synchronous replica.
15:17: The broken primary Postgres node was powered off, expecting the cluster to promote a new primary. This did not occur.
15:18: Error counts were cleared in Pacemaker to prompt state rediscovery. The cluster remained unwilling to promote the synchronous replica. Engineers spent the next hour attempting various approaches without success.
16:18: The team placed the cluster into maintenance mode, configured the synchronous replica as primary manually, and reconfigured backend applications to connect directly to the new primary’s IP address rather than through the VIP.
16:46: The manually promoted primary was operational; configuration changes rolled out to applications.
16:59: Monitoring confirmed API and Dashboard restoration.
The Immediate Fallout
After bringing systems online, the priority became restoring database cluster redundancy. The team introduced a second VIP managed manually by the infrastructure team as a precaution while Pacemaker remained offline.
Data integrity verification found no evidence of corruption. The team then began planning their next steps.
The Following Weeks
The team split into two subteams: one investigating the Pacemaker failure, the other planning migration to a new cluster.
Reproducing the Failure
Through log analysis, the team identified several relevant factors:
- The RAID controller logged simultaneous loss of 3 disks; all subsequent operations failed
- The Linux kernel set the filesystem into read-only mode
- Pacemaker observed Postgres was unhealthy but couldn’t decide where to promote a new primary
- A Postgres subprocess crashed on the synchronous replica during the disk array failure, causing Postgres to terminate and restart
- After restart, the synchronous replica attempted to restore a Write-Ahead Log file via
restore_command, failing with an “invalid file” message
Using Docker containers, the team created a script mimicking these failures. Through iteration, they identified three necessary conditions for cluster failure:
-
Pacemaker setting
default-resource-stickiness— This prevents resources from moving between machines unnecessarily. GoCardless had set this to a non-zero value. -
Backup VIP resource — A second VIP configured to never run on the same server as the primary Postgres instance, with a negative infinity colocation preference. During the incident, this VIP was running on the synchronous replica.
-
Two processes crashing simultaneously — A Postgres subprocess on the synchronous replica crashed nearly simultaneously with the primary’s failure.
All three conditions were necessary. Removing the stickiness setting or the backup VIP allowed successful promotion. The team discovered that reversing the colocation rule specification (using positive infinity preference for the backup VIP with a replica instead of negative infinity with the primary) resolved the issue.
The Invalid WAL File Red Herring
During investigation, engineers noticed the synchronous replica failing to restore a Write-Ahead Log file. This log records all writes in Postgres and ensures replicas stay synchronized with the primary.
The team initially pursued this as a potential cause, even reproducing the exact validation failure in their Docker setup. However, analysis revealed this played no role in Pacemaker’s failure to promote a new primary. The invalid WAL existed because the primary archived a corrupted file before failing, but the synchronous replica already possessed valid copies through streaming replication.
Moving to a New Database Cluster
While investigating the Pacemaker failure, the second subteam worked on migrating to a new cluster with minimal disruption. They leveraged existing zero-downtime failover procedures and infrastructure code published in a public GitHub repository.
The approach introduces PgBouncer on each cluster node as an additional layer. Clients connect to a PgBouncer VIP, which itself connects to the Postgres VIP. This allows:
- Pausing incoming queries at PgBouncer while promoting a new primary
- Moving the Postgres VIP without affecting client connections
- Resuming traffic after promotion completes
The team adapted this procedure for migrating between separate clusters rather than within the same cluster. After weeks of refinement and practice runs, they executed the production migration during a scheduled maintenance window without incident.
Wrapping Up the Incident
With the new cluster operational and the Pacemaker misconfiguration understood and corrected, GoCardless decommissioned the old cluster and closed the incident.
What’s Next?
The team identified key learnings:
Pacemaker complexity — Seemingly opposite configuration specifications produce entirely different failure behaviors. The team plans to reduce Pacemaker’s scope by moving away from VIPs, instead running proxies on application servers that direct traffic based on cluster state.
Fault injection importance — Misconfigurations surfacing only during simultaneous process crashes require rigorous fault injection testing. The team plans more aggressive testing and considers tools like Chaos Monkey.
Automation erodes manual knowledge — Two years of successful automation meant manual intervention skills had atrophied. The team plans to address this through game day exercises that artificially restrict automation options.
The Elephant in the Room
GoCardless periodically evaluates managed Postgres services but remains self-hosted for several reasons. Until recently, hosted providers lacked zero-downtime patch upgrades for Postgres. Additionally, most GoCardless infrastructure operates in bare-metal hosting providers; latency to external Postgres services would require significant application redesign.
The team remains open to future changes as hosting situations and provider offerings evolve.
Closing Thoughts
The company emphasized commitment to reliability and learning from failures. The team appreciates increasing adoption of blameless post-mortems in operations disciplines and hopes this detailed review proves useful to others.