Outage Post-Mortem for August 22nd
Published: August 23, 2016 Signed: Keith, Tim, and Sam
Overview
On August 22, 2016 at 17:21 UTC, Buildkite experienced a severe unplanned outage lasting approximately 4 hours. During this period, builds continued running and GitHub/Bitbucket pull request statuses were updated, but users couldn’t login, access build logs, or read documentation.
What a morning…
The Buildkite team, based in Melbourne, Australia, was asleep when the outage began. PagerDuty failed to alert the on-call developer due to misconfigured settings and silent phone notifications. The team discovered the problem at 21:00 UTC—almost 4 hours later—through emails, tweets, and Slack messages.
Temporary fixes were implemented at 22:28 UTC, restoring buildkite.com with poor response times and dropped requests. Full recovery occurred at 23:10 UTC.
Trying to reduce our AWS bill
Approximately one year prior, Buildkite received AWS Activate program credits. As credits neared depletion two weeks before the outage, the team paused product development to reduce costs. They downgraded from an m4.10xlarge Multi-AZ RDS PostgreSQL database to an r3.2xlarge instance without adequate load testing.
Lesson #1: Monitor AWS credits proactively. “AWS Billing Reports show $0 bills while you’re spending credits, so it’s really hard to know what your actual costs are.”
Database performance issues
The weekend maintenance window downgrading the database appeared successful initially. However, when US peak traffic arrived on Monday evening UTC, the database CPU maxed out. The new database had less memory than the previous instance, prompting a migration to centralized pgbouncer for connection pooling.
Lesson #2: Conduct load testing after infrastructure changes. Time pressure prevented testing the new database and pgbouncer setup before deployment.
Failing health checks
Buildkite maintains three Elastic Load Balancers and Auto Scaling Groups for dashboard, REST API, and Agent API traffic. Health checks performed every 30 seconds by hitting both SQL databases plus Redis and Memcache.
When database performance degraded, health check endpoints returned HTTP 500 errors, triggering automatic server removal from the ELB. New replacement servers couldn’t come online due to bootstrapping errors and recent infrastructure changes, creating a cascading failure cycle.
Lesson #3: Reconsider health check design. The team switched to returning “OK” without checking database connectivity, reserving comprehensive checks for a separate Pingdom endpoint.
Replacement servers not going healthy
New Auto Scaling Group servers deployed from an AMI containing a baked-in codebase version. Newly launched instances queried a special endpoint—https://buildkite.com/_secret/version—to determine the latest deployed revision for self-deployment.
When buildkite.com went offline, new servers couldn’t retrieve the current version. They deployed with stale code containing references to decommissioned databases, causing health check failures.
Lesson #4: Don’t use service endpoints for critical bootstrap operations. “We’ve now switched to storing the latest deployed revision as file on S3.”
Our “on-call” setup failed
Multiple monitoring services (New Relic, Pingdom, Datadog, Calibre, CloudWatch) feed alerts to PagerDuty, which should phone the designated on-call developer. The primary on-call’s “Immediately phone me” rule was somehow dropped from notification settings. Escalation policies failed because other team members had phones on silent.
Lesson #5: Verify on-call PagerDuty and iPhone settings before shifts. “We’ve all done manual tests to ensure calls are getting through.”
AWS issues prevented us from fixing anything
Once awake, the team attempted database upgrades but encountered AWS IAM interface failures. They couldn’t access the IAM role selector when launching new EC2 instances. As a workaround, they redirected Agent API servers to service dashboard requests.
After AWS IAM issues resolved, the team hit EC2 request limits from excessive server churn. They had to wait for limits to reset before successfully upgrading the database and rolling out corrected servers using aws-cli.
Lesson #6: Disable auto-scaling and health check rules during high churn scenarios to avoid EC2 request limits. The team discovered AWS supports suspending and resuming Auto Scaling processes, which would have been valuable knowledge.
Moving forward
The team apologized for downtime and thanked customers for their patience. They acknowledged the irony of AWS experiencing problems while preventing Buildkite’s repairs. The conclusion emphasized commitment to preventing repeated mistakes while recognizing inevitable future incidents.
For detailed questions, readers were directed to keith@buildkite.com.