By: Laura Nolan, Senior Staff Engineer
Overview
On May 12, 2020, Slack experienced a significant outage—the company’s first in a considerable time. This detailed technical analysis examines the root causes and contributing factors.
The Morning Incident
The customer-visible outage began at 4:45pm Pacific time, but the sequence of events started much earlier. Around 8:30am, the Database Reliability Engineering team detected a substantial load increase in database infrastructure coinciding with API request failures reported by the Traffic team.
A configuration change rollout triggered an existing performance bug in the database layer. The issue was quickly identified and reverted—it involved a feature flag with percentage-based deployment, enabling rapid remediation. Customer impact was minimal, lasting only three minutes with most users able to send messages throughout the incident.
Scaling Response
The morning incident prompted significant autoscaling of the main webapp tier. Slack had experienced increased usage due to pandemic-related stay-at-home orders. The company increased its webapp instance count by 75%, reaching the highest number of web hosts ever deployed.
Everything appeared stable for the next eight hours.
The Evening Outage
Around 4:45pm Pacific, alerts indicated elevated HTTP 503 error rates. The on-call engineer manually scaled the webapp fleet as initial mitigation, but this had no effect. Investigation revealed that only a subset of the webapp fleet experienced heavy load while others remained underutilized.
Root Cause: HAProxy Backend Management
Slack uses HAProxy instances behind a layer 4 load-balancer, with Consul for service discovery and consul-template for rendering healthy webapp backend lists.
Rather than updating HAProxy configuration files directly (which would require resource-intensive reloads), Slack employs the HAProxy Runtime API to manipulate server state dynamically. This approach balances the need for rapid autoscaling with operational efficiency.
HAProxy is configured with N ‘slots’ (server templates) per Availability Zone, supporting N × M total backends across M parallel pools. After the morning incident, Slack was running slightly more webapp instances than available slots—a manageable situation with adequate serving capacity.
The Critical Bug
The program synchronizing consul-template’s host list with HAProxy server state contained a critical flaw: it attempted to populate new webapp instance slots before releasing slots occupied by deprovisioned instances. When no empty slots remained, the program failed and exited early, leaving HAProxy state unupdated.
Throughout the day, as the autoscaling group scaled up and down, the HAProxy state grew increasingly stale, with older backends dominating the configuration.
By 4:45pm, most HAProxy instances could only route requests to webapp backends provisioned that morning—now a minority of the active fleet. While newer HAProxy instances had correct configuration, most were eight+ hours old with outdated state.
The outage materialized when US business-day traffic declined and autoscaling terminated older instances preferentially. The remaining older webapp instances in HAProxy’s stale state proved insufficient to handle demand.
Detection Failures
Despite alerting mechanisms designed to catch this exact scenario, monitoring had degraded over time. The system had operated reliably for an extended period without requiring changes. The wider HAProxy deployment remained relatively static, with few engineers interacting with its monitoring and alerting infrastructure. This reduced visibility allowed the alerting failures to go unnoticed.
Resolution and Prevention
Once identified, the outage was quickly resolved through a rolling restart of the HAProxy fleet.
Slack is migrating toward Envoy Proxy for ingress load-balancing, having already transitioned websocket traffic to Envoy. While HAProxy provided reliable service for years, the incident exposed operational limitations. Envoy’s native xDS control plane integration will replace Slack’s complex HAProxy state manipulation pipeline, eliminating the architectural vulnerability that caused this outage.
The new architecture using Envoy and xDS is not susceptible to the stale backend state problem that triggered this failure.
Conclusion
Slack acknowledges the outage as a failure to maintain service reliability. Recognizing Slack as critical infrastructure for users, the company commits to learning from incidents and continuously improving both systems and processes. The organization apologizes for the inconvenience caused and emphasizes its dedication to reliability improvements informed by incident analysis.