By Sean Madden, Staff Software Engineer. Published December 10, 2024.
Strengthen your system’s ability to recover by intentionally causing and resolving failures.
The Incident
Slack engineers use Kibana with Elasticsearch to save custom dashboards and visualizations of important application performance data. On January 29th, 2024, the Kibana cluster—and subsequently, the dashboards—started to fail due to a lack of disk space. Investigation revealed this resulted from an earlier architectural decision. Elasticsearch was configured on the same hosts as the Kibana application, tying storage and the application together on the same nodes, which were now failing. Slack engineers couldn’t load the data they needed to ensure their applications were healthy.
Eventually, the cluster deteriorated to such a poor state that it couldn’t be salvaged, requiring a complete rebuild from scratch. The team attempted to stand up a new cluster by cycling in new hosts and restoring Kibana objects from a backup. However, they discovered their most recent backup was almost two years old. The backup and restore method hadn’t received significant attention after its initial configuration, and it lacked alerts to monitor its functionality. Additionally, the runbook was outdated, and the old backup failed during restoration attempts. The team lost internal employees’ links and visualizations, and had to recreate indexes and index patterns manually.
The incident revealed a critical vulnerability: backup and recovery procedures had become obsolete and failed precisely when needed most.
Breaking Stuff is Fun
The team was determined to transform this incident into tangible benefits. Post-incident tasks included ensuring Elasticsearch clusters in every environment had scheduled backups, fixing runbooks based on experience, and verifying Amazon S3 retention policies were configured correctly.
To test these improvements, the team devised an unconventional approach: they would intentionally break a development Kibana cluster and execute the new backup and restore process. The development cluster mirrored production configurations, providing a realistic testing environment. The team carefully planned which cluster to break, the method of failure, and the restoration strategy.
Running the Exercise
The testing event was scheduled for a quiet Thursday morning with full team participation. Attendees arrived energized and delighted by the opportunity to intentionally cause failures. The team filled disk space on Kibana nodes, watched them fail in real time, and successfully triggered alerts. Following the new runbook, they cycled the entire cluster into a fresh rebuild. The system recovered successfully from the staged incident.
Though recovery succeeded, the team fell short of their one-hour recovery goal. Runbook commands weren’t well understood and proved difficult to execute during stressful conditions. Even copying and pasting from the runbook presented challenges due to formatting issues. Despite these rough edges, backups completely restored the cluster state. The exercise also revealed firewall rules needing addition to infrastructure-as-code—an unexpected discovery that prevented future issues.
In a final validation of their improved recovery process, the team migrated the general development Kibana instance and Elasticsearch cluster to Kubernetes. This high-use cluster provided an excellent opportunity to test the improved backup script. Thanks to enhanced understanding of the process and updated provisioning scripts, the migration succeeded with approximately 30 minutes of downtime.
Both exercises revealed minor runbook and restoration process deficiencies. The team spent time identifying gaps and improving documentation. Inspired by the exercise, they automated the entire process by updating the scheduled backup script tool into a full-featured CLI backup and restore program. Now Kibana backups can be completely restored from cloud storage with a single command. Breaking systems proved both enjoyable and an invaluable investment that would reduce future stress.
Chaos is Everywhere—Might as Well Use It
Every production system contains undiscovered failures. Teams should invest time locating issues and planning recovery before crises occur. Generate significant traffic, load test services, simulate unexpected outages, and upgrade dependencies frequently. System maintenance in software is often neglected because it can feel tedious, but organizations pay consequences when incidents inevitably strike.
The team discovered they could make system testing and maintenance exciting through strategic chaos: planned opportunities to break things. Beyond the enjoyment of diverging from typical repair work, it creates unique realistic situations never discovered through traditional maintenance approaches.
Teams are encouraged to break their own systems, restore them, and repeat. Each iteration improves processes and tooling for inevitable stressful situations. Remember to celebrate World Backup Day every March 31st.
Acknowledgments: Kyle Sammons, Mark Carey, Renning Bruns, Emma Montross, Shelly Wu, Bryan Burkholder, George Luong, and Ryan Katkov contributed to the planning, execution, and support of the recovery exercise.