June 15: Machines API Outage
(15:07UTC)
Our token service tkdb became unavailable, which broke our macaroon token verification path. This caused a broad control-plane outage for approximately an hour: dashboard login/SSO and many Machines API operations failed, while existing app traffic continued to route normally.
On this day, tkdb needed to be migrated between two internal hosts. Many Machines API operations rely on tkdb for macaroon minting and verification, and tkdb is itself a Fly App that relies on the Machines API.
Migrating tkdb is something we have a known, written procedure for. The general idea is to:
- Cordon the tkdb primary
- Stand up a new Machine, let it catch up as a replica, then promote it to primary
- Point all replicas at the new primary
During this window, minting new macaroons will be unavailable, but verifications will still work and the Machines API will remain functional.
A while back, a fly-proxy change inadvertently shifted the semantics of Machine cordoning. The new cordoning flow meant that cordoned Machines would not ever be load balanced to, but they were still eligible for requests that specifically name them. That is, requests using fly-replay to an instance, or the fly-force-instance-id header, would now each arrive at a cordoned Machine.
Because of that, when we cordoned the primary tkdb Machine, it would still receive replays from replicas. Additionally, stopping the primary wouldn’t work, as these same replays would auto-start the targeted Machine. The right decision here would have been to restore and uncordon this primary, and pause the migration. The less-right decision, which is what we did, was to destroy the old primary before promoting the new one. In theory this is roughly equivalent, and is a safe way to migrate the broadly similar petsem cluster, which is why it seemed reasonable.
The (overlooked) quirk for tkdb is that in order to guarantee a bounded lag on token revocation, replicas that fall too far behind the primary will throw their toys and proceed to replay all requests to the primary, rather than use a stale revocation list. In practice this meant that we were suddenly left with a cluster of replicas that would not function, which took out the Machines API for customers and for ourselves. Without the Machines API, we were unable to easily bring up the new primary Machine required to resolve this.
From here we had to resort to spinning up the required Machine by hand with flyd on a host, which has its own set of internal hurdles to overcome. The main responder in this incident was based in Europe, which additionally meant that their Machines API operations would prefer flaps, and thus tkdb, in Europe rather than the new primary in North America. These factors weren’t a problem in the initial incident, but they did complicate and slow the response for getting everything back up and running.
The technical root cause here is the cordoning flaw in fly-proxy, and the organizational root cause is that our documentation and runbooks for tkdb were not adequate to prevent this from happening. Both of these things are being fixed: In fly-proxy, cordoning will be fixed to prevent all proxy-routed traffic, and our internal documentation is being reviewed and improved for tkdb, and more broadly where other services have the same gap.