August 1: IAD MPGv2 Patroni etcd lease-expiration storm

Incident start: 16:40UTC

MPG v2 in the IAD region saw a burst of Patroni health-check failures and timeouts when the region’s shared Patroni etcd cluster became overloaded. Existing databases were unaffected because Patroni entered failsafe mode, but new clusters could not be created. Investigation pointed to an intrinsic etcd problem: adding a new user bumps etcd’s Auth Revision, invalidating existing JWT tokens and forcing clients to reconnect. On reconnect, a thundering herd of clients overloaded etcd and brought it down. We mitigated by scaling up the Patroni etcd machines in IAD and lowering the bcrypt cost, and we’re exploring longer-term fixes.