Outage Postmortem - January 24 2017

On January 24, 2017 starting at 17:53 UTC, we experienced system degradation (read only) followed by a site outage. The Stack Exchange network went into read-only mode for approximately 5 minutes before being offline for 12 minutes. It took us 2 minutes to notice the issue, 5 minutes to locate the source of the issue and 10 minutes to get service restored. Approximately 3.5 seconds worth of data may have been lost as uncommitted transactions were rolled back.

The cause of the issue was our primary SQL Server (NY-SQL02) triggering a bugcheck on the SQL Server process. This initially put the SQL server into a read-only state. We do not permit automatic SQL Server failovers, instead opting for application-level logic to keep the sites online by querying the standby SQL servers in read-only mode. However, due to a bug, application-level failovers were disabled, so Stack Overflow and the rest of the Stack Exchange network went offline completely when the primary SQL server failed. A restart of the SQL service on NY-SQL02 brought the network back online in read only mode. Once a sanity check on the SQL health was done, the sites were put back into read-write mode.

Currently we do not know why SQL Server caused a bugcheck, however logs indicate a potential bad DIMM. At our memory capacity (384GB), we opted to bring the network online rather than waiting for the memory dump to complete. We have since taken NY-SQL02 out of production and have it down for testing (including a memory test). All of the normal load is being handled by NY-SQL01. We also took this opportunity to bring this SQL cluster up to 2016 SP1 CU1.

January 26, 2017 (6:45 pm)