DB performance issue

Status: Resolved

Incident Timeline

Jul 07, 2015 - 09:55 UTC We are experiencing a performance issue on a DB which is affecting the whole platform.

Jul 07, 2015 - 16:25 UTC We have all resources focused on solving the issue with the DB.

Jul 07, 2015 - 18:39 UTC We’re deploying new, better DB hardware, and monitoring as it’s spinning up.

Jul 07, 2015 - 19:40 UTC We’re monitoring closely and seeing some fluctuations in the queues as we get the workarounds into place.

Jul 07, 2015 - 20:45 UTC We’ve cleared the queue and we are processing builds as they arrive right now.

Jul 07, 2015 - 21:45 UTC We’re going to switch over to the new DB hardware, which will cause builds to requeue.

Jul 07, 2015 - 22:26 UTC The switch to the new DB hardware is complete. Builds are running again.

Jul 08, 2015 - 00:10 UTC through 15:32 UTC A series of updates confirmed queue monitoring and build clearing continued through resolution.

Root Cause

A sudden surge of GitHub webhook activity overwhelmed the database, causing lock-up and cascading system failures.