Jobs not starting

Status: Resolved

Incident Timeline

18:19 UTC, Mar 14 — We are seeing a delay in starting jobs. We are currently investigating and will update shortly.

18:58 UTC, Mar 14 — Currently jobs are delayed in starting and not completing.

19:20 UTC, Mar 14 — We are continuing our investigation.

19:52 UTC, Mar 14 — We are continuing to investigate this issue. Thank you for your patience while we work toward a resolution.

20:19 UTC, Mar 14 — We are continuing to investigate this issue.

20:46 UTC, Mar 14 — We are continuing to investigate this issue.

21:09 UTC, Mar 14 — We are continuing to see degradation on our services including jobs not starting and UI impacts.

21:46 UTC, Mar 14 — Our debugging efforts have led to partial recovery of some internal services.

22:13 UTC, Mar 14 — We are seeing intermittent successes in the UI as some components have been recovered.

22:28 UTC, Mar 14 — We are seeing jobs running again, and are adding additional capacity.

22:59 UTC, Mar 14 — We are continuing to add capacity to process the backlog of jobs.

23:39 UTC, Mar 14 — UI and API access has mostly recovered. We are still working through capacity issues.

00:14 UTC, Mar 15 — UI access remains stable. We are still tuning capacity and resources to process backlogs.

01:03 UTC, Mar 15 — Docker jobs with contexts are starting to run successfully.

01:27 UTC, Mar 15 — Docker and Mac jobs are now becoming operational.

01:34 UTC, Mar 15 — Machine jobs are now running, we expect the backlog to clear over the next hour.

01:59 UTC, Mar 15 — This incident has been resolved. Docker, Mac, Machine jobs should be running correctly.