In November, GitHub experienced one incident that resulted in degraded performance affecting GitHub’s notification service.
Incident: November 19, 2024 (10:56–12:03 UTC, lasting 1 hour 7 minutes)
Impact: The notifications service experienced degradation, halting notification delivery to all dotcom customers. The average delay was approximately one hour.
Root Cause: A database host came out of a regular maintenance process in read-only mode, preventing the notifications service from functioning properly.
Resolution: The team restored write capability to the affected database host. This enabled notification delivery to resume and allowed previously failed delivery jobs to be retried. Full recovery was achieved by 12:36 UTC.
Preventive Measures: GitHub is working to improve observability across database clusters to reduce time to detection and improve system resilience on startup.