Railway Incident Report: January 28-29, 2026

Published: January 26, 2026 by Angelo Saraceno

Overview

Railway experienced intermittent GitHub authentication failures over multiple days due to exceeding GitHub’s OAuth token rate limit of 2,000 tokens per hour. Users encountered login failures and “GitHub repo not found” errors during peak US business hours.

Impact

The incident affected three main user workflows:

• GitHub authentication login attempts
• Connecting new GitHub repositories to projects
• Deploying from GitHub repositories

Failures followed an hourly pattern, with rate limits exhausted approximately 30 minutes into each hour.

Root Cause

The underlying issue stemmed from Railway’s rapid growth combined with unsustainable API usage patterns. Specifically, the dataloader creating GitHub installation access tokens “was not caching tokens across requests, resulting in ~82 new tokens per second during peak usage.”

Contributing factors included:

• Recent backend replica scaling that worsened the problem
• Multiple code paths making redundant GitHub API calls
• Frequent automated repository syncing workflows
• Security-driven real-time permission verification checks
• Secondary rate limits triggered after initial errors

Timeline

January 26th: Initial reports at 18:57 UTC; root cause identified by 20:23 UTC involving uncached token creation; initial fix merged by 21:14 UTC but rate limiting persisted.

January 27th: Degradation worsened; additional optimizations deployed; GitHub Support contacted.

January 28th: Background environment refresh identified as contributor; GitHub OAuth logins temporarily disabled at 21:16 UTC; GitHub Support confirmed OAuth token requests exceeded limits at 23:26 UTC; logins re-enabled after mitigations.

January 29th: Minor rate limiting resumed during peak hours; new incident declared; GitHub App permissions updated to receive additional webhook events.

Preventative Measures

Railway implemented:

• Installation token caching across full validity periods (reducing creation by over 90%)
• Consolidated deployment status updates
• Migration from user OAuth tokens to installation tokens where appropriate
• Cached permission checks using internal databases
• Switched automatic repository sync to manual customer-initiated sync
• Enhanced monitoring and alerting for login, repository, and rate-limiting issues