postmortem.io Beta
Incident

Issue with uneven routing distribution

Heroku Read the original status page ↗

Performance Degradation & Service Disruption on Heroku Common Runtime on July 18, 2025, until July 23, 2025

Root Cause Analysis

Published on August 13, 2025

This article and other items we publish on the subject, including through social media outlets, may contain forward-looking statements, the achievement or success of which involves risks, uncertainties, and assumptions. If any such risks or uncertainties materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed or implied by the forward-looking statements we make.

We sincerely apologize for any impact this incident may have caused you and your business. At Salesforce, trust is our #1 value and security is our top priority. We value transparency and want to take this opportunity to outline the facts regarding a recent service disruption that may have disrupted your ability to use multiple Salesforce services, as we currently understand them. Our investigation is ongoing, and we will provide customers with updated information in the future.

The information contained in this document is provided by Salesforce for general information purposes only and is based on information as of the date of distribution and is subject to change. The incident timeline displayed within this Salesforce Technology Service Delivery Root Cause Analysis reflects the incident investigation timeline (inclusive of remediation and monitoring). While it includes any period during which the Service was unavailable, the investigation, remediation, and monitoring period is generally much longer than the period of unavailability, if any, caused by the disruption.

Executive Summary - What Happened?

Beginning July 18, 2025, at approximately 22:00 Coordinated Universal Time (UTC) until July 23, 2025, at 00:12 UTC, some customer applications using Router 2.0 on Heroku’s Common Runtime experienced uneven balancing of HTTP requests across web dynos. The uneven balancing, measured as the percentage variance in HTTP requests per dyno, grew steadily. This scenario resulted in increased HTTP request latencies and timeouts. At 21:01 UTC on July 22, 2025, the Heroku Technology team (Technology team) identified a load balancing bug in the router and a fix was prepared for deployment. At 00:12 UTC on July 23, 2025, the rollout to all partitions of Heroku’s Common Runtime infrastructure was completed, fully restoring proper load balancing.

How did this issue impact Heroku services?

Some customer applications deployed on Heroku’s Common Runtime experienced uneven load balancing of HTTP requests across web dynos. The uneven balancing caused some dynos to receive more HTTP requests than others and become overloaded. This scenario led to applications suffering various symptoms, including:

  • Reduced availability, service degradation
  • Increased HTTP request latencies
  • Various Heroku Error codes in application logs and dashboards

Technical Details

Detection and Initial Impact

  • July 18, 2025, at ~22:00 UTC: Router 2.0, the default Common Runtime router, began unevenly balancing HTTP requests across applications’ dynos.
  • July 20, 2025, at 16:46 UTC: Heroku Support received first customer indications reporting uneven balancing of HTTP requests to dynos, resulting in increased request latencies and timeouts.
  • July 22, 2025, at 14:35 UTC: The Technology team recognized the issue was unique and an internal investigation was declared. The Support team began advising affected customers to disable Router 2.0, falling back to the legacy router. The Technology team investigated router dyno state, believing the root cause may be related to an incomplete state.
  • July 22, 2025, at 17:01 UTC: The Technology team abandoned the dyno state theory, dismissing initial false leads.
  • July 22, 2025, at 17:48 UTC: Internal investigation was escalated in severity and a formal incident was declared.
  • July 22, 2025, at 17:53 UTC: The Technology team recognized the same uneven HTTP request distribution was present in staging, according to logs for the router’s continuous load testing. Engineers began exploring load balancing metadata and its impact on load balancing selection as the root cause.
  • July 22, 2025, at 18:36 UTC: The Technology team posted a formal incident notice to the Heroku status site.
  • July 22, 2025, at 19:42 UTC: The Technology team determined that automatic cycling of router instances in staging improves load balancing for staging applications. The team initiated cycling of production router fleets, improving load balancing for impacted customer applications.
  • July 22, 2025, at 20:05 UTC: The Technology team identified the root cause in the broken load balancing entropy metadata.
  • July 22, 2025, at 21:01 UTC: The Technology team reproduced the broken behavior with a new load balancing benchmark test and prepared a code change to evenly balance HTTP requests across customer app dynos, passing the new benchmark test.

Remediation

  • July 22, 2025. at 21:13 UTC: The Technology team integrated and deployed the router change to the staging environment.
  • July 22, 2025 at 21:30 UTC: The staging deployment of the code change completed and the technology team monitored the effect on staging applications. The change was deemed safe and successful.
  • July 22, 2025, at 21:38 UTC: The Technology team initiated an expedited production deployment of the remediation. The impact on customers was further mitigated as the change rolled out.
  • July 23, 2025, at 00:12 UTC: HTTP request load balancing remediation was deployed to final partition and the incident was resolved.

Root Cause Analysis

The Technology team’s post-incident investigation and analysis determined that the incident was triggered by broken entropy metadata used in load balancing algorithms in Router 2.0, the new default router in Heroku’s Common Runtime.

The load balancing logic of Router 2.0 relies upon entropy metadata to distribute dyno selection. On July 18 at 22:00 UTC, router instances began hitting a code condition that caused the metadata and corresponding logic to reduce dyno selection variance. Once this condition in the router was hit, a router instance’s load balancing logic would remain broken for all subsequent HTTP requests. This did not completely break the router’s handling of HTTP requests, but rather broke the even distribution of HTTP requests over an application’s set of web dynos. The effects of the broken load balancing logic were slowly exacerbated over time as more router instances hit the code condition. This code condition was not introduced recently–it has been part of the router codebase since the Router 2.0 beta program began. However, hitting the condition became more likely as the number of HTTP requests served increases. With the Router 2.0 migration nearing completion (over ~94% of Common Runtime apps are utilizing Router 2.0), more HTTP requests are using the new router than ever before.

The Technology team determined during the incident that this router bug was affecting the load tests in staging environments. Poor performance in the load testing setup causes alerts to fire and deployments to block. However, poor performance is captured by reduced routing availability and increased HTTP request latencies. HTTP request distribution is not a signal captured by the current alerting. The HTTP request distribution issue in staging did not strain test applications enough to impact availability or HTTP request latencies significantly. Therefore, no alerts fired, and the Technology team was unaware of the issue until it began impacting customers.

Next Steps

To maintain the performance level that our customers expect from Salesforce and to prevent this defect from recurring, our focus is on continuous improvement. The Technology team has identified and is implementing the following actions:

Complete:

  • Fixed the broken load balancing logic in the router and completed a production roll-out of that fix.
  • Introduce gating and notifications to ensure correct load balancing behavior regressions do not occur.

In Progress:

  • Actively communicate the timeline and reminders for remaining legacy router application customers before resuming automated migrations.

We sincerely apologize for the impact this incident may have caused you and your business; Salesforce is fully committed to minimizing downtime when incidents do occur. We also continually assess and improve our tools, processes, and architecture to provide you with the best service possible.

The KA article is now available for the RCA: https://help.salesforce.com/s/articleView?id=005123907&type=1