Executive Summary
At 3:53 UTC on August 28, 2025, a Kafka failure triggered cascading issues disrupting event processing for some US-region customers. A second incident occurred at 16:38 UTC the same day. All systems returned to normal by 10:10 UTC (first incident) and 20:24 UTC (second incident). No data was lost.
Incident Narrative
The First Incident (03:53 UTC)
A logical error in a newly deployed API usage tracking feature caused the system to instantiate a new Kafka producer for every API request instead of reusing a single producer. This generated approximately 4.2 million new producers per hour at peak—84 times the typical rate.
The excessive producer metadata tracking caused severe memory pressure on Kafka broker JVM heaps, leading to garbage collection thrashing and eventual heap exhaustion. The issue cascaded across the entire cluster, blocking downstream services from accessing Kafka entirely.
Teams initially suspected isolated hardware failure on a single broker and expanded the cluster. Upon realizing the systemic nature, they doubled the JVM heap size and performed rolling restarts. Recovery of dependent services was prolonged due to cascading failures and complex interdependencies.
Duration: approximately 6 hours 17 minutes until full restoration at 10:10 UTC.
The Second Incident (16:38 UTC)
The team responsible for the API usage feature, unaware their code change triggered the first outage, proceeded with rolling out the feature from 25% to 75% of traffic on August 28. This higher load rapidly re-exhausted the newly expanded Kafka heap memory within approximately two hours.
Responders applied lessons from the first incident, implementing the same mitigation steps (doubling heap, rolling restart) much faster.
The correlation between feature rollout and heap growth became evident. Engineers identified the logical error in the code that resulted in a new producer being created for every request, and rolled back the feature by disabling Kafka message publishing.
Duration: approximately 50 minutes for initial mitigation; full restoration by 20:24 UTC.
Technical Details
Kafka’s Role in PagerDuty Infrastructure
Kafka functions as the backbone of asynchronous processing, enabling decoupled services to communicate through event streaming. Producers publish messages to topics; Kafka brokers maintain metadata for each active producer, tracking transactions and message sequences using unique producer IDs.
The Triggering Feature
PagerDuty was implementing enhanced API key usage tracking and observability for governance purposes. The feature involved publishing API usage events to a new Kafka topic, estimated at hundreds of millions of requests daily.
The rollout occurred incrementally: 1% on August 21, 5% on August 27, 25% by end of August 27, and 75% on August 28—both incidents correlated directly with rollout increases.
Why the Feature Failed
The implementation used the pekko-connectors-kafka Scala library, passing Kafka settings directly rather than a pre-created producer instance. Every time send() is called, a new producer is created instead of reusing an existing one. The lack of explicit allocation hints made this a blind spot visually due to the lack of an explicit new operator.
Why Recovery Was Complex
Multiple factors complicated restoration:
- Message Processing Complexity: Services exhibited unexpected behaviors; some required manual restarts to handle changes. Certain activities generated much higher message volumes than expected.
- Cascading Dependencies: Services tightly coupled to Kafka created chain reactions when the message system became unavailable. No single engineer on the call held a mental model of our entire system.
- Observability Gaps: Insufficient monitoring of Kafka producer/consumer telemetry, JVM heap usage, and producer tracking delayed problem identification.
- Transactional Outbox Pattern: While this pattern prevented data loss, it created massive backlogs requiring downstream reprocessing, delaying full system recovery.
Customer Impact Summary
Event Processing:
- API event creation: 18.87% returned 5xx errors (peak)
- API event updates: 4.35% returned 5xx errors
- Email ingestion: ~16% delayed, <1% not processed
- Change events: ~6.5% rejected
Notifications: Approximately 23% delayed by 5+ minutes; none completely dropped.
Integrations: Jira, ServiceNow, Salesforce, and Zendesk experienced delayed or dropped events for ~100 minutes.
Webhooks: Delayed, dropped, or duplicated deliveries (~100 minutes).
Mobile & Chat: 6.06% of iOS/Android users unable to acknowledge or resolve incidents; duplicate chat messages sent with delays lasting ~515 minutes.
Status Communication: Public status page updates delayed ~100 minutes due to automation failure in external communication processes.
Contributing Factors
- 84x increase in Kafka producer connections
- Observability gaps on tracked producers and JVM heap usage
- Lack of anomaly detection for unexpected workloads
- Feature rollout via configuration rather than distinct deployment obscured the connection to outages
- Infrequently-used framework/library with non-obvious producer creation behavior
Communication Challenges
Early in the incident, our automated publishing to the public status page did not execute. Teams relied on manual updates, which created delays. Additionally, internal and external communications were not well-aligned, causing confusion about outage status.
Mitigating Factors
Team Response: Over one-third of PagerDuty’s global engineering organization engaged, even outside normal hours. When the second incident occurred, teams applied lessons from the initial response to achieve mitigation in just 50 minutes.
Pre-built Tools: Existing backfill scripts and pause/resume runbooks protected critical data and gave responders confidence despite imperfect procedures.
Audit Trails: Code-based deployment practices maintained clear records, supporting rapid troubleshooting and post-incident analysis.
Identified Gaps and Learnings
Surprises
- System message processing complexity exceeded initial understanding
- Difficulty pinpointing which systems created heavy Kafka loads
- Status page automation dependencies created single points of failure
Key Difficulties
- Alert Fatigue: Critical system alerts were obscured by webhook notifications—18 of 19 high-priority pages involved webhooks, causing teams to miss core API errors.
- Recovery Gaps: Several applications required manual intervention after Kafka restoration. ksqlDB stream management runbooks lacked clear verification steps and error handling guidance, requiring untested backfill scripts that extended recovery.
- Framework Blind Spots: The Scala library’s producer creation behavior wasn’t immediately obvious without explicit allocation syntax.
Remediation Plan
Detect
- Expand JVM and Kafka monitoring (heap, garbage collection, producer/consumer health)
- Strengthen service dependency mapping
- Automate collection of customer impact metrics into incident workflows
Prevent
- Add stricter change management guardrails, readiness checklists, and slower ramp-up windows post-incident
- Improve recovery mechanisms in streaming systems
Adapt
- Automate status page updates and internal/external communication workflows
- Conduct monthly chaos engineering drills for major incident procedures
- Introduce clearer incident severity levels and role definitions with dedicated communications liaisons
Conclusion
Cascading failures are inherently hard to predict, and a small issue in one service can ripple into others in ways not obvious from system diagrams. We take these incidents extremely seriously, and are committed to transparent improvement: everyone makes mistakes, but it’s how we react and learn that defines us.
Note: An edit on September 5, 2025 at 16:06 UTC updated images for clarity, added a Scala inconsistency section, elaborated on status page communications, and corrected typos.