Sleeping beauty - How one of the oldest bugs in EVE got hunted down and resolved

Published: 2017-02-15 By: CCP Snorlax

How it all began

CCP karkur investigated an issue where the drone window would stop updating when drones were being recalled. Her investigation revealed the problem wasn’t in drone or UI code but rather in the tasklet system itself. The drone window would “start updating, then went to sleep for a bit, but never woke up again.” Tasklets were occasionally failing to return from their sleep state.

To reproduce the issue reliably, CCP karkur created a script generating 500,000 tasklets that sleep for specified durations with tracking to identify tasklets that never finish. The script revealed tasklets stuck in limbo—neither scheduled nor blocked.

Some background

Tasklets (from Stackless Python) run via PyScheduler::Run by calling PyStackless_RunWatchdogEx with a timeout value. When created, tasklets are scheduled. When sleeping, tasklets call Synchro::SleepWallclock, which creates a channel and calls receive on it, blocking until a value is sent. Synchro maintains a heap of sleepers—objects containing the wake-up time and channel. Synchro::Tick pulls due sleepers from the heap and sends values on their channels, unblocking tasklets and rescheduling them.

Initial investigation considered bugs in the heap code itself, but symptoms didn’t match a tasklet remaining permanently blocked on the heap.

Show me the code

The author examined code around tasklet killing and reference counting. SleepWallclock creates a channel with reference count one, stores it in a Sleeper object on the heap, then calls receive. When Synchro::Tick processes the sleeper, it sends on the channel and calls Py_DECREF, deleting the channel object and freeing its memory.

The Eureka moment

The critical issue: “When a channel object is deleted, its memory is now free and available for reuse.” If a tasklet kills exactly when another tasklet wants to sleep, they might receive the same channel pointer. RemoveSleeper would then remove the wrong tasklet from the heap, leaving a new tasklet in a non-scheduled, non-blocked state.

The fix involved moving Py_DECREF calls into SleepWallclock, preventing premature channel memory recycling and localizing reference counting.

Never assume

Despite EVE Online’s nearly 14-year history and thoroughly tested codebase, fundamental bugs can hide in plain sight. The authors suspected this underlying issue caused other mysterious bugs—overview not updating, shield/armor/structure bar glitches—especially after mass tests or heavy fights.