Northeast blackout of 2003

The Northeast blackout of 2003 was a widespread power outage throughout parts of the Northeastern and Midwestern United States, and most parts of the Canadian province of Ontario on Thursday, August 14, 2003, beginning just after 4:10 p.m. EDT.

Most places restored power by midnight (within 7 hours), some as early as 6 p.m. on August 14 (within 2 hours), while the New York City Subway resumed limited services around 8 p.m. Full power was restored to New York City and parts of Toronto on August 16. At the time, it was the world’s second most widespread blackout in history, after the 1999 Southern Brazil blackout. The outage, which was much more widespread than the Northeast blackout of 1965, affected an estimated 55 million people, including 10 million people in southern and central Ontario and 45 million people in eight U.S. states.

The blackout was due to a software bug in the alarm system at the control room of FirstEnergy, which rendered operators unaware of the need to redistribute load after overloaded transmission lines dropped in voltage. What should have been a manageable local blackout cascaded into the collapse of much of the Northeast regional electricity distribution system.

Immediate impact

According to the New York Independent System Operator (NYISO)—the ISO, responsible for managing the New York state power grid—a 3,500 megawatt power surge (towards Ontario) affected the transmission grid at 4:10:39 p.m. EDT.

For the next 30 minutes, until 4:40 p.m. EDT, outages were reported in parts of Michigan (Detroit), Ohio (Cleveland, Akron, Toledo), Ontario (Toronto, Hamilton, London, Windsor), New Jersey (Newark), and New York (New York City, Suffolk, Nassau, Westchester, Orange and Rockland counties, Rochester, Syracuse, Binghamton, Albany).

This was followed by outages in other areas initially unaffected, including all of New York City, portions of southern New York state, New Jersey, Vermont, Connecticut, as well as most of the province of Ontario. Eventually, a large, somewhat triangular area bounded by Lansing, Michigan, Sault Ste. Marie, Ontario, the shore of James Bay, Ottawa, New York, and Toledo was left without power.

According to the official analysis of the blackout prepared by the US and Canadian governments, more than 508 generating units at 265 power plants shut down during the outage. In the minutes before the event, the NYISO-managed power system was carrying 28,700 MW of load. At the height of the outage, the load had dropped to 5,716 MW, a loss of 80%.

Essential services remained in operation in some of these areas. In others, backup generation systems failed. Telephone networks generally remained operational, but the increased demand triggered by the blackout left some circuits overloaded. Water systems in several cities lost pressure, forcing boil-water advisories to be put into effect. Cellular service was interrupted as mobile networks were overloaded with the increase in volume of calls. Multiple cell sites were out of commission due to power outages. Television and radio stations remained on the air, with the help of backup generators, although some stations were knocked off the air for periods ranging from several hours to the length of the entire blackout.

High heat played a role in the initial event that triggered the wider power outage. Much of the affected region reached temperatures higher than 31°C (88°F) that day, increasing energy demand as people across the region turned on fans and air conditioning. This caused the power lines to sag as higher currents heated the lines.

Most of the Amtrak Northeast Corridor service was interrupted, as it relied on electricity for its signaling and crossing systems; electrified commuter railways also shut down. Via Rail in Canada was able to continue most of its service. All airports in the affected area closed immediately, there were no departures, and incoming flights had to be diverted to airports with power.

The reliability of the electrical grid was called into question and required substantial investment to repair its shortcomings.

Duration

Most places restored power by midnight, as early as 6 p.m. on August 14, and the New York City Subway had resumed limited services around 8 p.m. Some areas lost power for only four to eight hours; these are: Albany and parts of Long Island in New York; three-quarters of New Jersey; parts of Pennsylvania, Ohio and Michigan; New London County, Connecticut; parts of downtown Toronto, Mississauga, and London in Ontario; portions of western Ottawa in Ontario, including Kanata and south to Kingston; a number of areas of the Regional Municipality of Niagara in Ontario; and parts of Southwestern Ontario, particularly areas near the Bruce Nuclear Generating Station.

By the next morning (August 15), some areas of Manhattan regained power around 5:00 a.m.; Staten Island regained power around 3:00 a.m. Half of the affected portions of Ontario regained power by the morning. By early evening of August 15, two airports, Cleveland Hopkins International Airport and Toronto Pearson International Airport, were back in service. By August 16, power was fully restored in New York and parts of Toronto. 500,000 Detroit Edison customers were still without power at 10:00pm August 15; all were restored by 6:30am August 16.

Causes

Investigation efforts

A joint federal task force was formed by the governments of Canada and the U.S. to oversee the investigation and report directly to Ottawa and Washington. The task force was led by then-Canadian Natural Resources Minister Herb Dhaliwal and U.S. Energy Secretary Spencer Abraham.

In addition to determining the initial cause of the cascading failure, the investigation of the incident also included an examination of the failure of safeguards designed to prevent a repetition of the Northeast blackout of 1965. The North American Electric Reliability Corporation, a joint Canada-U.S. council, is responsible for dealing with these issues.

On November 19, 2003, Abraham said his department would not seek to punish FirstEnergy Corp for its role in the blackout because current U.S. law does not require electric reliability standards. Abraham stated, “The absence of enforceable reliability standards creates a situation in which there are limits in terms of federal level punishment.”

Findings

In April 2004, the U.S.-Canada Power System Outage Task Force released their final report, placing the causes of the blackout into four groups:

1. FirstEnergy (FE) and its reliability council “failed to assess and understand the inadequacies of FE’s system, particularly with respect to voltage instability and the vulnerability of the Cleveland-Akron area, and FE did not operate its system with appropriate voltage criteria.”
2. FirstEnergy “did not recognize or understand the deteriorating condition of its system.”
3. FirstEnergy “failed to manage adequately tree growth in its transmission rights-of-way.”
4. Finally, the “failure of the interconnected grid’s reliability organizations to provide effective real-time diagnostic support.”

The report states that a generating plant in Eastlake, Ohio, a suburb northeast of Cleveland, went offline amid high electrical demand, putting a strain on high-voltage power lines (located in Walton Hills, Ohio, a southeast suburb of Cleveland) which later went out of service when they came in contact with “overgrown trees”. This trip caused load to transfer to other transmission lines, which were not able to bear the load, tripping their breakers. Once these multiple trips occurred, a number of generators suddenly lost parts of their loads, so they accelerated out of phase with the grid at different rates, and tripped out to prevent damage. The cascading effect that resulted ultimately forced the shutdown of at least 265 power plants.

Computer failure

A software bug known as a race condition existed in General Electric Energy’s Unix-based XA/21 energy management system. Once triggered, the bug stalled FirstEnergy’s control room alarm system for over an hour. System operators were unaware of the malfunction. The failure deprived them of both audio and visual alerts for important changes in system state.

Unprocessed events queued up after the alarm system failure and the primary server failed within 30 minutes. Then all applications (including the stalled alarm system) were automatically transferred to the backup server, which itself failed at 14:54. The server failures slowed the screen refresh rate of the operators’ computer consoles from 1–3 seconds to 59 seconds per screen. The lack of alarms led operators to dismiss a call from American Electric Power about the tripping and reclosure of a 345 kV shared line in northeast Ohio. But by 15:42, after the control room itself lost power, control room operators informed technical support (who were already troubleshooting the problem) of the alarm system problem.

Outages in the Cleveland-Akron area

The cascading failure began in the FE-controlled Cleveland–Akron area; the area was significant because it was a “transmission-constrained load pocket with relatively limited generation.” On August 14, a number of power generators in and around the area were offline. This reduced the available resources for voltage control and reactive power management in contingencies but did not affect normal operation. Four or five of the area’s capacitor banks—also used to manage reactive power—were also offline for routine maintenance; this did not follow the best practice of performing such maintenance during low-demand periods, and their status was not reported for regional planning purposes because FE had deemed them to be non-critical infrastructure.

Incidents contributing to the blackout began after noon on August 14. FE’s reliability coordinator was the Midwest Independent System Operator (MISO). MISO maintained a regularly updated model of its area of responsibility with a state estimator (SE). At 12:15 p.m., the SE was mistakenly shutdown after producing anomalous results due to old data. Unscheduled grid outages followed. In the Cleveland-Akron area, Eastlake Power Plant unit 5 went offline at 1:31 p.m. In Southern Ohio, Dayton Power and Light’s (DPL) Stuart-Atlanta transmission line tripped from tree contact at 2:02 p.m. The outages did not affect normal operation. However, the loss of the Eastlake unit put the FE system at risk of overloads in certain contingency scenarios; FE did not notice the danger. At the time of the Eastlake unit outage, FE was asking for and receiving significant additional voltage support, but the situation was not unprecedented.

FE’s situational awareness was reduced by the development of faults in its General Electric Harris XA21 energy management system (EMS) from 2:14 p.m. to 2:54 p.m. The alarm and logging software in the control room failed first, followed by some remote terminals and data links, then the primary and backup EMS servers. Control room operators were unaware of the alarm failure for over an hour. Operators began to suspect a failure after receiving a notification from American Electric Power (AEP) at 2:32 p.m. about a short AEP line outage that the FE EMS had failed to raise an alert for. FE IT was automatically notified of the remote terminal and server failures. IT performed a “warm reboot” of the primary server at 3:08 p.m. but did not realize that the alarm function remained broken. The control room notified IT of the alarm problem at 3:42 p.m. A “cold reboot”—later discovered to be the vendor-recommended action—was rejected because the power system situation was precarious and operators were concerned about the greater loss of functionality during the reboot procedure. The servers were offline again at 3:46 p.m. to 3:59 p.m. as they were rebooted in another attempt to re-enable the alarms. FE failed to notify MISO of its degraded situational awareness.

Cleveland-Akron area transmission lines began failing shortly after 3:00 p.m. Three 345 kV transmission lines failed between 3:05 p.m. to 3:42 p.m. due to tree contact. The first was the Harding-Chamberlin line. Most of the load from that line was transferred to the Hanna-Juniper line, which then failed at 3:32 p.m. PJM and AEP began mitigation efforts to prevent the Star-South Canton line connecting FE and AEP from overloading; they had incomplete data on the status of FE’s system and underestimated the severity of the situation. The Star-South Canton line failed at 3:41 p.m. after its loading increased from 82% to 120% after the failure of the Hanna-Juniper line. The failures in the MISO SE and FE EMS prevented either from recognizing and responding to the failures. FE observed the loss of voltage after 3:32 p.m, but it was not until 3:45 p.m. that it recognized an emergency was occurring. Next, the 138 kV lines failed between 3:39 p.m. to 4:08 p.m. The loss of voltage shut down industrial equipment used by customers and blacked out Akron and its surroundings.

The MISO SE was reactivated at 2:40 p.m, but the results confused operators because the model did not match empirical data. The discrepancy was caused by the SE being unaware of the DPL line outage at 2:02 p.m; DPL’s reliability coordinator was PJM, so DPL did not automatically report data to MISO. MISO operators had the SE modelling correctly again by 4:04 p.m. It was too late for MISO to address FE’s critical status; the cascading failure started minutes later.

Cascade

Timeline

The following is the blackout’s sequence of events on August 14, 2003 (times in EDT):

• 12:15 p.m. Incorrect telemetry data renders inoperative the state estimator, a power flow monitoring tool operated by the Indiana-based Midwest Independent Transmission System Operator (MISO). An operator corrects the telemetry problem, but forgets to restart the monitoring tool.
• 1:31 p.m. The Eastlake, Ohio generating plant shuts down. The plant is owned by FirstEnergy, an Akron, Ohio-based company.
• 2:02 p.m. The first of several 345 kV overhead transmission lines in northeast Ohio fails due to contact with a tree in Walton Hills, Ohio.
• 2:14 p.m. An alarm system fails at FirstEnergy’s control room and is not repaired.
• 3:05 p.m. A 345 kV transmission line known as the Chamberlin-Harding line sags into a tree and trips in Parma, south of Cleveland.
• 3:17 p.m. Voltage dips temporarily on the Ohio portion of the grid. Controllers take no action.
• 3:32 p.m. Power shifted by the first failure onto another 345 kV power line, the Hanna-Juniper interconnection, causes it to sag into a tree, bringing it offline as well. While MISO and FirstEnergy controllers concentrate on understanding the failures, they fail to inform system controllers in nearby states.
• 3:39 p.m. A FirstEnergy 138 kV line trips in northern Ohio.
• 3:41 p.m. A circuit breaker connecting FirstEnergy’s grid with that of American Electric Power is tripped as a 345 kV power line (Star-South Canton interconnection) and fifteen 138 kV lines fail in rapid succession in northern Ohio.
• 3:46 p.m. A fifth 345 kV line, the Tidd-Canton Central line, trips offline.
• 4:05:57 p.m. The Sammis-Star 345 kV line trips due to under-voltage and over-current interpreted as a short circuit. (Later analysis suggests that the blackout could have been averted before this failure by cutting 1.5 GW of load in the Cleveland–Akron area.)
• 4:06–4:08 p.m. A sustained power surge north toward Cleveland overloads three 138 kV lines.
• 4:09:02 p.m. Voltage sags deeply as Ohio draws 2 GW of power from Michigan, creating simultaneous under-voltage and over-current conditions as power attempts to flow in such a way as to rebalance the system’s voltage.
• 4:10:34 p.m. Multiple transmission lines trip out, first in Michigan and then in Ohio, blocking the eastward flow of power around the south shore of Lake Erie from Toledo, Ohio, east through Erie, Pennsylvania, and into southern Erie county, but not most of the Buffalo metropolitan area. Suddenly bereft of demand, generating stations go offline, creating a huge power deficit. In seconds, power surges in from the east, overloading east-coast power plants whose generators go offline as a protective measure, and the blackout is on.
• 4:10:37 p.m. The eastern and western Michigan power grids disconnect from each other. Two 345 kV lines in Michigan trip. A line that runs from Grand Ledge to Ann Arbor known as the Oneida-Majestic interconnection trips. A short time later, a line running from Bay City south to Flint in Consumers Energy’s system known as the Hampton-Thetford line also trips.
• 4:10:38 p.m. Cleveland separates from the Pennsylvania grid.
• 4:10:39 p.m. 3.7 GW power flows from the east along the north shore of Lake Erie, through Ontario to southern Michigan and northern Ohio, a flow more than ten times greater than the condition 30 seconds earlier, causing a voltage drop across the system.
• 4:10:40 p.m. Flow flips to 2 GW eastward from Michigan through Ontario (a net reversal of 5.7 GW of power), then reverses back westward again within a half second.
• 4:10:43 p.m. International connections between the United States and Canada start to fail.
• 4:10:45 p.m. Northwestern Ontario separates from the east when the Wawa-Marathon 230 kV line north of Lake Superior disconnects. The first Ontario power plants go offline in response to the unstable voltage and current demand on the system.
• 4:10:46 p.m. The New England grid separates from New York, to stop the black out from entering New England.
• 4:10:50 p.m. Ontario separates from the western New York grid.
• 4:11:57 p.m. The Keith-Waterman, Bunce Creek-Scott 230 kV lines and the St. Clair–Lambton #1 230 kV line and #2 345 kV line between Michigan and Ontario fail.
• 4:12:03 p.m. Windsor, Ontario, and surrounding areas drop off the grid.
• 4:12:58 p.m. Northern New Jersey separates its power-grids from New York and the Philadelphia area, causing a cascade of failing secondary generator plants along the New Jersey coast and throughout the inland regions west.
• 4:13 p.m. End of cascading failure. 256 power plants are off-line, 85% of which went offline after the grid separations occurred, most due to the action of automatic protective controls.

Effects

Major cities affected included New York City and surrounding areas (14,300,000 people), Toronto metropolitan area and surrounding areas (8,300,000), Newark, New Jersey, and surrounding areas (6,980,000), Detroit and surrounding areas (6,400,000), Cleveland and surrounding areas (2,900,000), Ottawa (780,000 of 1,120,000), Buffalo, New York, and surrounding areas (1,100,000), Rochester, New York (1,050,000), London, Ontario, and surrounding areas (475,000), Kitchener-Cambridge-Waterloo and surrounding areas (415,000), Toledo, Ohio (310,000), and Windsor, Ontario (208,000), for an estimated total of 55,000,000 people affected.

With the power fluctuations on the grid, power plants automatically went into “safe mode” to prevent damage in the case of an overload. This put much of the nuclear power offline until those plants could be slowly taken out of “safe mode.” In the meantime, all available hydro-electric plants (as well as a number of coal- and oil-fired plants) were brought online, bringing some electrical power to the areas immediately surrounding the plants by the morning of August 15.

Some areas lost water pressure because pumps lacked power. This loss of pressure caused potential contamination of the water supply. Four million customers of the Detroit water system in eight counties were under a boil-water advisory until August 18, four days after the initial outage. Twenty people living on the St. Clair River claim to have been sickened after bathing in the river during the blackout. Cleveland also lost water pressure and instituted a boil water advisory. Cleveland and New York had sewage spills into waterways, requiring beach closures.

Amtrak’s Northeast Corridor railroad service was stopped north of Philadelphia, and all trains running into and out of New York City were shut down. Passenger screenings at affected airports ceased, and regional airports were shut down. Many gas stations were unable to pump fuel due to lack of electricity, and many oil refineries on the East Coast of the United States shut down as a result of the blackout.

Fatalities

The blackout contributed to almost 100 deaths, including several from carbon monoxide poisoning (from generators used indoors), fires caused by candles, and other incidental causes across the affected region.

Long-term effects

The blackout prompted the federal government of the United States to include reliability provisions in the Energy Policy Act of 2005. The standards of the North American Electric Reliability Corporation became mandatory for U.S. electricity providers.

The Ontario government fell in a provincial election held in October 2003; power had long been a major issue. The government may have been hurt by the success of Quebec and Manitoba, which were not affected whereas Ontario was shut down.

Statements made in the aftermath

During the first two hours of the event, various officials offered speculative explanations as to its root cause, most of which were later proven incorrect: officials from the office of Canadian Prime Minister Jean Chrétien initially blamed a lightning strike on a power plant in northern New York; Canadian Defence Minister John McCallum blamed an outage at a nuclear plant in Pennsylvania; and New York governor George Pataki initially blamed Canada. All of these initial explanations were later retracted or disproven as investigators traced the cascading failure back to the Cleveland-Akron area and FirstEnergy’s failed alarm system. New Mexico governor Bill Richardson, in a live television interview two hours into the blackout, characterized the United States as “a superpower with a third-world electricity grid.”