Final Report on the August 14, 2003 Blackout in the United States and Canada: Causes and Recommendations

U.S.-Canada Power System Outage Task Force, April 2004

(This is an excerpt covering the Introduction, the overview of recommendations, and Chapter 3 — Causes of the Blackout and Violations of NERC Standards — from the 238-page joint government report. The full report also covers pre-blackout system conditions, a detailed minute-by-minute account of the cascade, comparisons with prior blackouts, nuclear plant performance, and physical/cyber security findings.)

1. Introduction

On August 14, 2003, large portions of the Midwest and Northeast United States and Ontario, Canada, experienced an electric power blackout. The outage affected an area with an estimated 50 million people and 61,800 megawatts (MW) of electric load in the states of Ohio, Michigan, Pennsylvania, New York, Vermont, Massachusetts, Connecticut, New Jersey and the Canadian province of Ontario. The blackout began a few minutes after 4:00 pm Eastern Daylight Time (16:00 EDT), and power was not restored for 4 days in some parts of the United States. Parts of Ontario suffered rolling blackouts for more than a week before full power was restored. Estimates of total costs in the United States range between $4 billion and $10 billion (U.S. dollars). In Canada, gross domestic product was down 0.7% in August, there was a net loss of 18.9 million work hours, and manufacturing shipments in Ontario were down $2.3 billion (Canadian dollars).

On August 15, President George W. Bush and then-Prime Minister Jean Chrétien directed that a joint U.S.-Canada Power System Outage Task Force be established to investigate the causes of the blackout and ways to reduce the possibility of future outages. They named U.S. Secretary of Energy Spencer Abraham and Herb Dhaliwal, Minister of Natural Resources, Canada, to chair the joint Task Force. (Mr. Dhaliwal was later succeeded by Mr. John Efford as Minister of Natural Resources and as co-chair of the Task Force.)

Three other U.S. representatives and three other Canadian representatives were named to the Task Force. The U.S. members were Tom Ridge, Secretary of Homeland Security; Pat Wood III, Chairman of the Federal Energy Regulatory Commission; and Nils Diaz, Chairman of the Nuclear Regulatory Commission. The Canadian members were Deputy Prime Minister John Manley, later succeeded by Deputy Prime Minister Anne McLellan; Kenneth Vollman, Chairman of the National Energy Board; and Linda J. Keen, President and CEO of the Canadian Nuclear Safety Commission.

The Task Force divided its work into two phases:

• Phase I: Investigate the outage to determine its causes and why it was not contained.
• Phase II: Develop recommendations to reduce the possibility of future outages and reduce the scope of any that occur.

The Task Force created three Working Groups to assist in both phases of its work — an Electric System Working Group (ESWG), a Nuclear Working Group (NWG), and a Security Working Group (SWG). The Working Groups were made up of state and provincial representatives, federal employees, and contractors working for the U.S. and Canadian government agencies represented on the Task Force.

The Task Force published an Interim Report on November 19, 2003, summarizing the facts that the bi-national investigation found regarding the causes of the blackout on August 14, 2003. This Final Report updates and supersedes the Interim Report, presenting findings concerning additional violations of reliability requirements and institutional and performance deficiencies beyond those identified in the Interim Report.

Overview of Task Force Recommendations (Titles Only)

Group I. Institutional Issues Related to Reliability

1. Make reliability standards mandatory and enforceable, with penalties for noncompliance.
2. Develop a regulator-approved funding mechanism for NERC and the regional reliability councils, to ensure their independence from the parties they oversee.
3. Strengthen the institutional framework for reliability management in North America.
4. Clarify that prudent expenditures and investments for bulk system reliability (including investments in new technologies) will be recoverable through transmission rates.
5. Track implementation of recommended actions to improve reliability.
6. FERC should not approve the operation of new RTOs or ISOs until they have met minimum functional requirements.
7. Require any entity operating as part of the bulk power system to be a member of a regional reliability council if it operates within the council’s footprint.
8. Shield operators who initiate load shedding pursuant to approved guidelines from liability or retaliation.
9. Integrate a “reliability impact” consideration into the regulatory decision-making process.
10. Establish an independent source of reliability performance information.
11. Establish requirements for collection and reporting of data needed for post-blackout analyses.
12. Commission an independent study of the relationships among industry restructuring, competition, and reliability.
13. DOE should expand its research programs on reliability-related tools and technologies.
14. Establish a standing framework for the conduct of future blackout and disturbance investigations.

Group II. Support and Strengthen NERC’s Actions of February 10, 2004

15. Correct the direct causes of the August 14, 2003 blackout.
16. Establish enforceable standards for maintenance of electrical clearances in right-of-way areas.
17. Strengthen the NERC Compliance Enforcement Program.
18. Support and strengthen NERC’s Reliability Readiness Audit Program.
19. Improve near-term and long-term training and certification requirements for operators, reliability coordinators, and operator support staff.
20. Establish clear definitions for normal, alert and emergency operational system conditions. Clarify roles, responsibilities, and authorities of reliability coordinators and control areas under each condition.
21. Make more effective and wider use of system protection measures.
22. Evaluate and adopt better real-time tools for operators and reliability coordinators.
23. Strengthen reactive power and voltage control practices in all NERC regions.
24. Improve quality of system modeling data and data exchange practices.
25. NERC should reevaluate its existing reliability standards development process and accelerate the adoption of enforceable standards.
26. Tighten communications protocols, especially for communications during alerts and emergencies. Upgrade communication system hardware where appropriate.
27. Develop enforceable standards for transmission line ratings.
28. Require use of time-synchronized data recorders.
29. Evaluate and disseminate lessons learned during system restoration.
30. Clarify criteria for identification of operationally critical facilities, and improve dissemination of updated information on unplanned outages.
31. Clarify that the transmission loading relief (TLR) process should not be used in situations involving an actual violation of an Operating Security Limit. Streamline the TLR process.

(The full report lists 46 recommendations in total; the single most important, per the Task Force, was that the U.S. Congress enact the reliability provisions in H.R. 6 and S. 2095 to make compliance with reliability standards mandatory and enforceable.)

3. Causes of the Blackout and Violations of NERC Standards

Summary

This chapter explains in summary form the causes of the initiation of the blackout in Ohio, based on the analyses by the bi-national investigation team. It also lists NERC’s findings to date concerning seven specific violations of its reliability policies, guidelines, and standards. Last, it explains how some NERC standards and processes were inadequate because they did not give sufficiently clear direction to industry members concerning some preventive measures needed to maintain reliability, and that NERC does not have the authority to enforce compliance with the standards. Clear standards with mandatory compliance, as contemplated under legislation pending in the U.S. Congress, might have averted the start of this blackout.

The Causes of the Blackout in Ohio

A dictionary definition of “cause” is “something that produces an effect, result, or consequence.” In searching for the causes of the blackout, the investigation team looked back through the progression of sequential events, actions and inactions to identify the cause(s) of each event. The idea of “cause” is here linked not just to what happened or why it happened, but more specifically to the entities whose duties and responsibilities were to anticipate and prepare to deal with the things that could go wrong. Four major causes, or groups of causes, are identified.

Although the causes discussed below produced the failures and events of August 14, they did not leap into being that day. Instead, they reflect long-standing institutional failures and weaknesses that need to be understood and corrected in order to maintain reliability.

Causes of the Blackout’s Initiation

The Ohio phase of the August 14, 2003, blackout was caused by deficiencies in specific practices, equipment, and human decisions by various organizations that affected conditions and outcomes that afternoon — for example, insufficient reactive power was an issue in the blackout, but it was not a cause in itself. Rather, deficiencies in corporate policies, lack of adherence to industry policies, and inadequate management of reactive power and voltage caused the blackout, rather than the lack of reactive power. There are four groups of causes for the blackout:

Group 1: FirstEnergy (FE) and ECAR failed to assess and understand the inadequacies of FE’s system, particularly with respect to voltage instability and the vulnerability of the Cleveland-Akron area, and FE did not operate its system with appropriate voltage criteria. (Note: This cause was not identified in the Task Force’s Interim Report. It is based on analysis completed by the investigative team after the publication of the Interim Report.)

• FE failed to conduct rigorous long-term planning studies of its system, and neglected to conduct appropriate multiple contingency or extreme condition assessments.
• FE did not conduct sufficient voltage analyses for its Ohio control area and used operational voltage criteria that did not reflect actual voltage stability conditions and needs.
• ECAR (FE’s reliability council) did not conduct an independent review or analysis of FE’s voltage criteria and operating needs, thereby allowing FE to use inadequate practices without correction.
• Some of NERC’s planning and operational requirements and standards were sufficiently ambiguous that FE could interpret them to include practices that were inadequate for reliable system operation.

Group 2: Inadequate situational awareness at FirstEnergy. FE did not recognize or understand the deteriorating condition of its system.

• FE failed to ensure the security of its transmission system after significant unforeseen contingencies because it did not use an effective contingency analysis capability on a routine basis.
• FE lacked procedures to ensure that its operators were continually aware of the functional state of their critical monitoring tools.
• FE control center computer support staff and operations staff did not have effective internal communications procedures.
• FE lacked procedures to test effectively the functional state of its monitoring tools after repairs were made.
• FE did not have additional or back-up monitoring tools to understand or visualize the status of their transmission system to facilitate its operators’ understanding of transmission system conditions after the failure of their primary monitoring/alarming systems.

Group 3: FE failed to manage adequately tree growth in its transmission rights-of-way. This failure was the common cause of the outage of three FE 345-kV transmission lines and one 138-kV line.

Group 4: Failure of the interconnected grid’s reliability organizations to provide effective real-time diagnostic support.

• MISO did not have real-time data from Dayton Power and Light’s Stuart-Atlanta 345-kV line incorporated into its state estimator (a system monitoring tool). This precluded MISO from becoming aware of FE’s system problems earlier and providing diagnostic assistance or direction to FE.
• MISO’s reliability coordinators were using non-real-time data to support real-time “flowgate” monitoring. This prevented MISO from detecting an N-1 security violation in FE’s system and from assisting FE in necessary relief actions.
• MISO lacked an effective way to identify the location and significance of transmission line breaker operations reported by their Energy Management System (EMS). Such information would have enabled MISO operators to become aware earlier of important line outages.
• PJM and MISO lacked joint procedures or guidelines on when and how to coordinate a security limit violation observed by one of them in the other’s area due to a contingency near their common boundary.

Linking Causes to Specific Weaknesses

Seven violations of NERC standards, as identified by NERC, and other conclusions reached by NERC and the bi-national investigation team are aligned with the specific causes of the blackout. There is an additional category of conclusions beyond the four principal causes — the failure to act, when it was the result of preceding conditions. For instance, FE did not respond to the loss of its transmission lines because it did not have sufficient information or insight to reveal the need for action. (Note: NERC’s list of violations has been revised and extended since publication of the Interim Report. Two violations cited in the Interim Report were dropped, and three new violations have been identified in this report. NERC continues to study the record and may identify additional violations.)

A NERC team looked at whether and how violations of NERC’s reliability requirements may have occurred in the events leading up to the blackout. They also looked at whether deficiencies in the requirements, practices and procedures of NERC and the regional reliability organizations may have contributed to the blackout. They found seven specific violations of NERC operating policies (although some are qualified by a lack of specificity in the NERC requirements). The Standards, Procedures and Compliance Investigation Team reviewed the NERC Policies for violations, building on work and going beyond work done by the Root Cause Analysis Team. Based on that review the Standards team identified a number of violations related to policies 2, 4, 5, and 9:

• Violation 1: Following the outage of the Chamberlin-Harding 345-kV line, FE did not take the necessary actions to return the system to a safe operating state within 30 minutes.
• Violation 2: FE did not notify other systems of an impending system emergency. (Policy 5, Section A, Requirement 1, directs a system to inform other systems if it is burdening others, reducing system reliability, or if its lack of single contingency coverage could threaten Interconnection reliability.)
• Violation 3: FE’s state estimation/contingency analysis tools were not used to assess the system conditions. (This is addressed in Operating Policy 5, Section C, Requirement 3, concerning assessment of Operating Security Limit violations, and Policy 4, Section A, Requirement 5, which addresses using monitoring equipment to inform the system operator of important conditions and the potential need for corrective action.)
• Violation 4: MISO did not notify other reliability coordinators of potential problems. (Policy 9, Section C, Requirement 2, directing the reliability coordinator to alert all control areas and reliability coordinators of a potential transmission problem.)
• Violation 5: MISO was using non-real-time data to support real-time operations. (Policy 9, Appendix D, Section A, Criteria For Reliability Coordinators 5.2, regarding adequate facilities to perform their responsibilities, including detailed monitoring capability to identify potential security violations.)
• Violation 6: PJM and MISO as Reliability Coordinators lacked procedures or guidelines between themselves on when and how to coordinate an operating security limit violation observed by one of them in the other’s area due to a contingency near their common boundary (Policy 9, Appendix 9C, Emergency Procedures). Note: Since Policy 9 lacks specifics on coordinated procedures and training, it was not possible for the bi-national team to identify the exact violation that occurred.
• Violation 7: The monitoring equipment provided to FE operators was not sufficient to bring the operators’ attention to the deviation on the system. (Policy 4, Section A, System Monitoring Requirements regarding resource availability and the use of monitoring equipment to alert operators to the need for corrective action.)

Control areas and reliability coordinators operate the grid every day under guidelines, policies, and requirements established by the industry’s reliability community under NERC’s coordination. If those policies are strong, clear, and unambiguous, then everyone will plan and operate the system at a high level of performance and reliability will be high. But if those policies are ambiguous and do not make entities’ roles and responsibilities clear and certain, they allow companies to perform at varying levels and system reliability is likely to be compromised. Given that NERC has been a voluntary organization that makes decisions based on member votes, if NERC’s standards have been unclear, non-specific, lacking in scope, or insufficiently strict, that reflects at least as much on the industry community that drafts and votes on the standards as it does on NERC. Similarly, NERC’s ability to obtain compliance with its requirements through its audit process has been limited by the extent to which the industry has been willing to support the audit program.