By Matthew Lewis, Manager of Event Analysis, NERC, USA
“Gentlemen, we are always going to relentlessly chase perfection, knowing full well we will not catch it because nothing is perfect. In the process we will catch excellence…I am not remotely interested in being just good.”* If you are a Green Bay Packers football fan, you are probably familiar with these words and their author. If you are not a fan, you are likely familiar with the essence of the quote, but do not know from whom the words were uttered. Coach Vince Lombardi used this language to inspire arguably one of the greatest collections of ordinary humans to greatness on the gridiron. You see, Coach Lombardi recognized and understood that perfection is unachievable, but if you focus on the possibility, set high standards, and pursue their achievement you can capture excellence on the way.
* Chuck Carlson, Game of My Life Green Bay Packers: Memorable Stories of Packers Football (2012)
This principal helped guide the Packers of the 1960s to win five conference titles and the first two NFL super bowls. Football, like all endeavors requiring a winning mindset to achieve excellence in application, is driven by the idea of continuous improvement. The same is true for operating the Bulk Electric System (BES) because each event on the system tells a story that when analyzed reveals insights into continuous improvement opportunities. The path to top-tier performance and a serious pursuit of perfection is paved by review, reflection, analysis, practice, and implementation of experiential lessons learned. The bulk electric industry and regulatory communities embrace this idea collaboratively through analysis of real-world power system operations that impact the reliability of the BES.
A Basis to Pursue Performance Excellence
The Electric Reliability Organization (ERO) Event Analysis (EA) Program provides the statutory capability to analyze events that occur on the BES to support identifying emerging risks to the system and/or corroborating previously identified risks.
A primary challenge is to employ the capability in a way that leads to sharing knowledge with industry, policymakers, and other stakeholders in a manner that is consistent, informative, and constructive. The goal is to prevent recurrence of bad events while increasing the odds of capturing performance and reliability excellence routinely.
The North American Electric Reliability Corporation (NERC) Rules of Procedure (ROP), implementation guidance for Section 215 of the Federal Power Act, provides the formal guidelines for establishing and managing a performance monitoring and analysis activity. The guidance provides a description of the overall requirements for system reliability assessment and performance analysis that documents the following general objectives driving the principle focus of forensic analysis of BES events:
- Analyze off-normal and large-scale/major events
- Identify the root causes of events that may be precursors of potentially more serious events
- Assess past reliability performance for lessons learned
- Disseminate findings and lessons learned to the electric industry to improve reliability performance
The legislative authority translated through procedural guidance and managed by the EA program establishes a framework (see the figure on the top) that enables collaboration between the power industry and the regulatory community in the pursuit of performance excellence that is essential to a consistently reliable BES.
Spectrum of Analysis
BES events generally fall into two broad categories: off-normal events and major events. Off-normal events are typically incidents that take place during the daily operation of the system that result in direct reliability impacts to a portion of the system and/or reveal faint signals of system reliability becoming jeopardized. A major event generally results in a significant impact or impacts to the performance of the larger system, including cascading blackouts.
There are generally two types of scenarios for consideration when analyzing events on the BES: 1) a single initiating electrical cause that connects multiple facilities, and 2) separate incidents initiated by independent electrical causes, but that are tied together by a common operating condition/environment over an extended footprint (e.g., extreme weather periods). Analysis is scalable for both through systematic and interest-based approaches.
Value of Analyzing System Events
Conducting analysis of system events is critical to identifying lessons learned, trends and emerging risks that lead to developing mitigating strategies to minimize recurrence and/or modified trending. This demands participation by the involved operating entities along with robust coordination and collaboration with other relevant parties including regulatory authorities. The interaction with the regulator provides entities with a mechanism or convening platform to share knowledge on a wide area scale across the industry/grid that is typically not available to an individual organization. A shared mindset for continuous improvement is imperative to provide lasting effect on industry design and operating practices influencing BES performance. The ultimate objective is the pursuit of reliability excellence through effective and efficient reduction of risk to the reliability and security of the grid.
Approaches to Capturing Lessons and Sharing
There are several report types the ERO uses to share knowledge harvested from the analysis of events, occurrences, or incidents on the BES. Some are working documents that support publishable documents while others are finished publications. The four core types published for public use include the NERC Lesson Learned, incident review, system disturbance report, and system performance review (see Figure 1). These are scalable products that are focused on differing audiences, but useable by all.
Each type can generically be described as a disturbance report for either individual or multiple events with the primary function being presentation of lessons learned with some level of description or “how we got here” included.
- The most basic product is the NERC Lesson Learned. In short, this is a product that covers tactical improvements or enhancements for an issue(s) typically having known solutions. The intent is to share industry-wide to propagate system, human, and organizational fixes across entities while establishing good practices across the grid. A lesson learned is generally written in collaboration with industry and facilitated by ERO EA staff.
- The incident review is focused on strategic thought and considerations. As this description implies, its primary audience includes executives, decision-makers, policymakers, and comparable stakeholders. The intent is to identify operating challenges on today’s grid that can inform operations on the grid of the future. These reviews do not necessarily describe confirmed solutions but do provide considerations for potential solutions.
- System disturbance reports are typically written to inform system engineers, operators, planners and other “doers” on the system. These documents are technically rich and can be quite lengthy with detailed analysis and recommendations. They generally provide a full story on what occurred with analysis that fully describes the “how we got here” for the event. These publications provide solutions and inform potential “downstream” work for regulators and industry alike.
- A system performance review is intended for a large footprint/system-wide common operating (multiple unrelated events) condition or environment after-action review. These reports are written at a high-level from a backward-facing perspective with collaboration between industry, NERC, the Federal Energy Regulatory Commission (FERC), and the ERO Regional Entities depending on the need. The purpose of these reports is to review system operations during a specified period for selected operating entities with the goal of identifying operating practice sustainment and improvement. The intended audience is the public, stakeholders, policymakers, and decision-makers. The report informs the strategic/policy level for future grid operations consideration.
All reports developed by NERC for public consumption are available on NERC.com and linked to the Event Analysis Program page. In collaborative cases, joint reports are published on the NERC website or a partner Region’s site and linked to NERC supporting the overall ERO program.
Data Gathering Approaches
The ERO collaborates with industry in pursuing the analysis of electrical/operating events, incidents or occurrences through two types of processes each tied to a general model (see Figure 2).
One is a systematic approach that is criterion-based and the other is interest-based (see Figure 3). Both methods are driven by daily monitoring of system operations through various tools and means. The fundamental difference between the approaches is in the information/data gathering method. Although these approaches are preferably executed in a collegial manner when interacting with event-related entities, when necessary, they are reinforced by authorities for mandatory response to data requests given in the NERC ROP.
The systematic process or ERO Event Analysis Process (EAP) is a formal repetitive process tripped by pre-determined criterion. It is published publicly and updated periodically with combined input from the power system and regulatory communities. The core document with appendices is located at NERC.com. The EAP feeds the tracking and trending efforts of the ERO. Ultimately, the events reported through this formal process are cataloged using cause code assignment. The assignment process assists in tracking and trending using root and contributing causes. The ERO uses this information when corroborating known risks and impacts or identifying new risks or signals of developing risks to a changing power grid.
The interest-based approach is driven primarily by emerging system concerns identified through industry-regulatory interactions, communication, and routine system performance monitoring. Absent standing criteria, this approach relies on outreach directly to operating entities for inquiry and data gathering related to the incidents/occurrences that the ERO desires to pursue. Outreach involves initial inquiry followed by informal and/or formal data requests. This initial phase of inquiry may lead to focused conversation depending on the review of the initial feedback and generated interest level for the ERO. Feedback during direct conversations with the involved entity or entities determines the next steps in analyzing these types of incidents. These conversations may lead to joint publication or serve simply as a note for continued monitoring by the team.
The approach, techniques, and depth of required analysis to effectively articulate the causal path for an event dictates the types (examples shown in Figure 2) and volume of data requested. Typically, the conversation between industry and the ERO staff is conducted in an iterative manner regardless of the chosen approach. In general, root cause analysis principles apply to both approaches, albeit modified to align with the purpose, available resources, and need for the analysis. This varies by circumstance(s) and perceived consequence(s) to grid reliability. The involved utility or utilities are charged with providing an accurate disturbance narrative and supporting data required for the ERO engineers and other subject matter experts to conduct independent analysis. Additionally, there is an expectation that submitting utilities apply some level of internal root cause analysis that supports the narrative and other information being supplied to the ERO for either approach.
Example Publications
The approaches described in this article developed over many years. As with operating the grid, a mindset of continuous improvement is the catalyst for success in adding value through the development of “disturbance reports” conceived through persistent system performance monitoring and industry partnership. Experience in this effort has produced many publications that have shined light on important system operating adjustments, sustainment opportunities, and improvements shared across industry. These products serve to help industry implement better practices, inform the development of industry standards, and educate policymakers and regulators similarly.
Several examples (See Figure 4) are provided in this section to demonstrate the end products described in the Approaches to Capturing Lessons and Sharing section in this article. These examples are summarized here to provide a taste of their content. The reader is encouraged to pursue the fullness of the details for each situation in the completed publications as cited along with many others on the NERC public website.
IBR Control Lessons: A Lesson learned to share system-wide concerning battery energy storage systems and photovoltaic site commissioning. Improper controller integration for inverter-based resources (IBR) lead to interconnection-wide “oscillations” on multiple occasions. The operating utility had instances where disconnection was required for reliability purposes which ultimately led to the need for improvement in controller integration. Several corrective actions emerged from this operating scenario leading to improvements in operating and commissioning practices for IBR sites.
Shoulder Season Challenges: An incident review highlighting an increasing management challenge for power system operators. This review acknowledged a common thread for multiple incidents over a 13-month period. Each incident involved both planned and unplanned generation unavailability and transmission line unavailability. Under forecasted load was also a factor. The incidents occurred during “shoulder seasons” with real-time contingency analysis identifying loss of next transmission element concerns driving the need (avoiding instability or cascading) for pre-contingent load shedding in all cases. Entity reporting revealed resource mix, weather, and load forecasting challenges. Analysis of the incidents produced future considerations for grid operations in load pockets between peak operating periods that are restated here:
- Increase operating margin and unit commitment to improve adaptability to changing circumstances
- Leverage demand-side management to mitigate capacity shortfalls
- Outage planning must include the potential for atypical operating conditions
- Contemplate practical transmission relay loadability settings in accordance with NERC PRC-023 requirements regarding contingent overloading
- Review the outage plan and adapt to emerging system conditions
- Ensure operational planning analysis incorporates up-to-date day-ahead market unit commitment information with mismatches resolved
Voltage Sensitive Loads: A seminal publication that revealed the hidden challenges related to the inherent operating philosophies of data center and crypto mining facilities. Specifically, this incident review spotlighted the potential risk of large blocks of computational load disconnecting from the system during normally cleared system faults. Consequently, several NERC initiatives and outreach are currently being driven from this work such as:
- A collaborative industry regulator large load technical working group
- Multiple problem scoping white papers
- NERC Reliability Guidelines
- New proposed NERC registration criteria
- NERC Standards development projects
- NERC Reliability Insight
Cascading Blackout Disturbance: A disturbance report documenting a major outage within the northeast region of the Western Interconnection. A planned outage of a single 500 kV transmission line led to cascading outages impacting portions of multiple states and a quarter of a million customers. This report describes the relationship of circumstances to multiple causes leading to the outages. It also illustrates the complexities of operating on a seam between two Reliability Coordinators and the dangers of poor coordination and communication inside, outside, and between operating groups/entities.
Arctic Cold Events System Performance: The purpose of the product is to document an “after-action review” of the Bulk Power System’s performance during the January 2025 Arctic events (Winter Storms Blair, Cora, Demi, and Enzo). The review includes input from multiple electrical entities in the Eastern and ERCOT interconnections and natural gas trade associations capturing performance related to the following broad topics: weather conditions, operating conditions, electric grid performance, and natural gas system performance. A joint team consisting of FERC, NERC, and the Regional Entities conducted the review. The focus of the review is on a common operating condition and/or environment with the goal of industry and regulators alike identifying sustainment and improvement opportunities for future grid performance during extreme weather circumstances.
Conclusion
A shared continuous improvement mindset is critical to grid performance. The ERO provides support to this imperative by exercising the statutory powers granted in Section 215 of the Federal Power Act and translated through the NERC ROP into a performance analysis activity. The approaches to harvesting lessons and understanding event causes while sharing in a public manner are dependent on the committed participation of operating entities and subject matter experts within the stakeholder communities. The mix of coordinated and collaborative effort using systematic and interest-based approaches have proven informative to developing/improving baseline operating standards and strengthening of system reliability.
The evidence for this is published routinely in partnership with ERO Regional Entities and the bulk power industry through NERC Lessons Learned, incident reviews, system disturbance reports, and system performance reviews. Focusing on an assorted audience consisting of industry, regulators, policymakers, stakeholders, and the public facilitates a common understanding and commitment to grid reliability and security.
The ERO prefers to foster a collegial relationship, with industry necessary to effectively apply these approaches with an expectation that entities are open and transparent while reporting events, occurrences, and incidents within their footprint. This benefits everyone, including the regulator as an industry advocate in the pursuit of performance and reliability excellence through effective and efficient reduction of risk to the reliability and security of the grid.
Biography:
Matthew Lewis joined the North American Electric Reliability Corporation (NERC) on March 24, 2014 and currently serves as the Manager of Event Analysis and previously served as the Manager of Training and Education. He retired with over twenty-five years of service as a U.S. Army officer. Matt served in a variety of leadership and staff positions in the fields of operations, special weapons effects and response, technical intelligence, and joint exercises/training. During his military service, he participated in combat tours in Desert Storm and Iraqi Freedom. Matt has a Bachelor of Science in Physics from the University of Arkansas at Little Rock and a Master of Science in Applied Physics from the Air Force Institute of Technology.
