By J. Deverick, N. Bilimoria, J. Grimes, J. Schlank, S. Vajpayee, and P. VanSant, Dominion Energy, K. Gray, POWER Engineers Inc., R. Hunt, Quanta Technology, and T. Grigg, Schweitzer Engineering Laboratories, USA
ACKNOWLEDGMENTS: The authors would like to thank the following people for their technical guidance, contributions, and critiques: Jeffrey Barrero and Carynne Corvaia from Dominion Energy; Howard Self from Quanta Technology; and David Dolezik and Jason Dearien from Schweitzer Engineering Laboratories.
Dominion Energy Virginia is experiencing increasing load growth and a seemingly exponential demand for new generation interconnections. Dominion and its industry-leading vendor partners formed a collaboration to meet both current and future demand by developing a fully digital IEC 61850-based, mass-producible control enclosure. It’s a bold, multi-year project that included rethinking the basic design of a digital substation, simplifying connections, and re-imagining engineering, manufacturing, and installation.
What is a DICE?
While digital substations are not new, Dominion and its partners-Schweitzer Engineering Laboratories (SEL), Quanta Technology and Power Engineers, Inc-are placing their bets on the Drop-in-Place Control Enclosure (DICE), which is a self-contained, fully digital substation based on IEC 61850. The enclosure has been designed for mass-production. It can be fabricated, assembled, and tested offsite, with installation and commissioning onsite. The DICE will be the first enclosure of its kind by a U.S. utility.
The Need for a Mass-produced Solution
Dominion Energy Virginia currently serves approximately 2.7 million electric customers in Virginia and North Carolina. The company owns approximately 6,800 miles of transmission lines in Virginia, North Carolina, and West Virginia; and approximately 60,000 miles of distribution lines in Virginia and North Carolina. Over 900 substations are operational.
Two main factors drive the need for new substations in Dominion’s service area:
1.) The 2023 load forecast from PJM, Dominion’s grid operator, increased significantly compared to last two annual forecasts.
2.) The Virginia Clean Economy Act (VCEA), passed in 2020, set an ambitious goal of 100% clean energy by 2045. The Act requires the development of renewable energy generation resources and energy storage resources, as well as the submission of an annual plan to meet the development targets and the creation of a mandatory Renewable Energy Portfolio Standard Program.
Historically, Dominion constructs 15-20 substations per year. The projected demand of substations is 2-4 times that.
With such a short lead time to completely change Dominion’s model of power generation, a solution was needed to facilitate the generation interconnections by streamlining the engineering, fabrication, integration, and testing required for new digital substations. (Figure 2).
The five-year project began in 2021 as engineers and field technicians met multiple times to determine what the foundational philosophy of the DICE project should be. Given that the enclosure would be fully digital, the team selected IEC 61850 as the technological foundation.
What is IEC 61850?
IEC61850 is the international standard for defining devices within substation automation systems and how they interact with one another.
The IEC 61850 has four main parts:
- Semantics. The data model ensures semantic interoperability among different manufacturers
- Language. The System Configuration Language (SCL) provides the syntax for a seamless, and interoperable engineering process
- Communication. Standardized communication services, supported by a set of limited communication protocols, allows real-time communication between field devices.
- Security. A companion standard, IEC 62351, guarantees the security of the data exchanged with the IEC 61850 protocols
- Some of the ways Dominion’s fully digital substation is different from a conventional substation:
- Wires traditionally utilized for analog measurements and controls will be replaced by fiber between the merging unit and control enclosure, allowing for easier integration with the DICE installation
- Connections will be mainly accomplished by network engineering and relay programming
- Digital Human Machine Interface (HMI) will take the place of physical controls, lockout indicators, and test switches
- Using HMI and consolidating relay functions will reduce the panel space required for the control enclosure
- Cutting the enclosure’s footprint in half to 12’ x 50’ (instead of a traditional 24’ x 50’) makes it narrow enough to be transported on highways without special permits
A first-pass analysis suggests that about 75% of the protection panels could be eliminated by utilizing IEC 61850. Savings on the panel-to-panel wiring would be approximately 60%. Dominion estimates that one standardized IEC 61850 DICE substation will save four weeks of panel fabrication and installation time.
The Vision
Dominion Energy’s goal is to create a fully digital control enclosure (CE) that could be created quickly and installed efficiently to meet escalating demand. The fully digital design of the CE will utilize IEC 61850 with sampled values, and significantly consolidate relay functions. The vision is to design a safe, mass-producible, transportable, and cost-efficient control enclosure that would be able to scale to different use-case applications. The control enclosure will be manufactured and acceptance tested off-site to streamline on-site substation integration and commissioning. This would be done in close collaboration with internal departments and external industry partners. This vision will help achieve the goal of meeting escalating demand and our customer’s needs.
The first DICE, emulating a four-breaker ring bus transmission substation, will be installed at Dominion Energy’s training center to allow hands-on testing and training for field personnel, operators, and engineers. This will also allow the enclosure and its components to be examined from various perspectives: human performance, safety, operations, maintenance, commissioning, testing, training, and compliance. After successfully implementing the initial design, Dominion will expand the DICE application to other substation configurations.
To facilitate mass production, Dominion is building a new relay panel shop that will have a DICE Integration Bay to streamline the process of integrating the simplified panels into the enclosures. The DICE Integration Bay will allow up to four control enclosures to be integrated and tested at a time. Dominion will also be able to control the fabrication process, the materials, and the quality of each DICE.
The DICE Development Team
Because a fully digital facility needs to be designed essentially from scratch, all stakeholders were brought into the project early. Engineers and field technicians from 13 departments across Dominion collaborated on everything from philosophy to equipment to facility design and construction. (See Table 1.)
One unique aspect of the project is the degree to which vendors are involved. Dominion has three primary vendor partners: Powers Engineers, Inc., Quanta Technology and Schweitzer Engineering Laboratories. Most of their work was done during the design and development phases. The vendors helped develop the DICE’s philosophy and focused on diagnostics and testing, setting configuration, HMI, network development, and engineering workflow.
To help in engineering development, Dominion has three major partners:
- POWER Engineers, Inc. is familiar with Dominion’s substation standards and applicational guidelines. POWER is assisting with the protection and control drawing package (elementary, wiring diagram, functional diagram, data-stream flow, etc.), protection relay settings, data concentrator settings, and IEC 61850 configurations
- Quanta Technology is providing guidance and direction in the overall scope, philosophy, and integration of IEC 61850, including testing and commissioning
- SEL is consulting in the development of diagnostic human machine interface screens, network diagrams and configuration, IEC 61850 testing, and merging-unit cabinets. They provide feedback directly back to their R&D teams
The Project Plan
Conventional substations were originally designed over a hundred years ago. While those designs have been repeatedly updated as new technology was introduced, most substations today are still engineered by applying long-established standards and making incremental improvements.
In contrast, a digital substation requires rethinking large portions of the system and re-evaluating how it should be engineered. The degree of change is not unlike the industry’s transition from electromechanical to microprocessor relays.
Use-Case Meetings: For a fresh new design, Dominion needed to think through each use case, analyzing both how the protection and control schemes should be designed and how they would operate under various conditions. Use-case meetings were held with both internal and external stakeholders several times a week for six months. During the height of the engineering phase, meetings were held three times a week.
Illustrations of the use-case scenarios often needed to be drawn up to help team members understand an application. The use case meetings’ outcomes were then used to develop a full Protection and Control drawing package for the DICE.
The Lab: Next, the team updated the drawings as testing progressed, and created new mock-ups accordingly. A new type of drawing was created called the Signal Flow Diagram, which shows the signals being published, the inputs connected to each device, and the devices that subscribe to those signals. One signal flow diagram was created for each intelligent electronic device (IED) that published and/or subscribed to generic object-oriented substation event (GOOSE) or sampled value (SV) messages.
The IEC 61850 configuration, as well as each IED’s setting files, were based on the use cases and the P&C drawings. The team then created custom IED Configuration Description (ICD) files for the project. Custom ICD files were created for each IED to allow implementation of custom logical nodes to better describe custom relay functions according to IEC 61850.
A system configuration tool (SCT), as defined in IEC 61850-6, was used to generate the IEC 61850 configuration file. A major reason for using an SCT was its ability to replace ICD files without manually rebuilding the configuration. The custom ICD files will save countless hours throughout the project’s lifetime.
The final HMI design consisted of a single-line diagram, line protection detail screens, breaker control detail screens, alarm screens, and islanding control screens. Additional screens, designed for maintenance and troubleshooting, will show diagnostic details for the relay network cards and the software-defined networking (SDN) switches, as well as diagnostic information for GOOSE, SV, and the Precision Time Protocol (PTP).
Lab Testing: The level of laboratory testing required for an IEC 61850 substation goes substantially beyond that of a normal protection scheme. Because the communication equipment’s selection, design, and application settings-specifically the network and satellite clocks—become part of the protection scheme, they influence the performance and reliability of the system. The distributed functions used in an IEC 61850 substation introduce multiple dependencies between the devices and the network, resulting in multiple possible operating contingencies. For this reason, the final design of a digital substation needs to be tested as a complete system to understand its performance.
An initial build of the control enclosure was installed in a relay lab environment, which consisted of protective relay and communication panels designed to replicate an in-field DICE installation. These panels included all relays, merging units, data concentrators, satellite clocks, SDN switches and other microprocessor devices. The team also used a local human machine interface for operational controls and monitoring, as well as for diagnostics data. A third-party IEC 61850 test set was utilized to develop and run test plans for the protection schemes. A third-party network analyzer device verified GOOSE and SV communications between relays on the process bus, as well as manufacturing message specification (MMS) reports from the relays to the data concentrators on the station bus.
Dominion’s in-house Real-Time Digital Simulator (RTDS) tested the entire DICE protection scheme against various operating conditions. Since an RTDS models parts of the power system, it can easily test numerous system contingencies, including faults at various locations with different fault resistance values, and breaker failure conditions.
The RTDS was connected through amplifiers to the DICE in the lab and to non-IEC 61850 microprocessor relays for the remote line terminals. This Hardware-In-The-Loop (HIL) test was designed to identify latent issues in the system configuration, carrier-blocking coordination, line-differential network-time-delay compensation, and interaction among devices. The HIL process also tested the system performance during different operating contingencies and modeled device failures.
DICE’s Architecture
The DICE architecture is based on modularity, ease of construction and good utility practice-all while maintaining all applicable physical and cybersecurity, compliance, and environmental requirements. (Figures 3, and 4). The components of the DICE include:
Merging Unit Cabinet: A MU cabinet will be installed adjacent to each circuit breaker and will act as the interface between the primary equipment and the control enclosure. Each cabinet includes:
- A merging unit. Converts analog signals from the current transformers and potential transformers to sampled values for the network. Two merging units will be installed for full redundancy
- An equipment annunciator. A standard device that collects equipment-specific alarms and data and displays them on the enclosure’s HMI and transmits them to the system operating center
- Test and control switches. These are to provide airgaps for safety and isolation while testing, similar to their application on current relay panels
- Fiber management equipment
In the initial design, the merging-unit cabinets were positioned in the yard adjacent to each breaker. However, Dominion is currently in discussions with breaker manufacturers to develop a solution to house the merging units inside the breaker cabinet, streamlining the CE even further.
Control Enclosure Design: The outside dimensions of the enclosure will be less than 12 feet wide and 50 feet long. This will meet the Department of Transportation’s requirements and allow Dominion to avoid special transportation considerations or permits. (Figure 5).
The DICE will house relay, communication, and network panels, as well as AC and DC distribution equipment. For redundancy, two communication panels with touchscreen HMIs will be installed. There will also be provisions to mount monitor(s) on the wall to display various visual aids such as alarms, special instructions, one-lines, etc.
A separate battery room within the control enclosure will be isolated from the panel equipment to improve field and operation personnel safety.
Protection Panels in the Control Enclosure: Early in DICE’s development, the team looked for ways to utilize the fewest relays possible while maintaining all the protection and control features currently used in standard applications.
A common breaker-and-a-half configuration has five panels–two panels for the lines and three panels for the circuit breakers. In the DICE, the breaker controls, breaker failure, and reclosing are all integrated into the line relays. This design eliminated three breaker panels.
To save both engineering and testing time, the panel design was based on an existing standard that already incorporated line protection and breaker controls in one panel. It was modified to use Dominion’s custom lead/lag scheme so that two breakers could reclose, allowing reclosing and breaker failure to be initiated in a specific line relay, depending on the location of the fault.
Relay Panel Shop: Dominion is building a new relay panel shop that will have a DICE Integration Bay that will streamline the fabrication process. The shop will function as a type of assembly line, capable of integrating four CEs at a time. This should provide sufficient capacity to keep up with new substation demand. Plus, Dominion will be able to control the materials, fabrication process, and quality of each enclosure. (Figure 6)
Network Protocols: The DICE will have various IEC 61850 protocols embedded. GOOSE, SV, and MMS perform protection and control. IEEE 1588’s Precision Time Protocol provides high-accuracy time synchronization to all IEDs. Protection, controls, and time-sync are transmitted across an Ethernet network that is required to meet the same stringent physical and operational requirements as protective relays.
In 2019, Dominion partnered with a third-party contractor to perform an evaluation of the available networking technologies for IEC 61850 communications. After studying multiple options, representatives from the Data Communications group unanimously decided to pursue operation technology (OT), software-defined networking (SDN) as the IEC 61850 networking solution.
OT SDN is a deny-by-default networking solution, so it is inherently cyber-secure and provides a centralized configuration and monitoring of all SDN switches in the network. Since the networking data plane and control plane are parts of the software controller, network configuration can be fully automated from design documents. These documents include an architectural drawing that describes all the physical characteristics of the network, a data-flow diagram that describes all the logical characteristics, and the IEC 61850 substation configuration description (SCD) file that contains all the IEC 61850 configuration details.
Network Architecture: The network architecture for the DICE consists of OT SDN switches for the station and process bus networks. The station bus is configured to carry SCADA data, metering data, and equipment controls. Switches connected in a ring comprise the station bus failover network, which provides high-speed recovery from any network element failure. The failover station bus network eliminates the need for redundancy boxes. In addition, networked switches create a parallel redundancy protocol (PRP) network for the process bus. PRP uses duplicate networks between devices, so the recovery time from a network element failure is essentially zero.
OT SDN allows the PRP switches to connect to the failover network so that the data concentrator can monitor switch port status, port packet counts, and alarms, without interfering with any of the standard PRP communications.
A Work in Progress
While the design for the DICE is complete and has been lab-tested, much work remains. Dominion plans to start to fabricate the Training Center DICE at the end of 2023, and the teams are striving to produce deliverables to support the target date: developing standards, crafting engineering work practices, testing/commissioning work practices, documenting HMI, developing engineering and field training, and standardizing maintenance practices.
Mass-producing and installing completely digital substations will help integrate renewable resources and accommodate the escalating demands of substation.
Dominion anticipates that getting the DICE to the stage where it can be mass produced is still a few years off. The company will continue to work closely with vendor partners and internal teams to create this new standard in digital enclosures.
Biographies:
Jonathan Deverick, MBA, PE, Dominion Energy, Supervisor of Electric Transmission Long-Term Operaitonal Planning. Jonathan received his Dual B.S. degrees in physics and mechanical engineering and a Master of Business Administration from Virginia Commonwealth University. He joined Dominion Energy in 2008. For 15 years, Jonathan has provided various engineering services for Electric Distribution, Substation, and Transmission systems. His focus is designing and operating a grid that will protect the public, line personnel, and equipment while considering operational and customer impact.
Nilesh Bilimoria, PE, Dominion Energy, Consulting Engineer in the System Protection Standards group. He currently works on protective relay applications, support and scheme development with a particular interest in schemes using IEC 61850. Nilesh has been working with protective relays and substation engineering since 2008 and joined Dominion Energy in 2011. Nilesh holds a bachelor’s degree in applied science (BASc) in Electrical Engineering from the University of Toronto and is a licensed professional engineer in Virginia.
John Grimes, Dominion Energy, Lead System Protection Technician. His seven years of service have been in the System Protection Relay Lab. He has been involved with Dominion’s evaluation of IEC 61850 solutions since 2018. He graduated from the University of Pittsburgh at Johnstown, with a B.S. degree in Electrical Engineering Technology and Minors in Mathematics and Physics.
Jake Schlank, Dominion Energy, Supervisor of Substation Data Communications Engineering. Focused on company’s IEC 61850-based Digital Substation initiatives since 2019. He has been with Dominion for nine years, working as an engineer in System Protection and Communications Engineering. He holds a B.S. in Electrical Engineering from Virginia Tech and is a member of IEEE and NATF.
Savit Vajpayee, MS, Dominion Energy, Engineer III, has 10 years of experience in Electrical and Power Systems Engineering. Over the past 4 years he has focused on Substation Standards and Protection. His experience also includes work in Power Systems Protection, Smart Grid implementation and Renewable Energy resources. He has completed several projects and internships in Cyber Security, Data Centers and Wireless communications. Savit graduated from Uttar Pradesh Technical University, India with his Bachelor of Technology (B Tech.), 2011 and from Syracuse University with his Master of Science in Electrical Engineering (MSEE), 2013.
Philip VanSant, PE, Dominion Energy, Manager of the Electric Transmission Engineering Standards group. He has 18 years of industry experience in system protection and system operations. Philip has a Bachelor of Science in Electrical Engineering from Virginia Tech.
Keith Gray, MS, PE, Power Engineers, Project Engineer, received his M.S. in Electrical Engineering from Southern Illinois University Edwardsville in 2005. He is a protection, automation, and control engineer with 18 years of experience working for POWER Engineers. He has focused on digital substations and IEC 61850 since 2014 and contributes to the IEC 61850 standards via IEC TC57 WG10. He is currently participating in Cigre working group B5.68 covering the IEC 61850 engineering process. He is a licensed Professional Engineer in the state of Idaho.
Rich Hunt, MS,PE, Quanta Technology, EXECUTIVE ADVISOR, has over 30 years of experience in the electric power industry, specializing in protective relaying. Over the last 15 years, he has focused on specifying and designing digital substations, including IEC 61850, process buses, non-conventional instrument transformers, networking, and time synchronization. Rich earned his BSEE and MSEE from Virginia Tech and is a Senior Member of IEEE, a member of the Main Committee of the IEEE Power System Relaying & Control Committee, and a registered Professional Engineer.
Tim Grigg, PE, Schweitzer Engineering Laboratories (SEL), Senior Engineer is currently working in R&D Communications supporting Software Defined Networking. Tim has been an engineer working with automation products and systems for almost 28 years. He has been with SEL the last 13 years working with SEL automation products and supporting customers with IEC 61850 applications.
