By A. Kulmala, O. Raipala, P. Hovila, ABB, Finland, and B. Yazadzhiyan, R. Dantas, C. Scoble, UK Power Networks, United Kingdom
ACKNOWLEDGEMENTS
This work is carried out through the Constellation project which is funded by Ofgem’s Network Innovation competition. The project is led by UK Power Networks.
This article describes a virtualized, centralized protection concept that has been developed as a part of the Constellation innovation project. Protection functionalities include bay level protection functions that operate based on local site measurements and wide area protection functionality that utilizes 5G communication between the distribution network substations and DER sites and aims to improve the operation of DER loss-of-mains protection and to reduce unnecessary curtailment of DER. Laboratory testing results of the protection solution verify that the virtualized centralized protection is able to fulfil the protection requirements. Protection functions operate with the set delays and with the same reliability as on the dedicated hardware on condition that the hardware fulfils the requirements defined for it.
Electricity distribution networks are undergoing major changes due to the increasing amount of distributed energy resources (DERs) and the need to further improve the reliability of electricity supply. As a result, there is a need to revise the existing protection, control, and operation principles and to add novel functionalities such as wide-area protection and active network management.
This article concentrates on protection aspects and describes a virtualized centralized protection and control (PAC) concept that has been developed as a part of Constellation innovation project. The high-level goal of Constellation is to develop and to demonstrate a decentralized protection and control architecture that enables introducing new functionalities to substations and DER sites in a cost-efficient and easily maintainable manner. The aim is that several solutions can locate on a single substation computer and can utilize the same computational resources and communication infrastructure.
At present, intelligent electronic devices (IEDs) are normally used to realize distribution network protection. These IEDs are responsible for protecting one feeder and have visibility only towards the one feeder they are monitoring and protecting. Another option is to centralize protection functionalities on a single unit at a substation, which enables protection solutions that require broader visibility for the whole substation. Currently, both the feeder specific IEDs and the centralized protection unit use dedicated hardware on which no other functionality can be executed. However, when new solutions are decentralized beyond the control center, the number of devices in the substations increases to accommodate the associated functionalities.
To overcome this challenge, virtualization technology, that has been developing rapidly and is now capable of hosting applications that have rigorous real-time and reliability requirements, can be utilized. Virtualization decouples software from the underlying hardware and enables deploying applications from different vendors on the same hardware.
This article describes a protection concept that virtualizes existing distribution network protection functions and adds new functionality to enhance the operation of loss-of-mains (LoM) protection of DERs. The paper will, at first, discuss the requirements for protection virtualization. Thereafter, the virtualized centralized protection concept is described. Finally, different testing phases to be conducted during the Constellation project are described and laboratory testing results are presented.
Virtualization Solutions and Hypervisors
Virtualization technologies can be divided into virtual machines (VMs) and containers for which a wide range of commercial and open-source virtualization solutions exists. For non-real-time applications, virtualization is a mature technology that is widely used in modern cloud infrastructure.
For example, control or monitoring can be managed via non-real time solutions. Protection applications, however, have very stringent real-time and reliability requirements and, therefore, the solutions used in cloud applications cannot necessarily be used when virtualizing protection.
If protection fails, the safety of the distribution network can be compromised. Substantial damage to network components can occur, fires can be started, and people can be exposed to danger of electric shock which can in worst case lead to death.
Hence, real-time operation of protection needs to be always guaranteed and, therefore, it is vital to select the right virtualization solution and to configure it correctly.
Hardware virtualization is based on abstracting the underlying physical hardware from the guest Operating System (OS), by using specific software components called hypervisors for allocating virtualized system resources and managing the execution of virtualized applications, called Virtual Machines (VM). Hypervisors can be further divided into Type-1 hypervisors that run directly on system hardware, and Type-2 that run on top of the host OS. Type-1 hypervisors are more suitable for providing predictable performance to virtual machines (VMs) than Type-2 hypervisors, as they directly interact with the hardware and have full control of the hardware resources allocated to the VMs. The host OS layer between the hardware and the hypervisor in Type-2 hypervisors can introduce unexpected and unpredictable delays to the hosted hypervisor and then to the VMs on the hypervisor. In addition, any security vulnerabilities and flaws in the host OS could potentially compromise the virtual P&C VM.
Kernel-based virtualization (KVM) means that virtualized applications are executed as part of an operating system. Applications with different safety and security levels can be run on the same hardware, protected from each other by means of software partitioning (also called virtual machines). The partitions can contain different applications with their own operating systems.
KVM offers features such a live migration, resource scheduling and control. For virtual PAC this helps in balancing the computing capabilities between two physical host machines and eliminates hardware dependencies. Live migration offered may not be desirable for critical SMV traffic as the failover and recovery times are in the millisecond range.
OS-level virtualization is the newest way of handling virtualization, where the OS components (such as memory access, file system and network access) are abstracted to the guest OS. In this case, the host OS is the same as the one used in each guest OS. Due to so tight coupling, OS-level virtualization imposes low (if any) overhead in terms of resource usage. OS-based services, such as backup and recovery, security management as well as integration with Active Directory (AD) are available, which can help in compliance management for CIP.
Different solutions have their own strengths and weaknesses, and these have been extensively elaborated in literature. Hardware Virtualization provides the strongest isolation between different VMs, whereas OS-level virtualized containers provide the lowest overheads. Main aspect to highlight is that with all mentioned solutions it is possible to achieve the required deterministic operation and reliability for P&C purposes.
Virtualizing Protection
The centralized protection concept is depicted in Figure 2. All protection functionalities at substation level are centralized on the substation computer. Merging units (MUs) at each bay accept measurement data from instrument transformers as input and transform this to digital format and send the data to the centralized protection and control unit via an Ethernet network switch. The MUs also convey the control commands initiated by the centralized protection functionalities to the breakers. The protection concept depicted in Figure 2 includes redundancy i.e., both the substation computers and the networking are duplicated. The other basic architecture type is hybrid with single centralized protection unit and numerical relays at each bay. Alternative types of hybrid architectures combining MUs and numerical relays are also possible.
To guarantee real-time performance of centralized protection it is important to consider the following aspects:
- Voltage and current measurements from the IEDs at each bay need to be received with a consistent and low enough network latency so that input data for the protection algorithms is available in a timely manner
- Access to the computing hardware must always be available when the application needs it so that calculations are always completed within the required time. There needs to be mechanisms to reserve both central processing unit and memory resources
- Accurate time synchronization using precision time protocol (PTP) is required. As communication between MUs and substation computers is an integral part of protection, also support for parallel redundancy protocol (PRP) needs to be available
Virtualizing this kind of application poses some challenges since the mainstream virtualization case does not optimize for any of the above-mentioned aspects. When the centralized protection application is decoupled from the dedicated hardware and other applications are also running on the same substation computer, special care needs to be taken to guarantee that the listed requirements are always fulfilled, and the real-time operation of protection is never compromised.
Both virtual machine and container-based virtualization solutions have been studied and both provide some viable options for virtualizing centralized protection. Initial studies were conducted for several virtual machine and container solutions and based on the results of these studies three solutions were selected for further analysis. These are VMware ESXi and KVM virtual machines and Docker containers on top of Linux patched with PREEMPT_RT. All of these are capable of providing the required real-time performance if properly tuned and supported by suitable hardware. The results reported in this article have been accomplished using ESXi since it has been selected as the main virtualization solution to be used in Constellation project.
Proposed Protection Concept
The Constellation project aims to develop a decentralized protection and control architecture in which several functionalities utilize the same computational resources and communication infrastructure.
This is accomplished through adding substation computers to distribution network substations and DER sites. These substation computers host a virtualization environment and can host various applications from different vendors, including protection applications. The Constellation decentralized architecture is depicted in Figure 3.
In the proposed protection concept, all protection functionality is virtualized and at site level centralized to the substation computer. Measurement data is obtained from the IEDs at each bay using IEC 61850-9-2 sampled values (SV) and IEC 61850-8-1 Generic Object-Oriented Substation Event (GOOSE) messaging is used to communicate bay level status information towards the substation computer and to send breaker control commands to the IEDs that initiate the breaker operation. Bay level protection functions operate based on the local site information. In addition to local protection functions, the protection concept also includes wide area protection functionality that utilizes information exchange between the distribution network substations and the DER sites. This communication is implemented over 5G network and uses routable-GOOSE (R-GOOSE) messaging.
The Constellation virtualized architecture within sites (distribution network substations and DER sites) is presented in Figure 1. Several solutions including virtualized wide area protection are integrated on the same substation computer. Each solution with its operating system is running in its own virtual machine and ESXi hypervisor manages the VMs and allocates hardware (HW) resources to them. In the selected architecture, virtualized protection uses a dedicated network interface card (NIC) in passthrough mode to guarantee consistent and sufficiently low network latency.
Wide area Protection Functionality: The aim of the wide area protection functionality is to improve the operation of LoM protection and to reduce unnecessary curtailment of DER. LoM protection is required to prevent unintentional islanding of distributed generation (DG) units. However, most of the local monitoring based LoM protection schemes, such as rate-of-change frequency (ROCOF), fail to detect islanding when the imbalance between local generation and demand are of minor scale. On the other hand, there has also been concerns of cascading unwanted tripping of DG units due to wide-area disturbances. The proposed communication-based wide area LoM protection concept tackles these challenges by utilizing R-GOOSE messaging over wireless 5G between the distribution network substation and the DER sites. Transfer trip functionality is used to disconnect islanded DG units, while ROCOF-based blocking is used to keep the DERs connected during system-wide events.
A safe reclosing functionality, which is used to prevent unsynchronized reclosing, is also included. The developed concept also includes a communication supervision functionality for verifying that the exchanged R-GOOSE messages are being correctly received. Local ROCOF based LoM protection at the DER sites will operate as back-up protection and will be always active unless blocked by the wide area ROCOF based blocking functionality. The basic idea of the wide area protection scheme is depicted in Figure 4.
Testing During Constellation Project
During the Constellation project, the developed virtualized protection functionality will be tested and demonstrated in several phases. This article will present laboratory testing results that verify that the strict real-time and reliability requirements set for protection can be fulfilled in a virtualized implementation, and that the implemented novel wide area protection functionality enhances the operation of LoM protection. The next step of the Constellation project will be integration testing in the Power Networks Demonstration Centre (PNDC) in Scotland, with the virtualization environment hosting all the functionalities being developed. After that, demonstrations will proceed to trial at sites located in UK Power Networks’ distribution area to verify the Constellation solutions in a real operational environment.
Laboratory Test Setup: The experimental studies were executed in a real-time simulation test setup which consisted of a real-time simulator and two centralized protection units as illustrated in Figure 5.
In this test setup, a network model depicted in Figure 6 was modelled and run by the simulator. The two centralized protection units received all their measurements from the simulated network via IEC 61850-9-2 sampled values. Similarly, all the binary signal exchange between the centralized protection units and the simulator was handled via IEC 61850-8-1 GOOSE messages. The simulator was also configured to simulate the communication delay of the 5G communication link. This was done so that the simulator subscribed all the GOOSE messages from the centralized protection units in the setup, delayed these messages as desired, and then sent replicas of the original GOOSE messages to the other centralized protection unit. That is, the centralized protection units were not communicating directly with each other but only via the simulator for enabling the simulation of the communication delays. Local variables, such as local circuit breaker statuses and trip commands, were not delayed as they also would not be in a real installation.
Testing Results: Extensive testing has been conducted to verify that the virtualized centralized protection is able to fulfil the protection requirements i.e., protection functions operate with the set delays and with the same reliability as on the dedicated hardware. The testing also verified the correct operation of the new wide area protection functionality. An example case is depicted in Figure 7.
An example of how the ROCOF based blocking functionality of the wide area protection scheme works is shown in Figure 7. Starting from the top, the first graph presents the voltage frequencies. The signals “f 33 kV Busbar A” and “f 33 kV Busbar B” are the computed voltage frequencies at the 33 kV busbar sections Main 1 and Main 2 (see Figure 6).
The second graph presents the resulting rate of change of frequency indexes computed from the frequency values presented in the first graph and the third graph respectively presents the ROCOF indexes computed from the DG connection point frequencies. Finally, the bottom graph presents the local LoM protection trip commands and the wide area detection block DG 1-4 ROCOF signals. In this simulation, the system frequency falls below 47 Hz (which is the underfrequency threshold) for a short period of time (approximately 300 ms) with a rate higher than the threshold (1 Hz/s). However, no false DG tripping occurs because the blocking scheme activates and sends block-ROCOF commands to all four DG units, and because the frequency returns above the underfrequency protection threshold before the operate time (500 ms) elapses.
Conclusions
The increasing amount of DERs is creating a need to enhance distribution network protection and control capabilities. The existing functionalities will need to be altered, while completely new functionalities will need to be added to guarantee resilience and efficient operation of the future smart distribution network. New technology advances in other industries such as software engineering (virtualization) and telecommunications (5G) can be utilized to support accommodating the increasing amount of DERs cost-efficiently and with less investments on hardware. This paper describes a new protection concept that virtualizes all protection functionalities and enhances the operation of DER LoM protection.
Laboratory testing results of the developed virtualized centralized protection solution verify that protection requirements can be fulfilled and that the new wide area protection functionality enhances the operation of DERs during wide area disturbances and prevents unintentional islanding.
Biographies:
Anna Kulmala is an R&D Project Manager at ABB Distribution Solutions. She has been leading and working in various European and national projects for more than 15 years. Her interests are focused on aspects of the future smart grids such as novel distribution network protection concepts, active distribution network management and smart grid architectures. She received the M.Sc. and Ph.D. degrees in electrical engineering from Tampere University of Technology.
Ontrei Raipala received the M.Sc. and Ph.D. degrees in electrical engineering 2009 and 2018 from Tampere University of Technology. He currently works as a senior design engineer at ABB. His research interests include integration of distributed energy resources and power system protection.
Petri Hovila is a senior principal engineer in ABB Electrification, Distribution Solutions and is responsible for research programs. He is also the program manager for the Green Electrification 2035 program. He has authored or co-authored a number of conference papers and is a co-inventor in several patent applications.
Boris Yazadzhiyan is an Innovation Program Manager in UK Power Networks focusing Whole Energy Systems and Consumer Vulnerability. He is an experienced technical expert who led the Constellation project: a world first smart substation trial. The collaborative project with ABB, Siemens, Vodafone, GE and University of Strathclyde will see powerful computers installed in a series of substations, optimizing utilization of the substations to free up capacity and help facilitate the rise in renewable energy generation. Boris holds a Bachelor of Engineering from the University of Sunderland and a Master`s degree in Energy and Sustainability with Electrical Power Engineering from the University of Southampton.
Rui Dantas was born in Barcelos, Portugal, in 1990. He received B.Sc. and M.Sc. degrees in electrical and computer engineering from the Faculty of Engineering of the University of Porto, Porto, Portugal, in 2013. He received his Ph.D. degree in high-voltage DC grid discriminating protection from School of Engineering, Cardiff University, Wales, U.K., in 2017. Dr. Dantas currently works at UK Power Networks as a Senior Protection Settings Engineer. His interests include innovation, IEC 61850, standardization, protection applications and IED testing.
Colin Scoble is an Experienced Protection Engineer with a demonstrated history of working in the utilities industry. He is skilled in Energy, Power Systems, Renewable Energy, Power Plants, and Project Engineering. He is a strong operations professional with a Master’s degree focused in Power Distribution Engineering from Newcastle University.
