by Marco Nunes, ABB, Switzerland
The first centralized protection architectures pilots started 45 years ago, under Westinghouse and GE, with microprocessor-based technology relays, allowing multifunction protection and control in same devices.
Parallel paths were followed in North America (DPU) and Europe (SPACOM), almost simultaneously, with first microprocessor-based protection relay (SPAJ 3M2) prototypes issued to the market during the year 1980 and commercially released February 1982 (formerly Stromberg in Finland) (Figure 1). Few years later, serial communication was added and control in also during 1987, to have protection and control under same box, which was a tremendous shift in the way to design protection and control architectures.
Westinghouse (acquired by ABB in 1990) in collaboration with EPRI installed a P&C system called WESPAC with fiber-optic based network communication between data acquisition unit in switchyards and protection clusters in control room, back in 1978-1986. This 500kV substation owned by PG&E was one of the very first to have utilized Digital Communications in Substations.
The adoption of microprocessor-based technologies created and enabled a different approach in the protective relay development, requiring new competences, on hardware-software and applications specialists, combined to deliver turnkey boxes, each for a specific price, segment or application.
The first investigations about centralized protection based on the IEC 61850 standards started back in 2004 in Finland within a product management team, and the vision at that time was already to have “digital enabled” relays which could be standalone protection relays, but also Merging Units, be compatible with upcoming “centralized” technologies, based on standard technology, like IEC 61850 and Ethernet. (Figure 2).
At the very early stages of the new Relion platform development, the 61850-9-2 standard was already in the roadmap to enable fully centralized and digitalized substation architectures. The overall platform and tools have been considered since the beginning to support the centralized architectures, and therefore natively compatible. The time synchronization, redundant 61850 network and application configuration environment were all considered at that time. (Figure 3).
First pilot installations of IEC 61850-9-2 based centralized protection systems were executed back in 2007 after a research project together with universities and distribution utilities.
Centralized and Virtualized Architectures Standardization
A single vendor push in the market is not sufficient. New technologies need promoters, early adopters, open standards, committees, coalitions to support disruptive technologies.
The IEEE PES PSCC committee meeting in January 2021 approved the formation of P21 group: investigate the possibility to create a standard to address “System architectures supporting the virtualization of substation protection and control applications,” complementing underway IEEE PES PSRC Committee WG H45 developing PC37.300 “centralized protection and control applications.”
A paper based on the IEEE PES PSRC WG K15 report on centralized protection won the 2017 PES Prize Paper Award.
An EPRI USA project 3002017165 is underway since August 2019 with several utilities and industry stakeholders looking at a “Common Substation Platform (CSP)” and defining the application use cases.
IEEE PES, Protocols and Communication Architecture Subcommittee (P0) – P21 SG: System architectures supporting the virtualization of substation protection and control applications, conducted several surveys with different vendors during early 2022 to investigate the technological feasibility.
Other initiatives and coalitions have or will be announced sooner, initiated and supported by Intel Market Ready Solutions team. Intel has been one of the main promoters of the Protection Virtualization technology with public articles released back in 2018, and earlier, under the lead of Prithpal Khajuria.
This led to the development of the “Centralized Protection and Control” (CPC) solution that has been deployed in more than 40 countries, in all continents and in multiple industries, Power Utilities, but also semiconductors, oil & gas, marine, renewables, datacenters infrastructure and defense.
The virtualized version of the CPC planned for end of 2022, will enable further collaborations, as new IT/OT integrators will be able to add the protection and control software to their solutions.
Centralized Protection – first step towards protection virtualization
The main benefits of a centralized protection platforms:
- Innovative smart substation control and protection device with its advanced technology based on edge computing, standardized and widely available hardware
- Reduced operational and maintenance expenses with optimized system reliability
- Enhanced cybersecurity features with multiple hardware and software layers
- Easy and efficient process management with station-wide process visibility
- Improved operational safety with centralized protection concept
- Easy adaption to changing network protection requirements with scalable software architecture
It is not only about converting the conventional protection devices into a centralized unit, for the same protection functions. The solution envisioned is as a complete substation management system, including the HMI single line diagram, gateway, centralized disturbances recording, multiple devices in one, including low impedance busbar protection and arc protection, as open and interoperable platform based on the IEC 61850 standard. The benefits need to be evaluated as complete substation automation system. And the more functions are considered under the centralized protection, more obvious the benefits will be in terms of OPEX and CAPEX. (Figure 4).
While Centralized protection is sometimes understood as risky, disruptive, or complex, it can be deployed in different steps, from upgrading existing substations, to mixed architectures up to totally centralized or virtualized architectures (Ganoo, Joshi, Valtari, & Niveri, 2020), from the Bay level point of view, the CPC centralized or VPR/CSP virtualized architectures. (Figure 5).
The solution, centralized or even virtualized can accommodate different needs, plans, or requirements, with hybrid architectures (conventional relays with protection + merging unit functionality + 1 centralized protection) or full centralized (pure merging units + 2 centralized protections as main backup.)
Protection Virtualization Introduction
Sometimes, the people met during events and meetings were confused about the wording -“Does virtualized mean it is on the cloud?”, or “What does it even mean – virtualization“? (Figure 6).
Virtualization is not a must for every company. It mostly depends on internal IT strategy, resources or competences. When Power Utilities have already a wider experience with Virtual Machines and applications virtualization,
Virtualization additional benefits: (Figure 7).
- All protection functions, settings, and control configurations including local HMI become a single file which can be backed up, and easily maintained over time, like a digital twin of the whole system
- Relies on ready off-the shelf, standardized and ruggedized hardware according to standard IEC61850-3-1
Drivers for protection virtualization:
- Legacy assets to be replaced, with latest available technologies
- New technologies to provide equal or better reliability compared to traditional solutions
- Optimize and simplify the different layers of protection, automation, control and monitoring
- Reduce deployment time and maintenance costs
- Increased flexibility needed to accommodate new network architectures and bidirectional power flows, requiring new protection functions
- Interoperability and compatibility between different vendors and operating systems, leading to wider standards acceptance instead of proprietary protocols
Substation Protection and Control virtualization allows mixing different performance requirements:
- Protection: requiring sub milliseconds predictive reaction time, with microseconds time stamps, 100% real time
- Control: requiring milliseconds to seconds reaction time, and compatibility with different generations of Operating systems with local and remote interfaces (SCADA, DCS,)
The first substation pilot with “Protection and Automation” in a virtual machine environment was deployed during 2020, during the pandemic, at the Noormarkku primary substation in Finland (Keskinen, Reikko, & Ristimäki, 2020.) A prototype system was deployed with CPC centralized protection and upgraded with CPC VPR (virtualized protection) enabling highly accurate and time-synchronized touch voltage data from the network in order to detect and predict faults more precisely, while reducing the amount of false tripping.
The system contributed to increase the amount of gathered data for research, analysis and optimization purposes. Based on the conclusions, additional protection functions or applications were made possible to be upgraded to the Protection System (licenses and configurations update) directly from the central management system (VPR asset management system). The VPR architecture allows different VMs or containers of protections functions to be updated or added separately without shutting down the whole substation protection (Valtari, 2021)
The solution deployment was based on an Advantech ECU-579 with an Intel Xeon Gold CPU 16 cores 2Ghz. The OS/Virtualization Environment consumed 3 cores, and 2 instances (main + backup) of the virtualized CPC have been installed as virtual machine (4 cores for each). Each instance of the virtual CPC can protect and control up to 30 bays/breakers based on the 30 SV streams IEC61850-9-2LE from the conventional merging units with PRP cards.
Protection Performance of VPR under Virtualized Environments
a) The first recommended step is to test first centralized protection architectures, based on IEC 68150, without having to deal with Virtualization aspects. Those Centralized protection devices, either based on upgraded conventional IED’s or Edge Computing Servers, will enable operators and engineers to get familiar with the 61850-9-2LE technology, GOOSE, GPS PTP clocks, configuration 61850 managed switches with PRP, and naturally the applications configuration (function blocks and how to connect the different streams to single and multi-bay protection functions). Users will also be able to investigate other substation management features embedded in centralized protection solutions, like centralized disturbances recording, alarming, station single line diagram configuration, which were made through different software, different vendors and different platforms
b) The main benefit of a centralized protection device from a defined vendor is the real time performance and deterministic behavior, tested according to conventional IED’s standards and KPI’s
c) The challenge with VPR architectures, is the responsibility of the whole system, furthermore of the different layers (Hardware, VM Hypervisor-OS and the Applications) have not been testing and validated as a system by someone. (Table 1).
Selection and Deployment Recommendations for Virtual Protection and Control Architectures
The first surprise when we started discussing Virtualized protection, was the different understandings in the market.
Some companies assumed that every single individual relay would just be “converted” into an individual “containerized VM”, totally isolated, and that the hypervisor would allocate the different hardware resources to make it run in real time.
ABB’s approach to VPR has been rather different. We started from the existing CPC workstation, which in a factory is a substation management system, including gateway, SLD, protection functions, automation as a totally encapsulated and standalone solution, engineered at once from 1 up to 30 SV streams, with same application configuration file.
The reason why “individual and isolated VM’s” for every virtualized relay is not recommended today, is due to the system behavior. We still need to pin VM’s or containers to individual cores in order to achieve real-time and deterministic behavior.
Having 30 different virtualized relays to be allocated to 20 cores, would require the Hypervisor/OS to allocate dynamically the hardware resources to the different VM’s or containers. Doing that, the system could no longer be real time and deterministic, introducing delays and queues in the process. Therefore, the recommended architecture is to keep VM’s and Containers with strictly allocated CPU cores resources, not interfered by other processes or other VM’s and containers.
Nevertheless, the technology will continue evolving, more scalable in terms of functions and streams/bays, from small compact substations up to very large power substations including low, medium and high-voltage switchgears protection and control.
Once the technology clarification is settled down, the choice of the OS/Hypervisor by the customer is the second challenge. As there is no consensus in the market, those virtualized protection environments need to be tested and validated for the majority, and the market will concentrate on few validated and recognized solutions. Final choice might be a “commercially available vendor or open-source,” basically 2-3 type of Hypervisors.
What to take care of when heading to protection virtualization
- Prefer a stepwise approach, with hybrid architectures, IEC61850-9-2 compatible IEDs, expand to centralized protections architectures, moving applications to the centralized layer and finally migrate to the virtualized solution in the future. This approach ensures a competences development inside your organization while ensuring full flexibility towards protection virtualization migration
- Make sure you select Domain experts, independently (hardware, hypervisor, virtual relay & control) or packaged and validated solution
- New entrants can be good to expand the technology, try new solutions in laboratories, but do not replace 30 years of experience on protection algorithms design, validated already by millions of microprocessor based IEDs all around the world
- Verify your internal competences, available deployment resources or partners. Bundled certified packages ensure that you have an accountable vendor for the solution and training, with reliable partners
- Check the reliability of the different components. Costs savings in selecting cheap computer, non-verified open-source platforms, and new entrants protection applications can rapidly lead to massive penalties, blackouts, network safety and security concerns
The substation Protection and Control virtualization also allows other applications to be merged into the same system, and this includes Asset management, Monitoring and Diagnostics of primary equipment.
Testing and Commissioning Virtual Protection Architectures
The testing of traditional PAC and centralized CPC architectures have proven (Sivesind, Xavier, & Modak, 2021) to be similar. Majority of relay test set manufacturers have introduced GOOSE and SMV (Sampled Measured Values) simulation capabilities, through the communication network. Primary and secondary injections, tripping circuit testing are possible same as before. The cabling from the instrument transformers or sensors, switchgear terminals and other alarms are cabled to Merging Units, like to any conventional relay. (Figure 8).
Main difference is that Merging Units are mostly brainless, and protection functions, logics and configurations are remotely present in a centralized or virtualized environment.
In the SRP (Salt River Project – Phoenix/USA) pilot, the centralized protection could be set to Test mode and allowed injection of GOOSE messages and SMV to verify the configuration, performance and different protection function timings according to the predefined settings. Under a virtual environment, the procedure is identical to verify the complete assembling of different layers of hardware (HW), hypervisor/container (VM) and the virtual protection applications (VPR). (Figure 9).
The centralized (CPC) and virtualized (VPR) applications of the CPC, are both available with same functionalities available through the built-in webHMI.
The experience of CPC projects deployed around the world provided confirmation on the target benefits of the system:
- Simple addition of new protection applications, without any hardware change
- System performance and reliability improvement with advanced features, like busbar differential protection, without any extra wiring or additional devices
- The CPC and VPR solutions do provide the same range of applications and features, which will continue getting new upgrades every year. There is no difference between the Centralized or Virtualized Protection functionalities, however the VPR will allow to cover more bays and combine more protection functions under the same hardware, even for the years to come, no matter how large the substation will be.
- The CPC is an excellent, and necessary first step for Utilities and Industries looking for Virtualization as mid or long term objective. Some companies might not need to step into Virtualization immediately but will prefer a cautious, hybrid approach combining both technologies (PAC and VPR) (Joshi & Valtari, 2019)
Protection as a Service (private DataCenter)
Actual centralized and virtualized protection and control architectures are still in the same location as traditional systems, however growing distributed generation, bidirectional power flows, changes in energy consumption with new loads (EV) or producers (storage, renewables), are calling for more data collection and processing.
Centralized and Virtualized architectures are recommended to remain in the substations due to the stringent requirements in terms of data quality, avoid any loss of packets (real-time data to be processed for protections functions have time performance constraints).
However, growing communication infrastructure in the dense and metropolitan areas, with private fiber optic rings connecting the different substations, the communication latency could be good enough for “remote” protections. The centralized and virtualized architectures are ready for such projects, including the settings ready to accept some delays in the SV streams, depending of the network latency.
We might see during the next 2-3 years, some pilot projects already for Protection as a Service, in Power utilities, from their own DataCenters, and own communication FO infrastructure. For those kinds of applications, it will be required to have MUs 61850-9-2LE with backup protection functions, in case, for any reason, the complete communication network would break down.
Conclusions: One of the most important aspects to be considered in testing, and deploying centralized or virtualized protection architectures, is that those initiatives should not be handled by a single champion or specialist. Those projects need to fit into a company research and innovation plan, with a dedicated team assigned and necessary funding to set up a lab, and deploy those architectures on the field. Very frequently, competences and knowledge are lost, in case it was run as individual initiative, and will be delayed in time for many years if no funding, no lab or no team is allocated.
Initiatives to deploy digital substations based on 61850 need to be inclusive within the whole organization.
Biography:
Marco Nunes works as Global Product Marketing Manager for ABB’s Distribution Solutions, in charge of Digital Substation Products. He has a BSc in Electrical Engineering/Power Systems from the HES-SO University of Applied Sciences and Arts Western Switzerland, a Swiss Federal Diploma of Commerce and a Postgraduate Diploma in International Business Management from the University of Cumbria in UK. He has held Marketing & Sales and Product Marketing positions for ABB Switzerland before joining the Global Product Group organization in 2017. His primary areas of interest are innovations medium-voltage switchgears technology, next generation substation automation systems, and how to match custom
