By Onur Durak, OMICRON electronics, Germany
Imagine you are a Substation Automation Engineer in the early 2010s. You are on the road to commission a project, and you receive an email at the airport with the subject line: “Update on the RTUs.” The email reads: “I hope this email finds you well. We had to change the control center signal mapping; the latest backup is attached. Could you please test the mappings again?” There is a good chance the email doesn’t find you well and is responded to with “Well, it depends!”
The grid is changing along with the well-proven practices that we, electrical engineers, were taught, but it is changing relatively slowly compared to the industries that it is supplying. As we enter an era of centralization and virtualization of the protection and automation systems, these practices are beginning to allow substation engineers to replace their “it depends” with “sure will do.” This article focuses on the current testing practices for substations, combining the HMI observation checks within a PC without the need for hardware IEDs. It also addresses the time-saving benefits of this approach in the context of a changing grid.
Control Center Communication via IEC 60870-5-104
As we are in the era of peopleless substations, the importance of reliable communication with the control center is crucial. It is already common practice to use Remote Terminal Units (RTUs) for data concentration and protocol conversion, as well as to establish a real-time connection between the substation and the control center. One of the most common communication protocols is the IEC 60870-5-104 communication protocol (commonly referred to as IEC 104), which requires translation from the MMS protocol of the IEC 61850 standard used within the substation’s station bus network. (Figure 1).
There are several practical reasons why IEC 104 is often used for control center communication instead of IEC 61850. One of the main reasons is that IEC 104 has been widely adopted and used for decades for controlling and monitoring geographically dispersed processes in SCADA systems, especially in substation automation. Because of its long-standing use, many utilities have established infrastructure and expertise in IEC 104, making it a more convenient and cost-effective choice for control center communication.
On the other hand, while IEC 61850 offers more advanced features and better interoperability within substations, it is primarily designed for communication within substation environments. Although the use of the IEC 61850 protocol for wide area networks is becoming more popular recently, its establishment and standardization will take time. Therefore, IEC 104 remains a practical choice for communication between control centers and substations due to its established use, compatibility with existing systems, and suitability for wide-area networks.
By integrating these two standards, utilities can achieve seamless communication from the substation level to the control center. This integration allows for real-time monitoring and control of the power grid, enabling faster response times to faults and anomalies. Additionally, it facilitates better data management and analysis, leading to more informed decision-making and improved grid stability.
The use of RTUs as gateways and protocol converters plays a crucial role in this integration. RTUs can convert IEC 61850 protocols to IEC 104, ensuring that data from modern digital substations can be effectively communicated to control centers. This not only enhances interoperability but also reduces the need for extensive infrastructure changes, making the integration process more cost-effective.
Software RTUs: Why are they the future?
Forget about clunky, dedicated hardware. Imagine an RTU that exists entirely as software – no additional physical boxes, no wiring, just pure digital magic. That’s the power of a Software-RTU (also called Soft-RTU), which is revolutionizing the way we connect to and control substation automation systems. Think of it as a highly intelligent software agent that can run on almost any standard computer, ranging from rugged industrial PCs to virtual machines in the cloud. Unlike their traditional hardware counterparts, Soft-RTUs boast incredible flexibility and scalability. Need to add a new server? No problem! Just configure it in the software. Want to integrate with a different protocol? A quick update, and you’re good to go. These features drastically reduce implementation time and costs, making system expansion a breeze.
This software-centric approach not only cuts down on hardware expenses and maintenance but also opens up a world of possibilities for advanced data processing and integration. Since Soft-RTUs operate in a standard computing environment, they can leverage existing IT infrastructure, security measures, and powerful analytics tools with ease. This means you can gather data, process it at the edge, and seamlessly send it to your SCADA system, historians, or even cloud-based platforms for deeper insights. Soft-RTUs’ ability to act as versatile gateways between different industrial protocols makes them invaluable for modernizing legacy systems and bridging the gap between operational technology (OT) and information technology (IT). If you’re looking for a cost-effective, adaptable, and future-proof way to manage your remote assets, it’s time to let a Soft-RTU do the heavy lifting. Even though they are one of the most critical assets in the substation, they do not carry the burden of being responsible for protection functions, require less computing power, which makes them easier to digitalize earlier and safer than the IEDs
Digital Twins for substation automation vs. simulations
The Digital Twin has recently become the buzzword in the PAC world. The digital twin is a virtual representation of a physical object, and it has been around for many years. Thanks to better-performing, more reliable CPUs, digital twins are now gaining popularity in the grid. Using digital twins of IEDs in electrical systems to monitor asset conditions, perform risk analysis, and test the protection configurations is becoming the main trend, along with using vPAC systems in substations.
In a nutshell, a digital twin of an IED represents a far more comprehensive and dynamic virtual replica of a specific physical IED. Conversely, an IED simulation feature within a testing tool serves as a software-based proxy designed primarily to emulate the communication behavior and data model of a real IED. Its main purpose is to facilitate the testing of other devices or systems that interact with the IED, such as a SCADA master, other protection relays (for GOOSE communication), or devices consuming Sampled Values. This simulation typically uses the IED’s SCL configuration file to accurately mimic its external communication interface and data points, providing a static yet functional representation, which is essential for validating interoperability and protocol adherence during specific test campaigns. This makes it more convenient for the “on-the-go” testing. After all, it is costly to purchase an additional seat on a flight for an industrial PC that is capable of running the entire system when you need to test the changes you were asked to make en route to the substation
Signal testing in a virtual environment: an example
When asked to describe virtualization in our industry, I simply start by asking: “What is an IED?” The answer is quite simple: an IED is a microprocessor-based controller that specializes in handling protection, control, and automation functions. To me, that sounds like a PC with reduced, specialized capacity that can run the “digital twin” of the IEDs that we use in our substations – call it virtualization. Virtualization is often achieved through hypervisors that create virtual machines (VMs), which emulate hardware and allow you to run multiple operating systems on a single physical machine. Each VM acts like a separate computer, providing strong isolation. Containerization, on the other hand, virtualizes at the operating system level. Containers share the host OS kernel, making them lightweight and faster to deploy. They package an application with its dependencies, ensuring consistency across different environments. (Figure 2). A cutting-edge concept like vPAC (Virtual Protection, Automation, and Control) demonstrates how software-defined solutions are pushing the boundaries of industrial capabilities, especially in power systems. vPAC envisions a future where traditional, physical protection relays and other specialized IEDs are replaced by virtualized software functions running on powerful, standard industrial servers (often called IPCs).
After a short break of diving into the virtualization in our industry, let’s get back to the topic of being capable of testing the signals utilizing an approach. To do that, we do not need the entire virtualized solutions, for the reason explained briefly before, “simulation”. We will get back to the potential virtualization benefits at the latter end.
What does this have to do with our approach? Well, let’s put the pieces together with an example, firstly setting our electronic devices to flight mode again and creating our architecture. (Figure 5).
1) Power on the virtual machines: Have a PC available that has a hypervisor suitable for deploying multiple virtual machines. In this approach, Virtual Machine configuration is used. So that we can easily deploy the tools as:
a. VM1: the testing tool b. VM2: the RTU
c. VM3: the HMI d. VMx: whatever is necessary for the moment, but focusing on the signal testing, these 3 will do fine. We should already have our testing bed available. (see Figure 3).
2) Configure the virtual network: Depending on the project, it is easy to configure a virtual Ethernet switch within the hypervisor, even connecting it to a physical adapter. However, since we are working entirely in a virtualized environment for the purpose, it should be enough to create a network as shown in Figure 4.
a. vSwitch-1: This virtual switch creates the Station Bus, where we will have our simulated IEDs, the HMI, and one of the RTU’s port for the MMS communication. Along with those, a port of the testing tool has to be connected to this bus.
b. vSwitch-2: In our case, this is not required, but to make everything proper, just in case, we will also have a Process Bus for the BCU/BPU IEDs and the PIU/MU/SCUs.
c. vSwitch-3: This virtual switch is configured to connect the Control Center port of the Gateway. Also, in this setup described here, another port of the testing tool will be connected as well. The testing tool will also simulate the Control Center as a client to the Gateway’s IEC 104 Server
d. vSwitch-4: This virtual switch is configured to connect the Management ports of all the devices deployed in the hypervisor such as RTU, testing tool, etc.
3) Along with subtasks, in 2 steps, this solution made us capable of testing. For this, we need some of the configuration files, such as:
a. SCL file of the substation: For the signal testing of the Control Center communication, it is sufficient to have the SCL file to import into the testing tool. This will allow the testing to simulate the IEDs. In other words, “virtualize” the IEC 61850 signals of multiple IEDs so that the Soft-RTU can be a client to the MMS protocol of the IEDs
b. RTU configuration: Obviously required for the testing. In the end, we are testing the protocol translation in this case. The main idea is the “to-go” testing, therefore, the revised configuration can be updated easily through the management port
c. HMI Runtime: In our case, this hasn’t been updated but if we want to combine the signal testing along with the HMI tests, this is valuable
Feeding above documents to our setup, will give us a view as visualized on Figure 6. A single screen view where we have the testing tool to observe the testing that includes the simulated IEDs as per the SCL file, the HMI runtime to observe the displays, runtime of the RTU’s with a communication summary; “typically” the substation.
To configure the test cases, it is up to the testing tool. There are multiple tools in the market for testing, and a few virtually deployable equivalents. For this testing method, a virtually deployable testing tool is not a must, but a VM creates many opportunities for approaches. Diving into one of those tools, we are now creating the test case, or simply re-running the pre-created one. In this tool, it is required to import the required mapping lists. This approach allows the user to directly evaluate the signals. In a few simple steps, as described in Figure 7, the user can map the signals for a focused view. Followingly, defining the steps of the test case, choosing the command route (IEC 61850 <-> IEC 104), setting parameters, and execution. It is also convenient to have a report of the test results immediately. This test tool also allows automated assessment of the expected results, which is suitable for Gateway testing; however, in our case, HMI is under test. Therefore, a manual assessment option is favorable. The testing tool allows the drop-down notes to be added to the report, which can easily be shared with colleagues who are asking for verification of the signals due to the revised configuration.
No need to writing long e-mails to explain the issues, only “see attached for the result and notes” within an e-mail is enough. We might have the time to verify during a flight, but not enough leg space to do the corrections.
Why should we have this?
As mentioned above, testing is crucial. The protection tests are “arguably” more important, but before moving to that slightly more complex area, I will stay on the signal testing part for the reasoning and as a starter.
Thanks to (or due to) the complexity in the system, availability of the signals, and faster speeds accompanied by higher bandwidths, the number of signals to be transmitted from the substation to the control centers has been boosted. I can hear the sarcastic “yeeey” chants from our fellow field engineers. Traditionally, thorough substation testing requires significant hardware investments, complex wiring, and time-consuming physical setup in a lab or even on-site. This often means limited test scenarios, high costs, and a bottleneck in development and deployment cycles. Who has a warehouse suitable enough to have the real HV equipment for the FAT? Or how convenient is it to utilize it every time? Highly questionable. And more importantly, the Boehm’s Law…Named after computer scientist Barry Boehm, states that the cost of finding and fixing a defect in engineering grows exponentially with time. This means that later in the development lifecycle, the more expensive it will be to resolve. This approach from software development can also refer to engineering of substations.
So far, this article hasn’t focused on testing in virtual environments. The method detailed here is only a part of it and can be seen as the grounds for virtualized testing. This is precisely where virtualization emerges as a game-changer. Imagine, only by configuring some additional virtual switches and ports, it gives us the chance to test the system not only partially, routing them to testing tools, which will allow the engineers to test a live system before a firmware upgrade. Let’s take these thoughts to a further approach.
Virtualization (and also simulation as a coequal approach) allows you to have a substation on your working desk, giving the technicians the option to pre-qualify their engineering in real-time, configure once, and make minor edits to the setup. As soon as we have our “Substation-XX_first-draft.scd” file available, it is on our hands to qualify/validate immediately, allowing us to find the errors or overlooked issues. Virtualization also allows us to share the reference documents easily, either cross-team or with external partners. Imagine the file “Substation-XX_absolutely-the-final-revision.scd” file on a cyber-securely protected cloud, where all you need is sharing the where the folder is to our colleagues responsible for the FAT. They know the procedure, they know what to test, plus now they have everything in a single folder, even reducing the preparation times for the FAT. Virtualization makes engineers happy because they can focus on more important parts, such as system tests, injection, etc. As a fellow commissioning engineer, I would admire the confidence to walk to the substation, knowing that every bit is virtually proven to work out. And, back to Boehm’s law, we have a reduced cost related to saving time due to a lesser amount of finding and fixing problems during commissioning and SAT. Or, as said, even on the way to the SAT, considering the flight safety measures.
The widespread digitalization of substations, moving from hard-wired logic and discrete components to network-connected Intelligent Electronic Devices (IEDs) and software-defined functions (like in vPAC/cPAC systems), has fundamentally changed the approach to maintenance and security, ushering in the era of regular firmware updates. In traditional, isolated substations, security was often a matter of physical hardening and air-gapping. However, with the integration of communication networks (like IEC 61850) and increased connectivity, these previously isolated operational technology (OT) environments are now exposed to the same evolving cyber threats as IT networks. Regular firmware updates are no longer just about fixing bugs or adding new features; they are a critical cybersecurity imperative. Each update often contains patches for newly discovered vulnerabilities (Common Vulnerabilities and Exposures – CVEs), closes potential backdoors that could be exploited by malicious actors, and strengthens the device’s defenses against sophisticated cyberattacks. Neglecting these updates leaves digital substations exposed to potential disruptions, data manipulation, or even complete control compromise by adversaries, making a robust patch management strategy an essential component of modern grid security.
Here is the downside: While the cybersecurity imperative for regular firmware updates in digital substations is undeniable, utility companies often face significant hurdles and hence a reluctance to apply these upgrades immediately. The primary reason for this hesitation stems from the criticality and complexity of substation operations, which demand absolute reliability and stability. Any change to a live system, including a firmware upgrade, carries an inherent risk of introducing unforeseen bugs, creating incompatibilities with existing equipment or configurations, or causing unintended operational disturbances. Therefore, utilities must undertake extensive and rigorous testing after every firmware upgrade, a process that is far from trivial. This testing involves replicating the substation’s specific configuration, running comprehensive functional checks, validating protection schemes, and ensuring seamless interoperability within the entire system – a time-consuming and resource-intensive endeavor that often requires dedicated testbeds and skilled personnel. The potential for an upgrade to disrupt service, even momentarily, often outweighs the perceived immediate cybersecurity benefit, leading to a cautious, delayed, and highly controlled approach to firmware deployment.
This is precisely where virtualization provides a transformative solution to the utility’s dilemma regarding firmware updates. By enabling the creation of exact (or close-enough) digital replicas of substation environments, utilities can now conduct the necessary rigorous testing without impacting live operations or investing in costly physical testbeds for every scenario. These virtual testbeds can be spun up on demand, allowing for parallel testing of new firmware versions on multiple simulated substation configurations. Crucially, the virtual nature of these environments facilitates unprecedented automation of testing processes. Automated test scripts can simulate millions of operational scenarios, even maybe inject various fault conditions, and systematically verify the system’s behavior and the efficacy of protection and automation schemes post-upgrade, all at speeds impossible with manual methods.
This comprehensive, rapid, and risk-reduced validation process provides utilities with the confidence to apply necessary firmware updates more frequently, ultimately reducing their exposure to cyber threats and ensuring the continued stability and security of the grid with minimal, if any, disruption to power delivery.
Biography:
Onur Durak studied electrical engineering at Kocaeli University in Turkey. His primary areas of work were distribution and transmission systems, and he gained experience with multiple aspects of the power grid in various roles. Currently, he enjoys giving back to his community through projects that ‘reduce engineering time’ while working as a Product Manager of ACC Testing Tools at OMICRON. He is also a bedroom musician who has been working on his first album for the past 12 years.
