By Herbert Falk, UCA International User Group, USA
The UCA International User Group (UCAIug) Interoperability tests (IOPs) have occurred bi-annually since 2011. In looking forward to the 2021-2022 IOP, it is worthwhile to reflect what was accomplished in 2019. To understand the success of the IOPs, the objectives of IOPs need to be analyzed.
Unlike many plug fests, the UCAIug IOPs does not concentrate on testing for success. Rather the IOPs are looking for problems that occur during integration of multi-vendor IEC 61850 systems and thus are looking for “fails.” It is through detecting these types of issues during the controlled chaos that is a UCAIug IEC 61850 IOP that the issues can be triaged in a participant neutral manner so that:
- Vendors can use the IOP to find problems that would normally not be stageable in a test lab or would be extremely difficult to diagnose if deployed to the field. It is not a requirement that devices, or applications, participating in the IOP pass a conformance test or have been released to the market
- Witnesses (e.g. utilities and consultants) can help develop test procedures that reflect their concerns and observe and learn about IEC 61850 in a very compressed and multi-vendor environment. The global participation of vendors also allows witnesses to gain experience with devices and applications with which they may not be familiar. As a side note, the IEC 61850 IOPs represent a low-cost mechanism to get hands-on experience with IEC 61850 in an environment where vendors, experts, and test labs are all present
- Problems with standards can be noted, triaged, have developed suggested standard changes. The problem reports, with suggested changes, are then submitted to the relevant standards organizations for correction
Preparation for IOPs typically starts at least one year in advance. This allows for the group of the willing (e.g. IOP participants and vendors) to determine the scope and focus of the IOP. The 2019 IEC 61850 IOP represented the most aggressive testing campaign in the series of UCAIug sponsored activities dating back to 2011. The previous IOPs in 2011, 2013, and 2015 concentrated on interoperability testing one-to-one between individual IEDs and/or applications. In 2017, vendors and utilities requested an “integrated application” test. The emphasis of testing through the lens of an integrated application continued in 2019.
The 2019 IOP decided to have two different, non-related test areas. One area considered engineering, System Configuration Language (SCL) exchanges between System Configuration Tools (SCTs). The other area was a set-up of a typical digital utility, consisting of one control center and one substation with multiple bays, where the different applications were integrated.
The integrated applications allowed the IOP to utilize the top-down SCL engineering process. The Figure on page 18 is the High Level Design of the targeted substation.
Based upon the Single Line Diagram, signal and functional assignments were targeted as shown in Figure 2.
Since the engineering was top-down, the quality of the IED Capability Description (ICD) files was critical. New validation tools were tested as part of the IOP. The IOP’s use of an integrated substation allowed testing of Substation Maintenance sequences, resiliency and cyber-security for the various protocols used for information exchange. The exchange of information that the functions required was achieved through using the following: MMS Client/Server, GOOSE, Sampled Values, Routable-GOOSE (R-GOOSE), and Precision Time Protocol (PTP) in the form of IEC 61850-9-3.
There was a major emphasis on using the integrated application to represent a digital utility utilizing Sampled Value merging units. There were ten (10) merging units split across High Speed Redundancy (HSR) and Parallel Redundancy Protocol (PRP) process bus networks that exchanged information between the PRP and HSR Sampled Values (SV) networks to provide information to protection relays that supported a different technology (e.g. PRP or HSR) from the SV publisher. The Station Bus was PRP. Clock synchronization was provided from Station Bus to Process Bus, PTP Grandmaster, boundary, and ordinary clocks were distributed throughout the network.
The 2019 IOP was held at the Electric Power Research Institute facilities in Charlotte, North Carolina, United States. The actual installation of the network, IEDs, control center applications, and initial configuration and ad-hoc testing of integration took some 16-hours. The overall staging involved twenty-five (25) different vendors from twelve (12) different countries. The integration being perceived as controlled chaos may be an overstatement. However, it was almost completely accomplished in the allotted period of time except for networking issues.
Figure 3 depicts the summary of the detected IOP issues by test category. Most issues were detected during SCL testing/usage and security testing. More summary information on these categories follows as well as interesting findings regarding networking and the relationship of PTP to Sampled Values. The full report is available to UCAIug members.
SCL Engineering Process and Validation
There is an adage that of “Garbage in Garbage Out.” The 2019 IOP attempted to provide high quality IEC 61850 ICD files through testing through a range of five (5) different SCL validation tools. The tools most frequently used for ICD validation were from DNV GL (SCL Checker), Triangle Microworks (SCL Validator), and an EDF/Hydro Quebec developed tool (rise-eclipse). The various tools sometimes did not enunciate the same errors, and thus testing SCL validation with multiple tools is recommended.
Of the one-hundred fifty-nine (159) ICD files put through the IOP ICD validation testing, only ten (10) passed with no errors. Of those, five (5) passed using multiple tools for validation. The best-case interpretation of the results is that only 6.28% of all submitted ICDs passed eventually. This is a true indicator that some action needed to be taken to improve the quality of the ICD files that vendors are providing with their IEDs.
There were fifty-one (51) other reported issues encountered (e.g. the count does not include the massive amount of ICD issues) during the integration, staging, and maintenance of the integrated application as well as the SCL testing area. Of these, thirty-five (35) were triaged to be implementation issues spread equally across SCTs (17) and ICTs (18).
There were at least nine (9) standards related issues also discovered.
Actions Taken: Based upon the ICD quality issues, UCAIug is changing its testing philosophy. Previously, ICT testing was not part of IED conformance testing. The revised testing philosophy, part of Edition 2 Amendment 1 testing, will require SCL and partial ICT functionality to be tested as part of the IED conformance tests. There will still be separate conformance tests for ICTs and SCTs which have also been strengthened based upon the IOP results.
The discovered standard issues have been submitted to the IEC TC57 WG10 IEC 61850 User Feedback Task Force. The submitted issues, at the time of this article, have been resolved.
Cyber Security
IEC 61850 cyber security is defined by a set of IEC 62351 as well as IEC 61850 standards. The security standards that were tested at the IOP were related to Client/Server (IEC 62351-4), GOOSE and Sampled Values (IEC 61850-8-1 and IEC 62351-6), and key management (IEC 62351-9).
There were several changes in the landscape of IEC 61850 cyber security since 2017. First and foremost, IEC/TR 61850-90-5:2012 has been superseded by the combination of IEC 61850-8-1:2011+AMD1 (a.k.a. IEC 61850-8-1 Edition 2.1) and IEC 62351-9:2017 (key management). IEC/TR 61850-90-5 is not compatible with the new standards. Thus, the focus of the IOP was to test interoperability based upon the new standards. Several issues were found with the key management specification (e.g. IEC 62351-9 as well as related RFCs) during testing.
Likewise, IEC 62351-4:2007 was replaced by IEC 62351-4:2018. There was implementation of both versions at the IOP which should have been compatible. This was found to not be the case during testing.
IEC 62351-6:2007 was under revision and the draft of the revision was used as the basis of testing.
The distribution of the captured cyber security issues is shown in Figure 4.
Actions Taken: The problems found with IEC 62351-4:2018 were primarily backward compatibility issues with IEC 62351-4:2007. Although not widely deployed in IEC 61850 systems, the compatibility issues would have also impacted IEC 60870-6 TASE.2 (a.k.a. ICCP). These issues were corrected in IEC 62351-4:2018 + AMD1 which corrects the reported issues.
The problem found with IEC 62351-6:2007 was an ambiguity in how a specific set of encryption and authentication was specified (e.g. AES CBC and AES GCM). This issue has been corrected in IEC 62351-6 in a revision that should reach IS status (e.g. IEC 62351-6:2020) at the time of this article.
Five (5) issues relating to IEC 62351-9:2017 were reported and a consortium of implementors have continued to report an additional twenty-six (26) issues which have also been resolved in the upcoming standard. The result of the IOP is leading to the eminent release of products implementing the aforementioned standards with the issues corrected.
Network Issue Takeaways
The 2019 IOP network consisted of multiple vendors supplying routers, switches, and red boxes. The network was designed to reflect a typical digital utility. The sole exception to this statement was process bus where SV publishers using both PRP and HSR were supported. (Figure 5).
The HLD shows where the PTP Grandmaster Clock (GMC) was located as well as where Single Attached Nodes (SANs), Double Attached Nodes (DANP), and red boxes (RBOX) were used. Additionally, it shows the single switch for the independent SCL testing area (SCL-SWITCH).
While the network backbone was being configured, approximately seventy (70) IEDs and computers were being connected by the IED vendors and in an unsupervised manner. This was a mistake as one of the IED’s PRP implementation turned out to act as a bridge between PRP LAN-A and PRP LAN-B on the station bus. The end result all nodes receiving LAN-A and LAN-B traffic and also create a situation that consumed 100% of the station bus bandwidth thereby creating an unstable station bus. This was a difficult problem to trace down, and a lesson learned for the next IOP.
A similar, although less catastrophic situation, occurred through a misconnection of two red boxes where the output of Redbox-1 was an input to Redbox-2 and through a switch, the output of Redbox-2 became an input to Redbox-1.
The HLD also shows how the HSR and PRP of Process bus where “bridged.” Due to misconfiguration of the “bridging” network devices, SV from both the PRP and HSR process bus networks bled through to the station bus and caused further instability. There were multiple GOOSE and SV monitors on the station bus that detected this issue. Since the network consisted of multiple network device vendors, there were multiple configuration tools in use which slowed the mitigation and diagnosis of the problem.
Actions Taken: The take-away, if one has time, is to attach network devices and IEDs in a progression so that problems can be diagnosed when the devices are connected and not after the network is staged.
Impact of PTP on Sampled Values
The UCAIug IOPs include “disruptive” tests. These tests are intended to cause issues. This type of tests typically takes normal system occurrences and brings them to an extreme.
IEC 61869-9 (e.g. the Edition 2 replacement for the UCAIug 9-2LE profile) specifies how a sample value publisher should adjust time and sampling count when it loses global synchronization and its internal clock drifts away, and how it should adjust sampling when global synchronization is recovered. This applies also when the global clock adjusts its time or when the SV publisher changes the synchronization source. IEC 61869-9 requires an immediate jump to the new time scale. This requirement appears to make sense especially for protection applications that subscribe to Sampled Values from different publishers synchronized by different clocks.
The premise of the disruptive test was to execute a large time adjustment by forcing the grandmaster (the global time source) to jump one hour ahead, and once the integrated application stabilized, the grandmaster returned to actual time (e.g. -1 hour). The integrated application, and its SV publishers in particular, were diagnosed in regard to meeting the IEC 61869-9 “jump” requirement.
The results showed that some publishers misbehaved. The root cause of the incorrect behavior was not as expected (e.g. in the SV publishers) but in the upstream PTP Boundary Clocks that relay the time from the grandmaster. To adjust to the new grandmaster time, Boundary Clocks were skewing at their own rate of change (some quite slowly) but claimed a high accuracy in spite of being strongly degraded. The SV publishers could not address this situation by switching to a better master, since they were not aware of the large offset. AS a result, the subscriber could no longer compare the values.
Actions Taken: The first objective of the IOP was to capture issues that would have been almost impossible to diagnose in the field. The second objective of the IOP was to prompt the relevant standard organizations to correct the deficiency. This involved several of them. Indeed, the root cause is that IEEE 1588 (PTP) Boundary Clocks do not advertise their own clock quality, but that of their grandmaster, even if this one was lost and the Boundary Clock is in holdover (e.g. free-running). Since IEEE 1588-2019 has just been finalized, IEC 61850-9-3 and IEC 62439-3.4 will specify how a slave port shall behave when losing and recovering its synchronization. Figure 6 shows the proposed metrics for the transition of the port states.
In addition, these standards will define when boundary clocks become master to advertise their own clock quality rather than that of the lost grandmaster. Since the IEEE Conformity Assessment Program (ICAP) and National Institute of Standards and Technology (NIST) are responsible for the conformance testing of IEEE 1588, these organizations were approached and have agreed to add an optional test to validate that clocks behave as needed by SV publishers.
The last issue arising from the disruptive test regards IEC 61869-9. It turned out that the requirement of time jumping that makes sense at the first glance does not work as intended, since different SV publishers will not necessarily listen to the same master and they will not change their time scales at the same time. The issue has been submitted to IEC TC38 for further study.
Control Center and Substation Interaction for Maintenance
Although the IOP found issues, the highlight of the integrated application testing was using it to perform differential protection, utilizing Sample Values. Testing isolation was tested (e.g. using IEC 61850 Mode (Mod), Behavior (Beh), and Simulation (Sim)) under the control of the control center SCADA system and Single Line Diagram (SLD). (Figure 7).
The test involved Omicron and Doble test sets assuming the publishing role(s) of one or more of the SV publishers on the process bus. These test sets were placed onto the Process Bus network with the 61869-9/IEC 61850-9-2 Edition 2/2.1 Simulation Bit set. The subscribing IEDs of the SV publishers being simulated appropriately ignored the test equipment publications. A combination of the control center SCADA system and local configuration tooling changed the subscribing IEDs to accept the simulated publications through setting the LPHD.Sim value in the subscribing IEDs. Through varying the values published by the test sets, the SCADA system observed the differential protection trip operated properly. The SCADA system then changed the subscribing IEDs to a state of test/blocked and the quality information provided from the IEDs to the SCADA system changed appropriately. As the test sets varied their values, the differential protection function did not operate since the IEDs were in test/blocked mode.
Looking Forward to 2021-2022
Moving forward from the successes of the 2019 IOP, UCAIug is starting to plan for a 2021 or 2022 IOP. Due to COVD-19 uncertainties, a centralized face-to-face meeting in October 2021 may be risky. Ideas are being evaluated:
- In 2021 – if a full face-to-face is not possible:
- Proceed with a single physical face-to-face IOP site.
- Link region test centers (e.g. Europe, Asian RIM, North America, etc..) and link these together over the Internet using LAN-to-LAN VPNs, Routable GOOSE and Routable Sampled Values, etc.
- Delay the IOP to February 2022 and use a single site.
It would be anticipated that an integrated application will still be utilized and that SCL validation and error checking will still be a major focus.
Other areas that need expansion are cyber security and key management testing. This would include security for Layer 2 GOOSE and Sampled Values as well as Routable GOOSE and Synchrophasor measurements delivered in a secure manner via Routable Sampled Values.
Additionally, the Human Machine Interface (HMI) IEC 61850-6-2 standard would be anticipated.
If you believe that there are areas that need to be tested, please join the effort to develop those cases for the upcoming IOP. Ensuring that your issues are addressed by the IOP requires your participation.
Regardless of the site, planning is beginning as well as evaluation and developing test cases.
Biography:
Herbert Falk is the Vice President of Testing for the UCA International User Group (UCAIug) and is responsible for managing the Conformance and Interoperability Test Programs within UCAIug. He has managed IEC 61850 Interoperability Tests bi-annually since 2011. He is an editor of IEC 61850 and the US Technical Advisory Group (TAG) lead on Cyber Security to IEC TC57 WG15. He is an active member of the IEEE Power Systems Communication Committee (PSCC) , IEEE Power System Relaying Committee (PSRC), and the DNP Security Task Force. Herb graduated with MSEE degree from Northwestern University in 1979.
