Lessons Learned

Towards an Industrial Deployment of Fully Digital PACS at RTE

By Volker Leitloff, Olivier Lopez, Jean-Marie Boisset, Maud Merley, Xavier  Michaut, 

Jean-Etienne Lemaire, and Thibaut Ratel, RTE, France

Why to go for fully digital PACS?

In the past two decades, the policy of Rte has been to purchase “turn-key” integrated Protection Automation and Control Systems (PACS) from several vendors, based on a set of functional specifications. Rte has now launched the R#SPACE project aiming at an industrial rollout of fully digital, IEC 61850 based PACS implementing components from multiple vendors.  This project is motivated by several constraints and evolutions experienced by Rte over the last years, including: 

  Increased insertion of Renewable Energy Resources (RES)

  Limited flexibility regarding post-commissioning evolutions of turn-key systems

  Delays and high development costs in case of evolution of substation level functions

  Company policy of extension of remote maintenance and remote administration of PACS

  Capacity to interface Low Power Instrument Transformers (LPIT) and other communicating HV equipment and monitoring sensors via IEC 61850 process bus.

In the definition phase of the R#SPACE project, it became clear that there are components which should be purchased and others which should be developed internally. Most of the internally developed components will be hosted by a virtualized server structure.  It also became clear that every purchased component could (and probably should) be purchased independently. This requires to define specific purchase strategies for each component. The consequence of this is also that R#SPACE is a multi-vendor and multi-component system by construction. 

This means that, for R#SPACE, the position of Rte is that of a system integrator charged with the definition of the interoperability framework of the new PACS, including the configuration, data models, administration and cybersecurity.

Learning from Demonstrators and Pilot Projects

Prior to launching the R#SPACE project, fully digital PACS featuring IEC 61850 based process bus and station bus were tested in two demonstrator substations in the framework of the “Postes Intelligents” project (see PAC World Magazine June 2018). The experience feedback gained in this project enabled and influenced the design and specification of the R#SPACE PACS. The learnings included:

  • Operational experience with a functional protection chain including Stand Alone Merging Units (SAMU), the IED hosting the protection function and Switchgear Control Units (SCU) used for tripping. On several feeders, LPIT have been used. One of the identified issues, addressed in the R#SPACE specification, concerns the coordination between subscribing protection functions and publishing Merging Units under degraded synchronization or acquisition conditions
  • Qualification, FAT and SAT testing. This experience has contributed to the design of the R#SPACE test platforms and the overall definition of the test methodology. It is based on the use of the test features provided by IEC 61850 ed2 (see also CIGRE TB 760)
  • Design, maintenance and operation constraints of field cubicles containing process interface IED. This type of cubicles may be used in a later phase of the R#SPACE project
  • In addition to IEC 61850 based communication, a full functional interoperability between different equipment and functions also requires coordination in profiles and syntaxes. This can be achieved by using relevant product standards and profiles, such as IEC 61869-9 for the digital interface of the Merging Units, IEC 61850 based data models for the different PACS functions and the use of Basic Application Profiles (BAP). Interoperability on application level is also required. It can be based on functional standards such as the IEC 60255 series for protection function and on user specifications (see Figure 1).

Description of R#SPACE PACS

The R#SPACE system consists of several components which can be grouped in 3 main blocks (see Figure 2):

1.  The substation level is implemented in a virtualized infrastructure providing redundancy and hosting the virtual machines which run the substation level applications. These applications include the HMI for operating and maintaining the system, the configuration tools, the substation level automatons, the gateways and the cybersecurity functions

2. The Local Area Network (LAN) and the time synchronization architecture

3.The bay and process level consisting of:

  • IED hosting bay level functions and protection functions: BCU (Bay Control Unit) and other IED hosting protection functions
  • Process Interface Units (PIU) for data acquisition from the process interface: SCU  and SAMU. Some of the PIUs are redundant

Differentiated approaches with specific industrial and purchase strategies can be adopted for each of the components. While some components or functions are developed internally, others are purchased or developed in collaboration with third parties using an open-source approach. The corresponding external and internal developments have been launched in 2020. Pre-integration and FAT have started in 2022 and the commissioning of the first substation, preceding an industrial rollout, is planned for end 2023.

R#SPACE Architecture

The R#SPACE architecture is shown in Figure 3. Based on the use of a duplicated global LAN with Parallel Redundancy Protocol (PRP), it is composed of: 

  • 2 central switches for the general substation functions and the common substation bays (Busbar bay and General substation services bay) 
  • 2 switches per bay which are connected to central switches in a star topology These switches can be shared between several bays

In addition to this, if a permanently implemented test device is used, this device will be connected to a specific switch that aggregates the dataflow of the central switch and of each bay switch of one of the PRP networks. To secure an efficient process, segregation policy with VLAN was adopted rather than MAC filtering. MAC filtering was evaluated to be more complex to configure and to operate. Given the R#SPACE flow matrix, the following principles are applied:

  • Each IEC 61850 multicast flow (SV and GOOSE) between IEDs within a bay is carried by one VLAN, the range of which is limiited to the said bay
  • VLAN are organized as follows:
    • 1 VLAN for SV of each bay (intra-bay) 
    • 1 VLAN for GOOSE of each bay (intra-bay)
    • 1 VLAN for SV subscribed by protections and automatons requiring SV form other bays (e.g. busbar differential) (inter-bay) 
    • 1 VLAN for GOOSE subscribed by protections and automatons from other bays (inter-bay)
    • 1 VLAN for PTP synchronization
    • 1 VLAN for MMS between IEDs and substation level functions
  • In order to reduce port consumption, the VLANs from or towards a switch are trunked on a unique physical link

Quality of Service (QoS) policy ensures the non-disturbance of process flows in case of link congestions. Class of services (CoS) is used in R#SPACE PACS. Each class regroups applications flows with the same level of requirement and quality of services. VLAN ID and priority bit (from 0 to 7) is also set by the IEDs.  The information of VLAN or priority bit are described in the SCD file. The R#SPACE time synchronization system includes a synchronization server which will broadcast PTP signal through the PRP LAN. In addition to the IEC 61850 PRP LAN, a physical LAN is dedicated to transport the administration flows of the IEDs.

Virtualization of Substation Level Functions

PACS are experiencing a basic trend towards scalability, reduction of the number of devices, interoperability, industrial performance and reduction of operating and investment costs. One of the means to reach this goal is virtualization.

The implementation of virtualization in PACS can be realized in several steps with the long-term goal to virtualize all PACS functions on a standardized platform. 

As the first step, R#SPACE focuses on the virtualization of the general substation functions, which include local HMI (operation and administration), storage servers, telecontrol and supervision gateways, cybersecurity servers and station-level automatons with long response time (in general >100 ms). R#SPACE virtualization infrastructure connected to the R#SPACE LAN and the administration LAN is shown below in Figure 4.

Interoperability Framework 

As mentioned above, the interoperability between the different components is a major aim for the project. 

This concerns both the “run-time” interoperability, based on common data models, harmonized profiles, templates and Basic Application Profiles (BAP) and the interoperability of the configuration, administration and supervisions chains. In addition to the interoperability levels described previously, the R#SPACE framework also includes the following aspects, referred to as “non-operational interoperability” related to configuration, setting, supervision and administration of the different components:

  • Top-down configuration and setting on the base of a unique IEC 61850 Substation Configuration Language (SCL) file
  • Administration based on the proprietary ICT of the different IEDs, but using an API defined for R#SPACE. This provides a standardized interface and standardized administration procedures for all IEDs and enables automatic configuration upload for all IEDs and functions
  • Cybersecurity requirements coordinated between the different components of R#SPACE PACS
  • Supervision of IED based on IEC 61850 supervision Logical Nodes
  • Interoperability features implemented in R#SPACE include:

Data Published by PACS Functions: PACS functions in R#SPACE are modelled as IEC 61850 Logical Devices (LD) for all data published to other functions or interfaces (https://www.rte-france.com/fournisseurs/network-together#LeprojetRSPACE). In addition, the functional specification needs to explicitly refer to the DO and Data Attributes (DA) of the IEC 61850 model for consistent and interoperable implementation of the different PACS functions. This constitutes de facto an implementation profile. Examples of this include the publication and subscription requirements for the trip orders (PTRC.Tr) and the operational status of the circuit breaker (XCBR.CBOpCap).

The detailQual attribute defined in IEC 61850-7-2 ed 2.1 contains choices including out of range, bad reference, oscillatory, etc. This may cause the quality attribute to be “questionable” or “invalid.” On this base, a profile for Instrument Transformers with Digital outputs has been defined by IEC TC 38 (IEC 61869-9). Similar efforts have been launched by IEC TC 95 for the IEC 60255 standards covering protection functions.

Data Subscribed by PACS Functions: In order to use all quality handling features provided by the IEC 61850 standard in an efficient way, clarifications of the expected behavior of PACS functions regarding the quality attributes of incoming data is essential. Else, the behavior of functions in multi-vendor system may be inconsistent in degraded conditions, leading to a deterioration of functional interoperability which may be unacceptable for the user in some cases.

A formalized approach to define functional quality interpretation and the related expected behavior for each input of a function allows to specify the behavior of the function for each raised detailed quality bit of each input signal. Some of the detailed quality items do only apply to analog data and others only to binary data. 

Since the function cannot control the consistency of the subscribed data, the behavior should be specified for all. If the subscribed DO cannot be used but the function can continue to operate under degraded conditions, a default value for this input needs to be defined. This profiling approach has been applied in the R#SPACE project.

Phasors Representing Line and Busbar Voltage: The line and busbar voltages are used to verify if recloser conditions are met. This requires either one side to be energized and the other to be de-energized or, if both sides are energized, that the amplitude, phase and frequency differences of the voltages are below their corresponding settable thresholds. This can be easily verified by comparing the phasors representing both voltages. Compared to a SV stream, the use of phasors in GOOSE or report streams uses considerably less communication bandwidth and also facilitates the change of busbar voltage source depending on the position of busbar switches and breakers.

These phasors are modelled by feeder LDPHAS/MMXU0.PhV and busbar LDPHAS/MMXUbts.PhV for the line and busbar side, respectively. “bts” represents de busbar section on which the phasors is measured. The frequency of the line and busbar voltage signals is modelled by the DO “Hz” of the same Logical Nodes. In this case, the application has several kinds of degraded operation depending on the state of the subscribed phasor:

  • Invalid DO “PhV”:  No recloser operation is possible
  • Invalid DA representing angles or frequencies:  This condition still allows dead-line-life-busbar or live-line-dead-busbar recloser operation

For this reason, these conditions need to be verified by the recloser function. It is also worthwhile to note that for low voltages, when the line or busbar is not energized, the calculation of phase or frequency are not possible. These values need thus to be forced to “0” in this condition. This requires a detailed functional specification based on the subscribed data. 

Settings: Protection settings are described by Data Objects in IEC 61850. In particular, the data classes ING (integer values) for timers and ASG (analogue values) for thresholds are used. If these parameters are intended to be changed in runtime using services defined by IEC 61850, they need to be exposed via the IEC 61850 communication stack and be accessible via MMS services, as any other Data Object of the corresponding functions. In addition to the lack of some setting parameters in the available LN, a hindrance for this approach is that, for cyber security reasons, users may not want settings to be modified by IEC 61850 services. Alternatively, settings can be processed, after a conversion to manufacturer specific parameters, by the setting tool associated to the IED and downloaded using the proprietary mechanism of the manufacturer. This approach has been implemented for R#SPACE.

In any case, the use of IEC 61850 based settings to describe the user defined settings of a function enables the user to define manufacturer independent basic settings for these functions. This was done in the IEC 61850 model of Rte’s PACS functions. Although some shortcomings have been found, it has been shown, for the protection and automation functions used by Rte, that most of the settings required by the users can be defined based on available IEC 61850 Data Objects. The manufacturers usually employ a significant higher number of settings and parameters. The relation between these three sets of parameters is illustrated in Figure 5.

Figure 5 also shows that the volume of manufacturer specific settings is in general significantly greater than the setting parameters used by utilities or defined in IEC 61850. The latter two also have a big common section. Whereas it is not realistic to standardize manufacturer specific settings because of different approaches used by different vendors, it can be envisaged to standardize setting parameters used by utilities. An important part of them is already covered by parameters associated to IEC 61850 defined Logical Nodes.

Figure 6 illustrates the operational and non-operational interoperability features discussed above for the example of a functional protection chain. This functional chain is composed of three IEDs:

  • A SAMU hosting a logical device LDTM publishing a SV stream representing voltages and currents connected to its input terminals by wires from the secondary terminals of the corresponding Instrument Transformers
  • An IED hosting a distance protection function represented by the logical device LDPX
  • A SCU hosting the circuit breaker interface, represented by the Logical Device LDDJ

Figure 6 highlights the different features required to achieve operational interoperability. In particular, the use of profiles, subscribed data quality management and data models needs to be coordinated between the components of the functional chain implemented in the different IEDs. In Figure 6, the functions are represented as Logical Devices. For each function, the inputs need to be associated either to an input terminal or to a DO subscribed from another LD. 

Figure 6 also indicates different features belonging to non-operational interoperability mentioned above. For a PACS composed of IEDs form different manufacturers, such as R#SPACE, an industrial scale deployment requires interfaces for those features complying with a common specification.

Features Related to Industrial Deployment

An industrial deployment of PACS projects with a volume of several dozen systems per year poses additional constraints compared to the design and deployment of “unitary” demonstrator PACS. These constraints include:

  • The configuration process must be streamlined and efficient. This means that the engineering departments should only have to enter once key characteristics of the substation and the associated control and protection functions. This should then generate, with a minimal effort, the documentation and the configuration files both on system level (SCD) and on for the different IEDs composing the system. This can be achieved by an automated suite of tools, the use of templates and the versioning of available components and functions. “Automated” in this context means the use of scripts and of consistency checks. The efficiency of the engineering and configuration phase is a major stake for the global life cycle cost of the R#SPACE PACS
  • The different configuration and setting files must be available in a central repository. From this repository, they must be easily deployable to the contractors in charge of the assembly and FAT and to the different substation sites in case of updates during or after commissioning. This requires the creation and management of this process by the utility. It can use infrastructures shared with other applications and services
  • Implementation of centralized remote supervision and monitoring of the PACS. This includes the supervision of the PACS communication architecture, the functions implemented in the virtualized infrastructure, the IED and the other components of the PACS. This can also be extended to support the remote monitoring of HV equipment via sensors interfaced with the PIU and front-end functions implemented in the PACS
  • An efficient testing policy has to be defined, especially for multi-vendors systems, in which the combinatory of system versions can be high. Automatic testing based on SCD and setting files, with capabilities of IED simulation or monitoring, is a way to facilitate the industrial deployment of this type of PACS. The capacity of hardware to implement several configuration templates could also help to perform an efficient deployment

Conclusion: This article describes the general approach implemented for the R#SPACE project. Numerous features and specifications are based on the experience feedback gained from the “Postes Intelligents” fully digital PACS demonstrator project. This includes the definition of an interoperability framework based on IEC 61850. The interoperability aspects are relevant for a correct operational and non-operational interoperability of the different components of the R#SPACE PACS and its large scale industrial deployment. These subjects are likely to gain importance in the coming years, with the increase of the number of multi-vendor IEC 61850 based PACS. 

Functional specifications and profiles should be coordinated between users, manufacturers and standardization bodies. Efforts in this direction are undertaken by IEC TC57 WG10 with a Task Force for user feedback, by IEC TC38 which has standardized the digital interface for Instrument Transformers, and by IEC TC95 WG2 for digitally interfaced protection functions. Ongoing efforts to find homogenous and standardized ways to handle the different aspects also include several CIGRE B5 working groups and open-source approaches with contributing stakeholders from utilities, IED manufacturers and IT industry.

Biographies:

Volker Leitloff studied Electrical Engineering at the university of Stuttgart (Germany) and received the Dr. INPG degree from the Institut National Polytechnique de Grenoble (INPG, France). At the R&D Division of Electricité de France, he worked successively on network protection and on transformers and network technology.  Since 2003, he works at RTE on protection of transmission networks and substation control. He is chair of IEC TC38 and CENELEC TC38 (Instrument Transformers), convenor of IEC TC95 WG2 (Digital I/O of protection IED,) and actual or past member of several IEC and CIGRE working groups.

Olivier Lopez – Master’s Degree in Electrical Engineering, has a 20-year career in Analog and Digital Control Systems in the Energy field. In 2003, he started as an Application Software Engineer in the Oil& Gas industry, followed by several technical positions in the development and deployment of Protection & Control Systems in the Nuclear Industry. In 2019, he joined RTE, the French TSO, and is today, the Project Manager of the R#SPACE Project, RTE’s new generation IEC 61850-based PACS.

Jean-Marie Boisset  – (Master’s Degree in Automation) has a more than 30 year’s career in development of digital control systems for electrical utilities. In 1993 he joined GEC ALSTHOM as R&D embedded software engineer. He then takes several positions as technical development leader for the SPACE 2000 system, project manager for PACiS / DS Agile system, and R&D manager. He joined RTE, the French TSO, in 2010 as Digital Control Systems team leader in the National Centre of Expertise. He also now takes position of technical deputy manager and senior expert. His strong skills based on various technologies let him to define a vision of the substation control system.

Maud Merley joined RTE, the French TSO in 2008 after her master’s degree, to work on communication network and Protection Automation and Control System. In the last five years, she participated as expert and test engineer in the development of a full multi-suppliers IEC 6180 substation inside the R#SPACE project. She is also involved in CIGRE study committee B5 and in standardization, as member of IEC TC57 WG10 and the French national mirror committee.

Xavier Michaut (Engineering Degree in telecommunication systems) has a more than 15 years’ career in communication systems. He worked for Alcatel and Thales on radio transmission systems before joining RTE in 2006. He contributed to several RTE’s major telecommunication projects (Security Optical Network, Unified Telecommunication Infrastructure, …) until he joined the R#SPACE project in 2019 to design the architecture and communication network of RTE’s new generation IEC 61850-based PACs.

Jean-Etienne Lemaire is working since more than 6 years as a software and data engineer for the configuration of substation of the RTE RSPACE project; this project aims at developing next generation of substation with interoperability of IED of various vendors. His goal is to setup a fully automatized configuration chain starting from substation engineer needs to configuration files. In the past he has built a single line diagram referential on GIS software for RTE substation to enable system operator to have access to many data linked to line and substation component.

You may also like