By Janos Pattantyus, Denis Francesconi and Jean Raymond, Hydro-Quebec, Canada
In recent years, Hydro-Quebec has been developing a new synchronous machine protection paradigm for its future. Previously, instrumentation, monitoring, mechanical, and electrical protections were separate systems. In the new approach, they have been merged to form a single system linked by a process bus.
Modern large hydro generating stations exist within highly integrated networks of electricity generation, transmission, and distribution facilities together with Distributed Energy Resources (DERs) on customer premises, such as solar-generation, energy-storage, wind-power, and grid-interactive devices (home automation).
Several challenges contribute to the increasing complexity of the electricity network, including:
- Variability and intermittency of renewable energy generation
- Decline in frequency response capacity and system inertia
- Load model modifications and unpredictability
- Faster and unanticipated system dynamics that create emergent risks
- Need to manage a fast-growing number of generation and load variables
- Cyberattacks
- Climate change and power-grid exposure to extreme events
- Market volatility
- Disrupted supply chains for key grid assets
To address these challenges, Hydro-Quebec developed nine automation guiding principles that have been introduced in new developments and paradigms (Figure 2).
Hydro-Quebec developed its new instrumentation approach and design requirements for its future generating stations and hydroelectric developments based on these new guiding principles. This article describes the results of our proof-of-concept activities supported by an availability study comparing the traditional instrumentation architecture with a new paradigm.
Sensors and IEDs
To control, protect and monitor its fleet of hydro turbine-generator units, Hydro-Québec installs a number of sensors (Figure 1).
A subset of all the sensors, referred to as instrumentation sensors, measures parameters related to the mechanical state of each turbine-generator unit. These measurements typically include vibration, bearing temperature, oil level, flow, pressure, cooling water temperature, hydraulic pressure, stator temperature, and various contacts.
Vibration sensors are generally eddy current relative displacement types. They require specialized electronic devices to power and read the signal conditioner, and to evaluate the vibration signal for amplitude and validity.
Temperature sensors are typically three-wire RTD types. These also require specific electronic interfaces for magnitude and validity evaluation. This type of measurement is common in the industry, with many automation devices available on the market.
For level, pressure and flow measurements, 4-20 mA HART (Highway Addressable Remote Transducer) transmitters are generally used. These transmitters require appropriate mA loop electronic interfaces to power them and evaluate the return values. The measured values, along with diagnostic information, are accessible via the HART communication protocol, which operates over the mA signal.
The transmitters are also configured through the HART communication interface. Some sensors function as switches, such as shear pin monitors or level switches. These are read as digital inputs by the electronic interface that processes them.
Protection Process Bus and Broadcast Protocol
Figure 3 illustrates the alternator mechanical protection process bus architecture based on the IEC 61850 GOOSE protocol. To ensure redundancy, two Ethernet-based process buses are utilized. Certain sensors are installed in duplicate to supply data to each bus, while other sensors are distributed across the networks in a functionally redundant manner. Two alternator protection relays operate in a complementary manner, subscribing to digitized measurements transmitted via the two process buses. Their primary function is to evaluate these received values against trip settings and initiate shutdown of the generating unit when necessary.
The control system receives data that enables it to manage the generating unit effectively. It can generate alarms, evaluate parameters for startup, and handle normal startup and shutdown sequences. Other tasks include generating setpoints for speed and excitation systems.
The scheme is coherent with the model proposed by the “Technical Guidelines for Smart Hydroelectric Power Plant” working draft within IEC CD 63198-2775/Ed1.
Laboratory Testing and Lessons Learned
Initial testing involved proof-of-concept type testing of components within a local area network configuration. In this case, one of each type of measurement was implemented. Since these measurements were deemed successful, it was decided to build a complete replica of the system needed for a complete turbine-generator unit (Figure 4).
We will discuss some of the challenges faced and overcome by the team throughout the testing phases.
Evolution of Protocol Configuration and Use: As the exact needs and constraints of the system became more apparent, the way we use the GOOSE protocol and ICD (IED Capability Description) configuration files evolved as well. For example, one subscriber could only accept a dataset with a maximum of 60 elements. If we break down the elements per channel into one magnitude and one quality, it became clear that keeping a timestamp for each channel value was not helpful, as the timestamp was the same for all the data. The timestamp also increased the data size of the GOOSE packet. It was then decided to drop the timestamp per channel. The packet itself is still timestamped at creation.
We also identified the need for the acquisition IEDs to be able to do some data averaging, as some quantities like oil levels and temperatures could be affected by mechanical sloshing or electrical noise respectively. In addition, we also needed to identify which 4-20 mA sensors were HART enabled to take advantage of the HART protocol for retrieving the full scale and error diagnostics. Both these situations could be addressed in the ICD files generated as Data Attributes of the acquisition channels.
Further testing on a newer CPU (Central Processing Unit), discussed later in this paper, revealed that the GOOSE library had to be improved under conditions of behavior after bootup for a subscriber to time out if no data was received and stop data generation on the publishing side if the PLC (Programmable Logic Controller) failed. It was also noted that the subscriber module did a great deal of writing to an output file, which could be detrimental to the PLC and increase its chances of failure.
Vendor Obsolescence and Other Issues: As the testing period extended over a period of four years, changes occurred at the vendor level. One selected and approved PLC CPU module was discontinued. This required a quick replacement with a more flexible higher-performance module. However, the IEC 61850 protocol was less mature, and the module was also more complex to program. For one other vendor, we found the temperature measurement lacked resolution (0.5-degree steps), and they could not support the hardware quality testing that we required. Therefore, we could not continue using their products.
One of the challenges in terms of supporting the solutions that are developed is to manage any hardware and firmware changes that occur. These changes seem to be happening relatively often in response to component availability on the market, among other reasons. Our approach to this problem is to maintain close synergy with the vendors and keep up to date on any changes so we can analyze them. The risk is that if there is a miss on the vendor, side and a change is not well documented, we will experience inexplicable failures. This was the case for a firmware fix that was not well communicated, as well as a bug where a programming library and hardware mismatch were allowed to occur.
We experienced another major product discontinuation when a main CPU component was taken off the market. Since we were too far into the engineering phase to switch these out, and no viable alternative was ready, we opted to deploy these units temporarily and work on their replacement. An excellent vendor relationship is key to overcoming any adversity that will inevitably disrupt the project.
Networking Challenges: One of the main lessons learned is the importance of being strictly in control of the network and data flows in such an architecture. As all IEDs exist on a network and have network cards, they can be susceptible to overload and other disruptions that could affect the interface and the Linux operating system beyond. We discovered that some units produced a substantial quantity of UDP (User Datagram Protocol) and some hybrid HSR packets broadcast data to communicate amongst themselves. To prevent these packets from propagating beyond the first switch inputs, we managed to create Ethertype filters by destination MAC (Media Access Control) address as well as broadcast filters for the UDP packets by destination IP address. This also prevented a case in which some modules would come into possible configuration conflicts when installed on other machines.
We are also implementing several precautions, on modules that permit them. These precautions include load limiting on the IED network cards and on the switch ports. All GOOSE packets will be routed by the switch according to a destination MAC address scheme that directs the flows to the destination ports alone. This is in addition to the configured VLANs (Virtual Local Area Networks). Furthermore, when possible, all unused services and ports on the IEDs are disabled and closed. The built-in firewall function was used to limit communication to web page access only. With a strictly defined network architecture, everything is in place for dependable operation.
Availability Study
It was decided that to establish the viability of the new solution and identify any weak links, an availability study should be conducted to compare the current architecture with the new architecture for turbine-generator unit instrumentation. Figure 5 compares the two architectures.
The common points excluded from the study are all sensors, protections, power supplies and their components. For both cases, the temperature and vibration sensors are path redundant, but the oil pressure sensor’s paths are not. The current architecture uses the same vendor 1 for the path’s redundancy, and only copper cable wiring is used.
The new architecture uses three different vendors (vendors A and B for path redundancy), RJ45 and fibre optic cables. Table 1 presents the results obtained for both architectures subject to a 96-hour Mean Time To Repair.
Canonical Data Model and the Engineering Process for Instrumentation
This new paradigm also integrates new ways of managing information and engineering. The broadcasted information is linked to a canonical data model that centralizes information, and automatic documentation generation and IED templates for subscribing IEDs. The engineering processes were reviewed regularly and modified to ensure that this new paradigm would integrate seamlessly into new and existing installations.
The approach of using a Canonical Data Model (CDM), labelled as A in Figure 6, provides the company with a standardized method to represent data nomenclature to various systems through an API (Application Programming Interface).
Hydro-Quebec has an existing centralized IED configurator system, labelled as C in Figure6, for managing protection IEDs in generating stations and transmission substations. Its management function is being extended to manage the process bus IEDs (protective relays, local control units, ethernet switches, speed regulator systems, and excitation systems). This system now can communicate with the CDM storage through an API.
The centralized IED configurator accesses the CDM and extracts typed IEDs via a file formatted according to the Substation Configuration Language (SCL) as defined in the IEC 61850-6 standard. The centralized IED configurator front end client application, labelled as the B in Figure 6, provides an entry form to ensure the data is properly inputted from the listed information extracted from the SCL file. Once the entries are complete according to strict rules, the user, labelled as D in Figure 6, may compile the information in the desired format, compliant to a vendor IED configuration.
The CDM makes extensive use of the data model provided by IEC 61850. Hydro-Quebec extended the IEC 61850 data model and tweaked it to ensure a seamless interface with Hydro-Quebec’s existing point nomenclature. This generated Hydro-Quebec’s own namespace by removing IEC 61850 data attributes or data objects that did not add efficiency to the engineering process. IEC 61850 common data classes were extended to host extra data attributes that supply the parameters required in logic control programming for calculation purposes or communication addresses.
This approach helped to significantly reduce the information being published by real-time communication protocols such as GOOSE, Modbus, MMS (Manufacturing Message Specification), DNP3 (Distributed Network Protocol 3) and OPC (Open Platform Communication). The use of private parts as defined in IEC 61850 was avoided. This work is the basis for Hydro-Quebec’s new information ontology, as seen in Figure 7, from which its CDM takes form.
For instance, Hydro-Quebec extended IEC 61850 Measured Value (MV) common data classes by adding configuration (CF) data attributes with functional Constraint (FC). These data attributes are used as parameters within the logic controller. In one example, Hydro-Quebec added an avgTms data attribute which is the cycle period for averaging a process value (Table 2).
The benefits of a centralized approach to managing the configuration information ensures that all the information is available for inter-IED communication thus reducing the time spent in managing the engineering information to be exchanged within a project between system stakeholders, as shown in Figure 8.
In summary, this new paradigm and the associated work has yielded several tangible benefits.
- Our new machinery protection scheme enables us to greatly reduce the amount of cabling needed. In addition, the numerous lessons learned have prevented many problems that could have occurred later in the deployment stages
- From the simulations conducted in the availability study, we can conclude that the new architecture offers much better performance
- By applying the canonical data model and engineering process, we can see the gains in centralizing information and transmitting it back to the field IEDs
.
Biographies:
Janos Pattantyus has worked for more than 28 years, as rotating equipment specialist in machinery protection, vibration analysis and predictive maintenance. He holds a bachelor’s degree in mechanical engineering from McGill and a master’s degree from École Polytechnique. He is certified category 4 in vibration analysis and a licensed engineer in the province on Quebec. He has more than half a dozen publications. Janos is an active member and leader of the international working group IEC 61850 Dynamics Modelling Joint Task Force. Janos works currently in the instrumentation and mechanical protection systems team within the Hydro-Quebec automation expertise department.
Denis Francesconi is a system matter expert in control systems for Hydro Power Plants with over 30 years cumulated in the Hydro domain of which 24 years with Hydro-Québec. Denis graduated in 1987 at Montréal’s Concordia University with a bachelor’s degree in electrical engineering and is a licensed engineer in the province of Quebec. He participated in the IEC Technical Committee 57 in extending IEC 61850 to include Logical Nodes required for the Hydroelectric domain.
Jean Raymond has worked for more than 34 years for Hydro-Quebec, as a telecom network engineer and as an engineer responsible for the evolution and the long-term development of its transmission and electricity power systems. Jean holds a bachelor’s degree in electrical engineering from Universite Laval, and a Master and a Ph.D. degree in Electrical Engineering from Universite Laval, in collaboration with the Defense Research Establishments. He has more than twenty publications and a future book to be published. Jean is an active member in international WGs (IEC, IEEE), and convener of IEC TC57 AG22. He is a member of IEEE PES Magazine Board and reviews books for Wiley&Sons. He is involved in the overhaul of Electrical Engineering and Asset Management programs in universities.
