Evolution of intra and inter-substation communication

It is all about packets!

by Galina Antonova, Canada, Mathias Kranich, Switzerland, and Sergiu Paduraru, Sweden, Hitachi Energy

Communication is a fast-developing field that influences various industries, including the power industry. Communication enhancements affect both intra and inter-substation communication. The main direction for both intra and inter-substation communication is the use of packet-switched technologies. This article discusses the fundamental differences between switched, dedicated and multiplexed channels and summarizes modern communication technologies being deployed for intra and inter-substation communication. It also highlights the key considerations for achieving desired performance levels from packet-switched communication networks.

Introduction: The first substations in the power system were mere extensions of the power (generating) stations where all the data monitoring and reporting was done visually and manually. The commands were done by human proxy or by mirrored command switchboards (the switchboard in the substation was mirroring the one in the control room), much like ship machinery was controlled at the time.

With the development of the power systems, the substations grew in size and further apart.  The need for transformer substations and switchyard substations far exceeded the need for generator stations, hence, manual monitoring and control had to be replaced by remote monitoring and control systems. SCADA and RTUs were first introduced in the 1930s and – as a result – the problem of reliable communication first arised. In parallel, developments in protection relaying required information to be exchanged between relays at the two ends of the line – teleprotection was taking shape and its first physical form was the pilot wire. Low latency was now a requirement, as well.

These initial years of power system communication were marked by successes and failures, wild implementations and completely proprietary communication protocols. But fortunately, voice and data communication had a half-century lead over power systems engineering, so many of the standards used in substation communication had their grounds in the general communication standards.

As electric utilities are always under the spell of reliability, low cost and regulatory compliance, these were the factors that would shape substation communication development, as well. All of this while the volume of data to be handled has grown exponentially.

Development Story – Part I – Multiplexing: Three main communication channel types exist: dedicated, multiplexed and switched. The simplest communication channel type is dedicated. Dedicated channels have high availability and reliability, and don’t provide any data switching/forwarding. These channels support data transport only and fully utilize all communication resources for a single connection. While reliable and available, these do not utilize communication resources efficiently. Point-to-point connections such as copper pilot wire, dark fiber, optical ground wire (OPGW) and RS232 communications fall into the dedicated channel category.

To improve the utilization of communication resources multiplexing was invented. Multiplexing is based on resource sharing so that the same communication resources carry multiple communication channels (Figure 1).  The main multiplexing methods are:

• Time Division Multiplexing (TDM), where time slot is assigned to a given channel, as, for example, in Synchronous Digital Hierarchy (SDH), Synchronous Optical Network (SONET), channel banks (T1 multiplexers), IEEE C37.94 framing, etc.
• Frequency Division Multiplexing (FDM), where a frequency is assigned to a given channel, as in the case of traditional Telegraph, Wi-Fi technology and Power Line Carriers (PLC)
• Wavelength Division Multiplexing (WDM), where a wavelength of light is assigned to a given channel. WDM is specific for optical fibers and would require different light-emitting sources

Multiplexed channel, being essentially a set of dedicated channels using the same physical resource, inherits characteristics of dedicated channels such as high availability and reliability. It also provides a more efficient use of communication resources, as a single cable/media supports multiple channels separated by dedicated time slots, frequencies or wavelengths.

Development Story – Part II- Switching: The switched channel is fundamentally different from the first two (dedicated and multiplexed), as its operation and performance depend on the availability of resources (hence its performance is non-deterministic without special provisions). The principle of switched communication is shown in Figure 2.

These require sufficient and well-organized internal data buffering, well-designed Address Resolution Tables for data forwarding decisions, and availability of incoming and outgoing channels (also called ingress and egress).

Examples of switched channels are many and more and more communication technologies utilize this method, as it provides arguably better use of communication resources. These include old traditional Public Telephone Network (Called Public Branch eXchange, or PBX) equipment stations, where channel availability was calculated using Markov chains (generally speaking only 80% of the connected numbers can in fact make connections in a given time), Asynchronous Transfer Mode (ATM) switches, Internet Protocol (IP) routers, and of course Ethernet switches or bridges (per the proper term defined by IEEE).

The term packet-switched network  has been generally used for digital technologies with switched channels. Both IP routing and Layer 2 Ethernet switching belong to this category.

Development Story – Part III – Packet Switching:  Packet switching means the data is divided into packets. Each packet is given a header containing information about the destination. Each packet is forwarded separately over the network via the most optimum route to the destination using the header information. At the destination the data packets are reassembled.

Packet switching is used to optimize the use of the channel capacity available in digital telecommunication networks, minimize the transmission latency (the time it takes for data to pass across the network), and increase the robustness of communication.

There are basically two modes of packet switching: “connectionless” switching and a “connection-oriented” switching.

Examples of connectionless protocols are Ethernet, “pure” Internet Protocol (IP), and the User Datagram Protocol (UDP). Connection-oriented protocols include X.25, Frame Relay, Multiprotocol Label Switching (MPLS), and the Transmission Control Protocol (TCP).

Use of Packets Inside Substations

Horizontal communication inside a substation was not “a thing” until much later. Of course, there were the RTUs, which would take the information from measuring transformers, relays and sensors and send it to the hierarchically superior system, but the need for intra-substation communication was not there yet. Several factors contributed to the emergence of the substation communication and Substation Automation Systems (SAS):

• Substations became larger and more complex, so a full-fledged control center was needed inside the substation; faster command and response times were required (than with network control)
• Complex substations required control functionalities that would ensure reliability in operation – so interlocking and reservation were required functions, station-wide
• Protection functions (like busbar protection and breaker failure protection) needed to send info (like BBP TRIP and BRF signals) across the substation
• Maybe more importantly, the emergence of microprocessor-based relays, with enough processing power to take care of protection, control and communication created the opportunity for the SAS

Initially, the protocols for (horizontal) communication inside a substation were popping up like mushrooms after the rain (one utility manager counted up to 100). One could find in this mix re-purposed internal device bus protocols (like CAN-bus), industrial programmable logic controller (PLC) protocols (like Modbus) or ones originally created for control in buildings (like LON). At the same time, vertical integration protocols got standardized, so the likes of DNP3 and the IEC 60870-5 series dominated the market.

It took a long time for the engineering community and the standardization bodies to come with a universally accepted, all-covering true substation automation standard and a lot of factors had to converge for that to happen: hardware technology advancements (with 100BASE and 1000BASE Ethernet), language made for structured, conceptual representation (XML) and precise timing (with IRIG-B and GPS). When it finally got published, IEC 61850 was ready to mirror all technological advancements in networking and computing, including packet-switched communication.

Packet-switched communication utilized for digital substations is built on IEC 61850 technology whereas communication data streams are used to exchange binary signals (commands and status) and analog data (digital samples of currents and voltages). The former uses Generic Object-Oriented Substation Event (GOOSE) messages, and the latter – Sampled Values (SV) data.  Fiber optic communication media at 100 Mbits/s or a higher rate is specified and used due to fiber’s electromagnetic compatibility characteristics.

Both GOOSE and SV messages as defined by IEC 61850-9-2 are mapped to Layer 2 Ethernet with dedicated Ether Types and multicast destination MAC addresses. Time synchronization service also utilizes packet-switched communication: Precision Time Protocol (PTP) messages are mapped to Layer 2 Ethernet per IEC/IEEE 61850-9-3:2016.

Though Ethernet was originally introduced as a “best effort technology”, i.e., only best effort is made to deliver data, so data delivery was not guaranteed; multiple enhancements were made to provide available and reliable Ethernet channels. These include full-duplex mode, advanced forwarding table lookup mechanisms, traffic prioritization, buffer space reservation, etc. Sending data to multiple recipients and data re-transmissions also help.

Multicast communication (one-to-many) is selected to increase the reliability of data delivery. IEC 61850 GOOSE messages in peer-to-peer communication are sent continuously at a configurable rate as a heartbeat by all devices publishing them. Upon a status change, new binary data is also multicasted in randomized bursts (with various time randomization methods) to increase the probability of receiving it by the destinations reliably. IEC 61850 SV messages with analog data samples are sent continuously, most commonly at 80 or 256 samples per power cycle rate. Multiple recipients can subscribe to this multicast data at the same time.

Both GOOSE and SV messages use IEEE 802.1q tags (Figure 3).  In IEC 61850 the priority value of 4 is set by default to prioritize the processing of these messages over other traffic. (The value of 4 was selected because older Ethernet switches only had 2 or 4 priority queues). VLAN ID value is set to 0 by default, identifying that no VLANs are configured, i.e., GOOSE and SV by default are priority-only frames.

Overall, there are different modes of switching – store-and-forward, cut-through, adaptive, wormhole etc. The main difference is that cut-through switches start forwarding immediately after receiving a fraction of the frame (64 byte – because it is in the first 64 bytes where possible frame collisions – in half-duplex – are detected), hence reducing switch latency but increasing the possibility of forwarding a bad frame. The store-and-forward switches wait for the whole frame to be stored before forwarding it – it is slower but you’re sure that frame is good. For reliability purposes, store-and-forward switches are used in substation automation.

When designing an SAS, it is very important to use switches that have precise characteristics that would allow for performant, reliable and cost-effective communication:

• Scalability – allows for tailoring the communication infrastructure to the size and the needs of the substation (primary + secondary equipment)
• Redundancy – when referred to in the context of the hardware devices, it usually means redundancy of critical components (power supply, supervisors); in the larger context of the system itself it requires the switch to support different redundancy protocols (like Parallel Redundancy Protocol – PRP, High-availability Seamless Redundancy – HSR or Rapid Spanning Tree Protocol – RSTP – see Figure 4)
• Speed and time – for some of the substation protection applications (like BBP, and BFP), minimal latency is a must for switches; for timing (like PTP), knowing exactly what’s the latency is a must, as well
• Flexibility – in providing crucial services for the management, configuration and filtering (e.g., IEEE 802.1q support for IEC 61850, RMON, MIB support)

Lately, a fifth characteristic has become important in all good SAS implementations:

• Cybersecurity – switches must support cybersecurity protocols and services (like MAC-based port security, IEEE802.1x authentication, Radius, configurable login attempts, login trail and many more)

A picture of a relay installed in a digital SAS is shown in the following Figure.

Use of Packets between Substations

To replace traditional SDH/SONET for communication between substations different packet-switched technologies have been discussed like ATM, Carrier Ethernet (CE), and Multi-Protocol Label Switching (MPLS). While ATM has been dropped from the market due to high complexity and high costs, pure Carrier Ethernet (CE) solutions have been focused on data center communication and are not seen in operational networks anymore. Today mainly MPLS is deployed. The two flavors of MPLS are IP/MPLS and MPLS-TP, which will be briefly explained below.

What is MPLS-TP?  MPLS is the technology building the backbone of the Internet today. It is a connection-oriented packet-switched transport technology. It frees up the network elements (routers/multiplexers) from learning the addresses of every individual device connected to it (as switches or routers must) and provides strong separation of different services through individual pseudo wires (MPLS-labeled services). MPLS, as a technology adds an additional header to the Ethernet frame and IP packet and encapsulates the original message in the pseudo wire. The MPLS header consists of 32 bits (4 bytes) and as shown in Figure 5, it includes 3 experimental bits used for Quality of Service (QoS) / priority information. Time–to-live (TTL) indicates the number of retransmissions or network hops.

MPLS is available in 2 flavors: IP/MPLS, a dynamic technology based on unidirectional connections, and MPLS-TP, an evolution of IP/MPLS, where missing features from SDH/SONET have been brought in. IIP/MPLS, designed for service providers, is relatively complex (due to the dynamics) and is considered unfit for real-time requirements without additional protocols (e.g., Resource Reservation Protocol for Traffic Engineering (RSVP-TE). MPLS-TP has bidirectional connections with full end-to-end supervision and configuration in a network management system (Figure 6).

With this, the technology gets much more simplified, and the devices focus primarily on the task of traffic forwarding, thereby improving the real-time behavior of MPLS-TP substantially as compared to IP/MPLS. Globally there is an increased interest in MPLS-TP for operational communication networks for critical infrastructure – especially in Power Utilities.

In comparison to IP/MPLS, MPLS-TP provides several features, that are inherently beneficial for operational networks. The main ones are:

• Bidirectional services:  essential for symmetrical communication channel characteristics
• End-to-end supervised communication channels:  essential to ensure the highest availability and real-time communication
• Static configuration of communication channels: essential for deterministic latency and asymmetry requirements demanded by critical services
• less than 50msec switchover times for protected services: essential for the highest availability
• Ease of use: the Network Management System (NMS)-based service provisioning with static configured services simplifies the provisioning and operation of the network substantially

MPLS-TP enables a relatively smooth migration of operational telecommunication networks in critical infrastructure due to SDH/SONET-like behavior and Network Management functionality.

Concepts are known from SDH/SONET, and the performance is comparable for many services including differential protection (with special provisions). CIGRE recommends this migration path in their Green Book – Utility Communication Networks and Services, where performance aspects and operational simplicity have been mentioned as key reasons for selecting MPLS-TP as the replacement for SDH/SONET in the future.

It minimizes power consumption due to the simplification of routers/switches/multiplexers. The use of complex protocols on IP/MPLS devices drives the need for power-hungry computing capacity on each network element. OAM (Operation Administration and Maintenance) enables inherently SDH/SONET-like Network Management due to end-to-end channel supervision, SDH/SONET-like redundancy schemes, and less than 50 msec switchover times.

Transformation from SDH/SONET to MPLS-TP:  Ethernet protocols include mechanisms for restoring connectivity when links become unavailable due to faults (e.g., Spanning Tree Protocols). However, the time needed for restoration is much longer than that of SDH/SONET (less than 50 msec). The same is also unsuitable for wide area networks since it does not scale for large networks. IP/MPLS uses IP-based protection algorithms (e.g., Open Shortest Path First (OSPF)-based), but the reaction times for such systems are in the range of 10s of seconds which is unsuitable for critical services requiring, e.g., 99.999% availability.

Further, all packet-switched technologies in wide area networks are based on store and forwarding principles, leading to the accumulation of jitter, wander, and delay asymmetries. Consequently, critical services such as teleprotection need specific functionality beyond the offering of pure MPLS-TP to meet all the requirements including those specified in IEC 60834-1. An application-specific Inter Working Function (IWF) is required to transmit such critical application data over packet-switched networks. Without considering and properly handling communication channel delay asymmetry in packet-switched networks, there is a high probability of wrong trips for differential relays.

The migration from SDH/SONET to MPLS transport   network   requires connecting the several hundred existing TDM-based teleprotection circuits connecting a wide variety of relays and Remedial Action Scheme (RAS) controllers to packet-switched network. The existing direct relay communication interfaces primarily use traditional IEEE C37.94, ITU-T G.703, RS232 and RS422 technologies, with a few ‘very legacy’ analog tone systems. To convert these traditional interfaces Circuit Emulation (CE) is used, which needs to ensure appropriate delay handling especially for differential protection. (Figure 7).

Modern trends in digital substations include the use of packet-switched networks for IEC 61850 (GOOSE and SV) messaging for inter-substation communication exchange between Intelligent Electronic Devices (IEDs). MPLS-TP can be enhanced to provide hitless zero loss communication path protection for such IEC 61850 channels.  (Figure 8).

Conclusion: When Tomorrow Comes: This article aims to convey the fact that the development of the substation communication goes hand-in-hand with the development of the power system itself. There are few trends that are shaping the industry already:

• Electricity usage will increase and will be more widespread, so the “boundary” between industrial and social applications and electric utilities will become more and more blurred
• For inter-substation communication, MPLS-TP will often be the preferred packet-switched technology due to its SDH/SONET – like behavior
• Distributed generation, renewables, DC-to-AC-to-DC integration and wide area functions will evolve to the point where intra and inter-substation communication will converge using IEC 61850
• Security and depth-of-information concerns will require that the state-of-the-art protocols of today will be integrated with Common Information Model (CIM) protocols, resulting in hierarchical views of the same physical environment; in places where the system will be extremely complex, software designed networking (SDN) might have a role to play

Whatever the future might bring, packet-switched communication is here to stay. 

Biographies:

Galina S. Antonova serves as a Technical Sales Engineer at Hitachi Energy in North America. She has 20+ years of experience in electrical engineering, data communications and time synchronization, applied to the electrical power industry. Galina received M. Sc. with Honors (1993) and Ph.D. (1997) from the State University of Telecommunications, St. Petersburg, Russia, and spent one year at University of British Columbia on a scholarship from Russian President. She is actively involved with IEEE PSRC, PSCCC and is a Canadian member of the IEC TC57 WG10. In her spare time Galina enjoys ice dancing, playing piano and growing lavender.

Mathias Kranich graduated from the University Karlsruhe in electrical engineering in 1994 and earned a diploma in economic sciences in 1995. He has worked for over 22 years in Hitachi Energy (formerly ABB) in the field of product management & technical marketing in utility communication and has vast experience in different telecommunication applications and technologies. He was also the representative of Switzerland in CIGRE D2 telecommunication work group and contributed several papers to CIGRE, CAGRE & IEEE. Currently his role is head of technical marketing for wired telecommunication solutions in Hitachi Energy based in Baden, Switzerland.

Sergiu Paduraru graduated from the University of Craiova in electrical engineering in 1990.

He has worked for over 25 years in Hitachi Energy (formerly ABB) in the field of product management, sales & technical marketing in grid automation and has dealt will all things – large and small – in the field of substation automation products and systems. Currently his role is head of technical marketing for grid automation products in Hitachi Energy based in Västeras, Sweden.