Scalable Network Architecture based on IP Multicast for Synchrophasor Applications

by Maik G. Seewald and Ruben G. Lobo, Connected Energy Networks, Cisco

Grid modernization efforts, as well as the implementation and roll-out of smart grids raise the bar for the underlying communication architecture. New technologies and the integration of distributed energy resources along with the growing demand of energy are the main drivers to establish greater situational awareness of the power grid state, and are resulting in the need to control a high number of devices and systems in a scalable and reliable way.

This is especially true for systems and protection schemes that make use and process synchrophasor data. The transmission of synchrophasor data, often in a streaming way, needs scalable network architecture in order to meet the requirements regarding reliability, performance and security.
The article describes a network architecture, which is based on IP Multicast technologies and applies the TR IEC 61850-90-5 protocol specification to meet the mission critical requirements of Wide-area Monitoring, Protection, and Control (WAMPAC) systems.


Wide-area Monitoring, Protection, and Control Systems
Synchrophasor-data is being used in Wide Area Measurement Systems (WAMS), where time-synchronized phasor information is used to augment traditionally polled supervisory control and data acquisition (SCADA) information for the purpose of state estimation. The natural evolution of this technology is to enable a closed loop system where Protection, and Control functions are enacted on this monitored data. Such a WAMPAC system involves real-time analysis of system-wide information to detect instability, and the ability to communicate actionable data to one or more remote locations to counteract propagation of large disturbances.

One example of such a wide area system is a centralized Special Protection Scheme (SPS). A major objective of SPSs like System Integrity Protection Systems (SIPS) and Remedial Action Schemes (RAS) is to protect the transmission system from overloads after faults and failures.
Typically, such SIPS or RAS systems operate in an isolated fashion, responsible for a section, or a transmission corridor, of the power system.

However, with the growing demand of electrical power and the integration of Distributed Generation (DG) and Distributed Energy Resources (DER) systems, power flows have become more dynamic leading to higher level of operational uncertainties. In such conditions, distributed SPSs working in isolation without being aware of broader system conditions could potentially lead to uncoordinated operations resulting undesirable conditions having system wide effects.

In order to address the ramifications and the coupling between the different parts, a centralized logic is needed to coordinate and respond with full situational awareness. Such a system is capable of performing holistic protective actions based on the central controller's view of the entire monitored wide-area. Based on pre-programmed algorithms, the controlling instance will take action based on synchrophasor measurements. In order to build an even more comprehensive system, the controller can also use IED data to process and trigger control functions. Typical use cases are to respond on measurements or after a protection relay trips in order to shed loads or trigger generation. The system interacts with primary equipment such as breakers, switches, and different types of controllers. Performance and reliability are essential requirements.

The wide-area approach of this solution has very important implications on the communications paradigm between the controllers, and the numerous and widely dispersed monitoring and mitigation entities involved in the system. Effective communication between these elements lies at the heart of maintaining a stable grid, and special emphasis needs to be given to end-point processing, security, bandwidth consumption and latency sensitivity. Typically, upstream data flows for monitoring purposes involves communication between PMUs/IEDs in the substation and the distributed regional PDCs, centralized super PDCs, and controllers at redundant control centers.

Downstream data flows involving protection and control actions are between centralized controllers and IEDs/Relays residing at dispersed substations for the purpose of mitigating a disturbance when detected by the control center analytics. Multicast transport is well suited for this application since the communication model involves point to multipoint connectivity. Instead of burdening the endpoints (IEDs at the substation or central controllers) with the task of duplicating the high data rate messages to multiple destinations using unicast traffic flows, while at the same time having to meet the strict latency required for the application, it is more efficient to relegate this task to the network elements that are well suited to perform this function as shown in Figure 1 and Figure 2 respectively.

Recognizing these implications, the IEC has released a Technical Report IEC TR 61850-90-5 which specifies a protocol for transmitting digital state and time synchronized power measurement over wide area networks using IP multicast. As detailed in the section below, besides defining a secure and scalable alternative to IEEE C37.118.2 for transporting synchrophasors utilizing IEC 61850 constructs, the TR also specifies the notions of routable GOOSE (R-GOOSE) and routable Sampled Values (R-SV) by defining Layer 3 encapsulations for these constructs in addition to their existing Layer 2 definitions. This allows the WAMPAC system to use digital state information from IEDs in addition to the synchrophasor data from their PMU function in making control and protection decisions.

This is especially relevant to the wide-area communication network because it simplifies the transport model by facilitating a single Layer 3 transport mechanism to be used for both event driven digital state information and streaming synchrophasor data, as opposed to using a cumbersome mixed mode model of using Layer 3 transport for synchrophasors and Layer 2 transport for digital state information carried in L2 GOOSE.

Network Architecture based on Multicast Transport
The proposed network architecture for a typical WAMPAC system is built on multicast transport. In the case of private utility Wide Area Networks (WAN), the transport can be based on native IP multicast, Label Switched Multicast (LSM), or on Multicast VPN (MVPN) transport if isolation is required from other services. In cases where substations are not reachable over utility private assets, and leased service provider backhaul is used, multicast transport can be based on Group Encrypted Transport VPN (GETVPN) in order to encrypt the traffic over the 3rd party network. The latter case however is more likely to be used only for state estimation using a Wide-area Measurement System (WAMS), as closed loop control over a leased backhaul will require very tight service level agreements to meet the necessary latency budgets.

The packet based network architecture shown in Figure 3 considers the publisher of the data (be that an IED or the central controller) to be a classical multicast source. The publisher sends a continuous stream of synchrophasor measurement data, or event based digital state information, or protection commands to a pre-determined multicast destination address at which a certain set of subscribers are listening to. As specified by IEC TR 61850-90-5 the subscribers use IGMPv3 to signal group membership from a specific source sending to the group in order to enable a Protocol Independent Multicast-Source Specific Multicast (PIM SSM) forwarding model over the WAN. The PIM SSM forwarding model allows for bandwidth efficient multicast transport over the Layer 3 network by forwarding data down the multicast distribution tree only where subscribers are present.

For monitoring functions in the WAMPAC system, the super PDC in the control center acts as the subscriber for the synchrophasor and digital state information originated by the IEDs in the substation. It issues an IGMPv3 join (S1, G1) for group G1 from Source S1. In this case the source S1 is the monitoring IED that is connected to the IEC 61850 station bus in a given substation. The IGMP join from the super PDC is switched through the control center LAN up to the control center router that acts as the last hop router for the Layer 3 multicast traffic. The control center router issues a PIM join towards the S1 in order to build the multicast distribution tree rooted at the substation router acting as the first hop router. The monitoring IED in the substation streams synchrophasor data in R-SV format to the multicast group G1, which is switched through the substation LAN up to the substation router. The substation router forwards this traffic over the WAN multicast distribution tree using layer-3 PIM towards the super PDC.

With respect to the control and protections functions in the WAMPAC system, the mitigation relays connected at the station bus level in a set of substations involved in the mitigation action act as receivers for the digital state information and protection commands issued by the central controller. The mitigation relays issue IGMPv3 joins (S2, G2) for group G2 sourced by the central controller S2. The IGMP join(s) are switched through the station bus up to the substation router that that acts as the last hop router for the multicast group G2. The substation router generates a PIM join towards the source S1 to enable a multicast distribution tree for the group G2 rooted at the first hop control center router. When the central controller identifies a disturbance that needs to be acted upon based on the monitored data fed to the RAS analytics, it triggers the decision logic in the arming and mitigation system to deal with the contingency and the power system control function issues the appropriate control and protection commands in the R-GOOSE format to the multicast group G2. This multicast traffic is switched through the control center LAN up to the control center router which then forwards this traffic over the WAN multicast distribution tree for group G2 using Layer 3 PIM towards the mitigation relay.


BeijingSifang June 2016