Measuring Propagation Delays and Assessing Performance of Power Utility Communication Networks

Author: Fred Steinhauser, OMICRON electronics, Austria

Many of the concerns originated from network arbitration and congestion problems that occurred in the early days of Ethernet, which are much less an issue with today's switched Ethernet networks. As long as only client/server and GOOSE traffic are present on the network, there are no real challenges because the network load is relatively low. But when Sampled Values become involved, high packet rates and sustained network load occurs. The real-time traffic must arrive timely at the subscribers. Delays and jitter become an issue.

This article describes circumstances when Sampled Values are involved and illustrates some key issues. It also shows that such effects can be predicted to a certain degree and how these predictions are verified by measurements. This is useful for the design of a power utility communication network and for the verification of its performance.
For the propagation of protection-related messages over wide area networks, certain performance criteria have to be met, but these criteria are difficult to verify. The measurement methods used for evaluating the effects of packet interference can also be applied in a distributed measurement system to measure and assess the propagation delay characteristics of wide area networks.

Traffic Segregation with Buses
The term "fully digital" power utility protection, automation, and control system targets at an installation that utilizes all kinds of communication defined in IEC 61850, i.e. client/server communication, GOOSE, and Sampled Values.

Although hardly mentioned in the IEC 61850 standard, the terms station bus and process bus are often used to characterize segments of a power utility communication network that transport different kinds of traffic. It is widely assumed that client/server traffic (also often called MMS after the transport protocol) is only present on the station bus, while the process bus is exclusively for Sampled Values. GOOSE may be on both of them. When no Sampled Values are used and only a station bus exists, the GOOSE messages are of course also sent over the station bus. Typically this is not a problem, because the bandwidth required for GOOSE is low in most cases. When a process bus is actually deployed, the GOOSE messages will more likely be on the process bus, because of their process-close real-time nature.

But the strict segregation between client/server traffic and Sampled Values as often assumed is not necessarily the case, especially when real bus topologies are used. The technical report IEC 61850 Part 90-4: Network Engineering Guidelines, contains a figure titled “Traffic patterns” (figure 57) that is shown as Figure 2. It illustrates where the different kinds of traffic client/server (C/S), GOOSE (GO), and Sampled Values (SV) occur side-by-side in a power utility communication network, even with dedicated station bus and process bus.

The essential point in the given context is the fact that the station controller will not only communicate with the IEDs connected to the station bus, but also with the CB controllers connected to the process bus. The station controller may send controls to operate the CBs and receive reports about the execution of the commands. To reach the CB controllers, client/server traffic must be exchanged over the process bus, where it will interact with the Sampled Values.

Points of Interference
The components that manage the forwarding of the data packets in the local communication network are the Ethernet switches. Each IED has an individual connection to a so-called edge port on a switch. Between the switches, the connections are established by so-called trunk links, connected to trunk ports. The IEDs can send data to the switch at arbitrary times. The switch has then the task to forward the packets, either to another IED on the same switch, or to another switch via a trunk link. When multiple packets arrive from different IEDs at the same or almost the same time and need to be forwarded over the trunk link, the switch has to schedule the packets somehow, because the packets cannot go over the trunk link in parallel, but only sequentially.

IEC 61850 specifies the use of VLAN tags in order to create a fast lane for time critical traffic like GOOSE and Sampled Values. The use of the VLAN feature was not primarily intended to logically segregate the network, but to make use of the priority information that comes with the VLAN tag. The switch sorts the incoming packets into different queues according to the priority in the VLAN tag. When the next packet is to be forwarded over the trunk link, the packet from the highest ranked queue gets the preference. The exact selection depends on the actually implemented packet scheduling strategy in the switch.

But this does not magically resolve all issues. A high priority does not at all guarantee that the packet is immediately re-sent after its arrival in the switch. The trunk port must first become idle before a new packet can be transmitted. When the switch is already transmitting a packet, any newly arrived packet has to wait, regardless of its priority.

Both IEDs in Figure1 transmit packets that need to be forwarded over the trunk link. The illustrated packet timing refers to a case where a large, low priority packet from the IED on the top left arrives in the switch slightly before the high priority Sampled Values packet from the merging unit. The large packet (labelled MSEP for "maximum size Ethernet packet") is sent over the trunk link and the trunk port is occupied until the packet is completely transmitted. In the shown situation, the Sampled Value packet is delayed almost by the entire duration of the large packet, just because it arrived at the switch shortly after the transmission of the large packet had beg

Protecting your electrical assets? today and tomorrow