Measuring Propagation Delays and Assessing Performance of Power Utility Communication Networks

Author: Fred Steinhauser, OMICRON electronics, Austria

An Experiment with Evaluation

An experiment shall give insight into the effect of load traffic on the propagation of time critical traffic in power utility communication networks. The measurements were made with Sampled Values, but the results apply to GOOSE messages as well. The test setup is the simplest possible layout that allows observing the described effects and it is shown in the following Figure 3.

The network is comprised of two switches S1 and S2, interconnected by a trunk link. A Sampled Value source (a merging unit) publishes one Sampled Values stream into the network. The measurement device captures the Sampled Values packets coming from the Sampled Values source before they enter the network at switch S1 and then again when they are broadcasted from the other switch S2, after traversing through the network.
A PC connected to switch S1 generates load traffic by "pinging" an IED which is connected to switch S2. This forces ICMP messages to be exchanged over the trunk link and therefore interfering with the Sampled Values. The ping utility used for generating this load traffic allows specifying the size and the frequency of the ICMP packets.

Theoretical Examination
The test setup is minimal, all traffic in the network is known and all parameters are under control. The only "noise" on the network is a few administrative messages which cause insignificant traffic. In such a well-defined environment, it is possible to estimate the effect of the traffic interference beforehand, giving some indication for the expected results of the measurements. Thus, the validity of the measurement method can be assessed.

The ICMP messages have the size Sp. At a given link speed nl, each packet occupies the link for a time


When the packets are issued with the frequency fp, they occupy a share p of the total bandwidth:


This is also the probability for a Sampled Value packet to get in conflict with an ICMP packet when being sent over the trunk link. Equation (2) shows that the probability for the occurrence of an additional delay due to interference increases with the size and the frequency of the ICMP packets, while it decreases with increasing link speed. In the worst case, a Sampled Value packet is delayed by the duration tp of the ICMP packets. A share of (1-p) of the packets will not be affected at all and will pass just as if there was no load traffic. The other packets will be delayed by a fraction of tp, with only a few packets showing the maximum additional delay tp. The affected Sampled Values packets will randomly meet the ICMP packets in different states of the sending progress with equal probability, thus a uniform distribution of the observed delays can be expected.

In the used test setup, all Ethernet links, also the trunk link, operate at nl =100 Mbit⁄s. The ICMP packets will be issued at a rate in the range of about 1000 packets per second, limited by what the ping source is able to achieve at a certain packet size. For the actual values applied in the experiment, the figures in Table 1 apply.

Measurements and Evaluation
Now, the delay times of the SV packets are to be measured under the different load conditions. This is done by performing a large number of individual measurements and evaluating the distribution of the delay times. The evaluation will be done from 10,000 individual measurements, which will deliver solid statistics. With 4000 Sampled Values packets per second, the necessary traffic can be captured in in only 2.5 seconds.

One condition is that the Sampled Values packets interfere with the ICMP packets randomly, that means that the rate for issuing the ICMP packets must not be correlated with the publishing frequency of the Sampled Values. Since there is no time synchronization between any of the components and the source of the ICMP packets is a PC with its typical mediocre timing properties, the timing is not exact and there is enough jitter to ensure the required randomness.

A first measurement is done without any ICMP traffic, with the Sampled Values alone on the network. This serves as a reference, because also in this case, a delay will be measured that comes from the storing and forwarding of the Sampled Values packets in the switches. Further delays due to interferences will add on top of the reference figure. The delay time distribution of the reference measurement is shown in Figure 4.

The average delay for the packets is about 26 µs, the deviations are minimal. All measured values remain in the interval between 25 µs and 28 µs. The delays are made up by the two store-and-forward processes in the cascaded Ethernet switches. The Sampled Values have a packet size of 152 bytes, so their duration is about 12 µs. The total delay is slightly larger than twice this value and the difference comes from the processing of the packets inside the switch before they are re-sent. In this example, the processing takes on average only about 1 µs per switch.
Figure 5 shows shows the delay time distribution of the Sampled Values when the load traffic with 500 byte ICMP packets is present

9592 of 10,000 packets, this is very close to the expected (1-p) = 96 % still have the delay value of the undisturbed case. The other 408 packets have different delays that look uniformly distributed, which makes perfectly sense and matches the expectations. The maximum delay value is 66 µs, which is exactly tp = 40 µs more than the reference value.

Figure 6 shows shows the delay time distribution of the Sampled Values when the load traffic with ICMP packets of maximum Ethernet packet size is present.

8894 packets, this is again close to the expected (1-p) =89 % still have the delay value of the undisturbed case. The delays of other 1106 packets look again uniformly distributed. The maximum delay value is now 149 µs, which is exactly tp = 123 µs more than the reference value.

The measurement results are in conformance with the expectations derived from the theoretical examination of the experiment. Thus, the measurement method and instrument can be considered valid.

Let?s start with organization in protection testing