Authors: S. Ward, RFL Electronics, USA and A. Saciragic, Maritime Electric Company, Limited (A Fortis Company), Canada
Ethernet communication products and applications are becoming more entrenched as the new standard for data transport. Carriers are cutting costs by eliminating the need for overlay networks that require different equipment for different service types.
The primary goal for carriers is to accommodate all services on a single network and Ethernet IP (Internet Protocol) appears to be the clear winner. Ethernet networks cost less to build due to lower cost switches and they handle data transfer more efficiently than the traditional circuit-switched, timedivision- multiplexed (TDM) alternatives.
Connecting traditional voice, video and data over Ethernet networks has become an attractive alternative to running parallel voice and data networks. It saves money on call and leased-line service charges, while consolidating management, cutting maintenance costs, and increasing user productivity. This is achieved by converging two important traffic types onto one infrastructure, and takes advantage of the simplicity and efficiency of IP routing and Ethernet switching.
Voice over IP (VoIP) was first introduced in the mid-90s and provides an acceptable QoS (Quality of
Service) for telephone calls, but its' latency makes it unsuitable to use for synchronous TDM data, such as T1/E1. Consequently, a complementary technology, TDM over IP (TDMoIP), was developed.
TDM over IP duplicates traditional TDM services over an IP network. TDM services can be used to transport synchronous data, asynchronous data, 4 wire telephone quality voice, POTS, high quality audio, analog telemetry, or any other low speed communications requirement. T1/E1 data streams generated from a multiplexer are transported over a packet based network, recovered and turned back into synchronous data streams at the receiving location.
Ethernet Teleprotection Case Study
In 2006, Maritime Electric Company Limited (A Fortis Company) decided to perform teleprotection over Ethernet microwave radios. For cost reasons, this was the only communication media that was readily available. The procurement of new communications equipment was not cost effective. The Ethernet radios were already in existence for SCADA and telephone traffic. The communication network consists of "backbone system" with powerful 4T1 Ethernet radios, while ½ T1 Ethernet radios were installed in a "spur system". Ethernet microwave radios installed in the spur system have 768 kbps bandwidth, which is shared between the protection channels, the SCADA channels and telephone circuits. SCADA was transported with DNP over TCP/ IP, and phones had VoIP converters so they were connected directly to the router.
The system comprised four substations and two additional microwave radio repeater stations. The relatively simple protection signaling was as follows:
TDM over Ethernet
Our first approach was to use standard teleprotection equipment with TDM over Ethernet (TDMoE) converters to transport the synchronous teleprotection system (TPS) channels over the Ethernet network. The converters are similar in performance to TDMoIP but use a proprietary method. (There is a TDMoIP standard, but the technology is owned by RAD, which is why some other manufacturers prefer to use proprietary methods to avoid paying royalties).
The TDMoE Ethernet gateways have a T1 interface on the TDM side and each 64 kbps channel needed one TDMoE gateway that converted the synchronous 64 kbps channel for the teleprotection system to Ethernet packets. On the receiving end, the reverse process took place. The Ethernet portion of the communications scheme is shown within the dashed lines in figure 3.
It seems logical that there could have been just one TDMoE T1 converter at Station 4, but unlike a T1 multiplexer, the TDMoE device can not do grooming, i.e. separate the Ethernet packets to different destinations. Station 4 needed to send each 64 kbps channel to each of the other three substations; consequently three converters were required.
Note: The TDMoE devices were tested in the factory, and the results were reassuring.
Typical end to end delay times were in the order of 5 - 9 ms which met the requirements for the application. However, these times did not include any delays introduced by a network, and field results proved to be very different, for a number of reasons. One of the main problems was the fact that the Ethernet microwave radios have very limited bandwidth (768 kbps) and this bandwidth is shared between the protection channels, the SCADA channels and telephone circuits.
During commissioning it was noted that the TDMoE converters were consuming around 200 kbps of IP network bandwidth per 64 kbps channel.
The reason for this extra bandwidth on the IP side is that the TDMoE gateway needs to convert 64 kbps synchronous data into IP packets and during this process additional overhead such as IP framing bits and bits for clock recovery are added. Clearly this was a problem, especially for Station 4 where three 64 kbps protection channels were used. As the three 64 kbps channels, when converted to IP required a continuous bandwidth of 3 x 200 kbps, this practically consumed all of the available bandwidth. Not unexpectedly, field testing showed that a phone call would interrupt protection communications and the scheme was determined to be unsuitable for the application.
In addition, the measured oneway trip times between Station 1 and 4 with the TDMoE converter's jitter setting setup at 15 ms, were in an average of 38 ms and between Station 2 and 4, an average of 39 ms.
Shorter jitter buffers, that would have decreased the latency, resulted in an error message of "data underflow". Data underflow means that the converter did not receive a packet within the maximum allowed time period for TDM conversion.
While the poor results in this application were mainly due to the limited bandwidth, it does point out the importance of taking the network design into account. While the "theoretical" bandwidth might seem more than sufficient, converters needed for legacy devices may consume much more bandwidth than what would be obvious at first glance. This is certainly of importance when transitioning into an IP-based communications network that also needs to transport a large amount of data from legacy devices.
GOOSE over Ethernet
Having concluded that the TDMoE gateways could not be used, a new approach with Ethernet Teleprotection Systems was tested. This approach uses a new device that is based on IEC 61850 GOOSE messaging. Note that these substations did not use IEC 61850 and it was only the teleprotection devices that used this protocol for communications over the Ethernet network. After this upgrade, the system looked as in figure 4.
The TDMoE converters were taken out and replaced with a direct, IEC 61850 compliant, Ethernet module in the Teleprotection System. Station 4 required three Ethernet modules due to the fact that the router did not support VLAN configuration. The router provided virtual point-to-point connections via proprietary software and again, as Station 4 needed to communicate with all three remote substations; three separate ports on the router were required.
Using VLAN over a router that supports multicast messages would have eliminated two of the three Ethernet modules, as the Teleprotection System supports multicast as defined in IEC 61850.
Factory tests of the Ethernet teleprotection system were performed. The factory trip times for 500,000 trips at a rate of 5 per second through a fiber optic Ethernet switch were recorded.
Re-transmission times were 4 ms, i.e. the GOOSE trip message was repeated 2 times, each 4 ms apart.
The test results indicated that 99.7% of trips were based on the first GOOSE message, 0.29% tripped on the first re-transmission and 0.01% needed the second re-transmission. The teleprotection device sequentially, numbers and logs the GOOSE messages making it easy to verify network performance.
Tests were repeated at commissioning, and the end-to-end trip times over the network were 8 -10 ms.
It was concluded that the communication system was providing the performance required for the application and the available, even though limited bandwidth was sufficient for the devices.
A typical GOOSE message is 300 bytes, and only requires bandwidth when trip messages are sent (plus during the periodic broadcast every 60 seconds), and the burden on the network is minimal. Each GOOSE message occupies only 3.125 ms (300 x 8 bit / 768 kbps). However, the shared bandwidth with SCADA and phones did affect reliability. With the alarm timer set at 90 seconds, alarms for nonreceived GOOSE messages would occur several times per day.
Note that there is no re-transmit of the routine GOOSE message so if it is dropped due to network congestion, an alarm will result if no message is received within the set time.
This problem was "resolved" by increasing the alarm timer to 300 seconds.
However, reliability of the network was still considered sufficient for the application as a trip GOOSE message would be re-transmitted for a total of 3 times (up to 16 times are possible).
The lesson learned from this project is that:
When communicating over an Ethernet network, native Ethernet devices are much more efficient than legacy devices with converters.