Private Wireless Networks with Combined Technologies - The Future of Utility Communications

Authors: Steven Kunsman and Mathias Kranich, ABB, USA

Private Wireless Networks with Combined Technologies – The Future of Utility Communications

by Steven Kunsman and Mathias Kranich, ABB, USA

Increasingly, electric utilities are deploying intelligent electronic devices (IEDs) and other smart apparatus in substations and along distribution feeders as well as equipping field workers with laptop, tablet and handheld computers. They are also installing specialized computers and software in substations plus enterprise software applications in their data centers to automate various facets of utility operations.  These technologies are often labeled Smart Grid.  They enable valuable applications such as automated metering infrastructure (AMI), substation automation, distribution automation, outage management, automatic load shedding, and the ability to manage alternative energy sources.

While in the core network these applications can be served by the already existing telecommunication network, in the access area telecommunication infrastructure is missing. Therefore, an additional component is required to make most of this intelligence productive – two-way communication, as shown in Figure 1. Two-way communication enables intelligent devices in the field to provide data to applications running on computers in substations and data centers. 
A modern wireless communication network is a key element in this overall electric utility communication infrastructure. Private wireless networks built using wireless mesh, broadband point-to-point (PTP), point-to-multipoint (PTMP) and narrowband PTP/PTMP technologies extend communication from the substation control house to the substation yard and along distribution feeders.
This article describes utility communication network architecture and the main wireless technologies with their characteristics. Then focus is set on FAN networks and elaborates how a combination of private wireless mesh and broadband PTP/PTMP technologies, coupled with narrowband wireless technologies for specific applications, can meet the requirements for modern wireless utility communication networks. It also looks at the vital role of cybersecurity and the need to protect these networks through the functionality and integration of wireless IP field area communication networks, offering utilities improved security and substantial value.

Utility Communication Network Architecture
Most utilities implement a two- to four-tier smart grid communication network architecture, depending on the specific applications they plan to deploy.  The tiers are defined and illustrated in the figureon page 38.

  • Tier 1: The utility's core IP network serves as the communication backbone connecting all transmission and distribution substations. This network is generally implemented with fiber-optics mainly based on SONET/MPLS-TP. In most cases, it provides a 10/100/1000MB Ethernet service with a virtual local area network (VLAN)-capable Ethernet equipment creating the demarcation point for the Tier 2 FAN. As a backbone it usually is optimized for speed, spans a large area and consists on a few nodes, with a limited number of interfaces.
  • Tier 2: The Field Area Network (FAN) is optimized to cost-efficiently connect a large number of automation and control devices, as well as provides convenient connectivity for field personnel. Substation automation devices (e.g., breaker controllers, voltage regulators, and remote terminal units (RTUs)), distribution automation devices (e.g., capacitor bank controllers, recloser controllers, and smart transformers), AMI collectors, and mobile workers equipped with laptops, tablets or handhelds easily connect to the FAN. As wireline carriers discontinue and sunset traditional analog leased line services, utilities are also deploying FANs to provide connectivity to rural substations and other remote assets. FANs are most often implemented with wireless networking technologies, usually a combination of broadband wireless mesh, point-to-point (PTP) and point-to-multipoint (PTMP), and/or cellular data links. Endpoint connections to the FAN can use wireless or wired Ethernet or serial links.
  • Tier 3: The Automated Metering Infrastructure network, also known as the Neighborhood Area Network (NAN), includes smart meters and AMI collectors, optimized even further on addressing a vast amount of devices at a very competitive cost point. Meters may communicate directly with collectors or via other meters using a meter meshing system. The NAN is generally implemented using narrow-band wireless mesh or cellular data. When a broadband wireless mesh network is used to implement the Tier 2 network, the AMI collectors in the Tier 3 networks are generally co-located with the mesh routers that form the Tier 2 network. The collectors attach to a wired Ethernet port on the mesh routers. The AMI network may also provide the communications interface for the Home Area Network
  • Tier 4: The Home Area Network (HAN), even closer to the household devices, is usually implemented using ZigBee or HomePlug technology, provide connectivity to smart grid devices, applications and displays inside homes and businesses. If supported by the meters used in the AMI system, HANs can connect to NANs, and, in turn, the rest of the utility communication network, via the smart meters deployed on the customers' premises. Otherwise, the HAN will generally connect to the utility's operations center via the Internet

A different view of the typical utility communication network architecture, showing how the tiers relate to components of the electricity distribution system, is show in Figure 2.  Let’s focus on the Tier 2 FAN.

Field Area Communication Network Technology Choices

Field Area Communication Network Technology Choices
Due to missing fiber infrastructure and high investment costs for the same, this tier is typically served by wireless telecommunication networks. As shown in Figure 3, numerous wireless technology choices exist for implementing FANs. While some, e.g., broadband wireless mesh networks, score highly across the board, none achieves the highest marks for all FAN requirements. This leads to a natural conclusion – a network that uses a mix of technologies, generally broadband mesh, PTP and PTMP, coupled with narrowband wireless PTP/PTMP for specific applications, will best be able to meet FAN requirements for a utility with a large and varied service territory.

Highly reliability: Broadband mesh networks provide high availability by automatically selecting the best route through the network from multiple radio frequency (RF) paths, channels and bands. Using these capabilities, wireless mesh networks can be designed to deliver five 9s system availability. At the other extreme, according to a study by Root Metrics, Verizon Wireless’s network has 99.2% reliability while AT&T was 98.3% reliable under normal conditions. This means you can expect 5 hours of downtime on Verizon and 12-14 hours of downtime on AT&T per month. Narrowband and broadband PTP/PTMP fall in the middle, with narrowband perhaps having a bit of a reliability advantage over broadband when operating in lower frequency, licensed bands.

Operating expenses: Because utilities’ rate structures often link rates to a specified return on assets, many utilities prefer making capital investments (buying infrastructure that they operate) over incurring operating expenses (purchasing a service). As a result, they require field area networking technologies to have low operating expenses. All private wireless network options – mesh, broadband PTP/PTMP and narrowband PTP/PTMP – have low operating expenses. Even some ongoing costs, e.g., extended warranty and maintenance, may be able to be treated as a capital expense if an up-front, multi-year agreement is negotiated. As one would expect when paying for a monthly service, cellular data options are opex intensive. An additional challenge for cellular networks is the dependability of the cellular network technology. We are now facing the sunset of 3G technology, which has huge impact of the connectivity to connected end-point and may result in a full replacement of the telecommunication infrastructure.

Interoperability: Broadband wireless mesh networks based on IEEE 802.11 provide the best interoperability because they support open standards including TCP/UDP/IP, 802.11 (Wi-Fi) and 802.3 (Ethernet). To integrate field devices and avoid unconnected assets, some wireless mesh routers can also support secure network connections to devices supporting modern automation such as DNP-3 protocol and IEC 61850 MMS and GOOSE communications. As well, the routers can provide connectivity to legacy devices utilizing RS-232 or RS-485 serial interfaces. Recently, broadband and narrowband products that also support this range or interfaces and protocols have come to market with the important caveat that none provide integrated Wi-Fi support. Most cellular data modems can interface only to the network of one carrier and support Ethernet as their only additional interface.

High capacity: The modern utility’s need for information demands a high capacity network that can manage the amount of data from modern high performance intelligent field devices and increasing demands from the digital mobile worker. However, a high capacity network is not the same as a network in which every link operates at blazingly fast speed. High link throughput is required where the network is aggregating data from a number of applications or the need to support a particularly bandwidth intensive application such as video surveillance.  Broadband wireless mesh networks have the capability to meet the high demands and application requirements proven to support in excess of 1 TB per day of transferred data.

Low latency: All private wireless network options can provide latency sufficient for most utility applications. Even time sensitive applications, which traditionally require dedicated wired connections, and the addition of IEC 61850 GOOSE layer 2 network traffic are being utilized for mission critical detection systems. With PTMP technologies, care must be taken to ensure that the scheduling algorithm does not cause latency to extend beyond acceptable limits. Latency for LTE-based cellular services are about an order of magnitude higher than for private wireless options, in the neighborhood of 90 ms for LTE versus sub-10 ms for private wireless technologies.
VLANs/QoS: Broadband mesh networks and broadband PTP/PTMP systems have long supported VLANs and QoS, enabling traffic from different applications and user groups to be segregated and permitting security, as well as QoS, policies can be tailored to the needs of each application/user group. QoS capability ensures that traffic for latency-sensitive mission-critical applications are prioritized relative to latency-insensitive communications such as metering data. More recently, narrowband PTP/PTMP systems have added these capabilities.

Mobility of end points: Cellular data services provide the best high-speed mobility capabilities. Broadband wireless mesh networks can also provide seamless, session-persistent roaming at vehicular speeds. PTP/PTMP technologies do not support mobility.

Manageability: Broadband mesh networks are, largely out of necessity, usually supported by a network (as opposed to device) management system. PTP/PTMP technologies generally provide device or link-by-link management only. However, as PTP/PTMP technologies become more sophisticated, they are beginning to support some network management functionality as well.

Cybersecurity: Using Enterprise Tools to Secure Utility IP-Based Wireless Networks
In today’s age of digitalization and the need for connected assets, cybersecurity concerns and fears become a barrier to enabling utility grid modernization.
Applications discussed earlier such as automated metering infrastructure (AMI), distribution automation (DA), substation automation (SA), and mobile workforce automation require two-way information flow. Systems that have traditionally used physically isolated, proprietary networks are evolving toward integrated, open standard, IP-based architectures to facilitate communications.

The need to protect these networks has become a paramount requirement in the utility’s digital transformation.
The functionality and integration of wireless IP field area communication networks offer utilities substantial value. They provide interoperable communications for a multitude of diverse endpoints, it is easy to add new applications, and utilizing common communication network benefits from consistency in areas such as security policies, as well as lower implementation costs.
But there are concerns. Like all networks, wireless IP field automation networks come with potential vulnerability to cyber-attacks. A strong protection for potential threats can be deployed by bringing enterprise-class security to field area networks.

The Evolution of Field Area Communication Networks

The Evolution of Field Area Communication Networks
Utilities are increasingly using wireless networks to monitor and control thousands of automation devices in the field, as well as to communicate with mobile workers. In the past, utilities have often used proprietary low-speed wireless communication systems with little security to implement their field area networks, generally deploying a separate network for each application.  Increasingly, utilities are turning to secure, open standard IP-based technology to provide field communications over broadband networks that can support many applications.
IP-based wireless field area communication networks provide many advantages. When built using standard technologies such as 802.11 and/or 802.16, they provide high speed and low latency compared to the proprietary networking technologies traditionally deployed in the field, enabling many smart grid applications to run on one network. They are very reliable, especially when tools such as mesh routing and TCP with reliable data delivery are employed. IP networks provide interoperable communications for a plethora of diverse endpoints. Unifying communications for many smart grid applications on one network provides for economical implementation, central management and consistent, end-to-end security policies.

Bringing Enterprise-Class Security to Utility Field Area Communication Networks
While IP-based field area communication networks present security challenges, they also bring security advantages. Chief among these is that the tools and techniques used to thwart cyber-attacks on IP networks have been honed for years by enterprises and are constantly being updated by the security community to battle emerging threats.
For more than a decade, enterprises have faced the same security challenges that now confront utilities using IP-based field area communication networks. As a result, a robust set of tools and techniques that are proven and time-tested are available to combat cyber-attacks on enterprise networks. Enterprises with stringent security requirements such as financial firms and the federal government have successfully transitioned to IP while strengthening their security capabilities. Enterprise network security tools that can and should be leveraged in utility field area communication networks include:

  • Internet Protocol Security (IPsec) virtual private networks (VPNs) authenticate the endpoints of a network connection and encrypt data transmission between the endpoints. They secure both system access and transmitted data from one end of the system to the other in a continuous tunnel. (See Figure 4)
  • Firewalls permit traffic for only authorized applications, protocols and users to travel over the network while blocking users and classes of traffic that are not permitted by the forwarding policy. When extended to the edge, firewalls can be used as an effective mechanism for protecting field area assets
  • RADIUS, 802.1x, and 802.11i authentication prevent unauthorized users and devices from accessing the network and enforce strong endpoint authentication
  • AES encryption prevents eavesdropping on management and control traffic as well as data transmission
  • HTTPS-based remote access enables secure device management
  • Virtual local area networks (VLANs) enable traffic from different applications and user groups to be segregated and permit security policies to be tailored to the needs of each application/user group. (See Figure 5)

The capabilities noted above have significant overlap. This is desirable as it permits implementation of a multi-layer security model that provides defense-in-depth. (See Figure 6.) For example, a field area communication network could control access using both firewalls and IPsec VPNs. If an intruder was able to defeat the firewall by, for example, spoofing the IP address of a legitimate user, they still could not access the network because they would lack the credentials required to log in to the VPN. Employing a defense in depth strategy can make the amount of resource required to penetrate the network prohibitively high for the would-be attacker.

Take it to the Edge

Take it to the Edge
Where security is implemented in a field area communication network is just as important as what is secured and how it is secured.  For maximum protection, security must be enabled at the edge of the network, in addition to locations closer to the network’s core.
As shown in Figure 7, in many utility networks, defense-in-depth security is limited to data centers or extended only to the router in the substation control house that connects to the utility’s IP backbone network. Security within the substation and along distribution feeders is limited to proprietary mechanisms or obscurity. This approach falls short of what is required for adequate network security.
A better approach extends the multi-layer, defense-in-depth model to the network’s edge, as shown in Figure 8. A good rule of thumb is that any security mechanism that a utility uses in its data center - including encryption, authentication, access control, firewalling, and VPN capabilities - should also be deployed in the substation yard and along distribution feeders. One way to do this is to have this essential security functionality (Figure 6) provided by the wireless routers located at the network’s edge.

Wireless routers at a network’s edge are often physically co-located with the IEDs, smart transformers and AMI collectors that attach to the FAN. Ideally, each of the connected smart grid endpoints would themselves be VPN capable. When they are not, which is often the case, it’s important that the wireless routers provide authentication, access control, firewalling, and VPN capabilities at the Ethernet or serial port where the smart grid endpoints connect.
Other security functionality, including firewalling, authentication and encryption, should also be implemented in the network infrastructure devices to which field automation assets connect.  Enforcing security policies at the edge of the network prevents unauthorized traffic from consuming network capacity and unauthorized users from probing network resources closer to the network core for security vulnerabilities.

Secure IP Communications to Legacy Endpoints
A common challenge in migrating from proprietary to IP-based field area communication networks is integrating legacy field automation endpoints that don’t support IP, Ethernet or standard wireless connections. Not only must legacy endpoints be able to communicate over the IP field area communication network, they must be able do so securely. Stranding legacy field assets, forcing their wholesale replacement or leaving them unsecured simply are not options.
To ensure successful integration, IP field area communication networks must support the physical interfaces used by legacy endpoints, most commonly RS-232 or RS-485 serial and convert them so they can be carried over standard wireless and Ethernet connections. The networks must also support translation or tunneling mechanisms so that data originally encapsulated in widely used utility and utility automation protocols, including DNP3 and IEC 61850, can be transported securely across the IP network. Finally, points where legacy devices connect to the IP field area communication networks must be as secure as interfaces to field automation devices that natively run IP.

Wireless IP field area communication networks for utilities provide substantial value but like all networks, come with potential vulnerability to cyber-attacks. The robust, time-tested set of tools and techniques that have been developed to combat cyber-attacks on enterprises can provide cybersecurity for field networks that is comparable to that of the most mission-critical enterprise networks in the world.
While properly securing utility field area communication networks is necessary to ensure field automation cybersecurity, it is not the only consideration. In addition, where applicable, proper deployment and use of anti-virus and anti-spyware software, intrusion detection systems and role-based authentication with robust passwords as well as strong physical security is required to fully protect valuable field cyber assets.  

Biographies

Steven Kunsman Director Product Management & Applications, ABB Power Grids – Grid Automation in North America. Steve joined ABB Inc. in 1984 and has over 34 years of experience in Substation Automation, Protection and Control. He graduated from Lafayette College with a BS in Electrical Engineering and Lehigh University with an MBA. He is an active member of the IEEE PES PSRC & PSCC Committees, PSCC Cyber Security Subcommittee chair and has held multiple WG chairs, past IEC TC57 US delegate in the development of the IEC61850 communication standard and UCA International Users Group Executive Committee Co-chair.

Mathias Kranich graduated from the University Karlsruhe in electrical and earned a diploma in economics in 1995. He has worked for over 19 years in the field of product management in utility communication and has vast experience in different communication applications and technologies. He is currently head of product management for telecommunication solutions in ABB.

Let?s start with organization in protection testing