By Ameen Handon, SUBNET Solutions Inc., Canada, and Indrek Kunnapuu, Elektrilevi OU, Hando Luus, Eesti Energia AS, and Rene Voog, Enefit Connect OU, Estonia
ELEKTRILEVI Background
Elektrilevi covers 95% of Estonia with its electrical grid and brings electricity to almost all Estonians. Elektrilevi has more than 533,000 electricity network service customers. To ensure electricity supply, Elektrilevi maintains and upgrades 63,000 kilometres of power lines and 25,300 substations throughout Estonia.
Due to Estonia’s low population density, a single consumer has to support a much larger portion of the network in comparison to many other European countries. This also affects the network-related costs.
Elektrilevi maintains and develops essential network services to create a more comfortable and simpler environment both for people and companies.
Elektrilevi aims to continuously improve our network service while maintaining a reasonable price range. To assure the quality of our service, Elektrilevi uses effective and efficient solutions, continuously work on weatherproofing the network and digitize the power distribution grid management.
In order to improve the quality of Elektrilevi network service, Elektrilevi makes daily efforts to enhance the automation of the network, develop innovative solutions and carry out data analysis. Elektrilevi believes that through the implementation of smart IT solutions, Elektrilevi makes life easier and more secure for everyone. The AREP deployment is one of the programs driven by this belief.
ELEKTRILEVI Remote Engineering Vision – the Beginning
Toomas Hendrik Ilves, the fourth president of Estonia (2006 to 2016) was quoted to say “Not many know where Estonia is, but everyone knows Skype. So now I say I’m the president of the country where Skype is.”
Skype revolutionized voice communications by leveraging IP networks to support voice calls around the world. With that same goal of using IP technology to improve life, in early 2010s, Elektrilevi began investigating efficiency improvements and cost savings that may be achieved through the deployment of remote engineering functionality of IP-connected IEDs.
At CIRED in 2017, Elektrilevi presented an article reporting on its two initial trial deployments of remote engineering systems using different user interface and communication architectures.
The first approach, that was in use until 2014, (Figure 6) enabled specialists to use either their own PC or two dedicated RM workstations to securely connect to IEDs in 40% of Elektrilevi’s Primary Stations via the utility’s secure SCADA WAN and Station VLANs. This approach added more flexibility for users but did require each user to have their own copy of IED management software installed and licensed. Also, it was considered not secure enough, since the user computers were simultaneously connected to SCADA WAN and corporate network. As a result, the system usage was limited to 30% (12/40) of potential users.
The second approach (Figure 7) added a centrally administered Terminal Server to which authorized users could RDP from their own PCs. This expanded system enabled users to remotely access IEDs at 320 sites. This second second generation system was more usable since it only required the central system to have all the necessary IED Management software installed and licensed.
This new system was more secure since this terminal server had no access to public networks. This approach also improved and simplified the user experience, resulting in usage statistics improving up to 70% (28/40) of all potential users. As this second system was expanded and usage increased, so did the technical challenges of managing all the users and the RM manuals that describe site-specific connection details. Another unexpected issue was encountered when trying to effectively run all the different IED application-specific software on the same operating system (OS).
These initial internal approaches did, however, validate the benefits of Remote Maintenance. It was estimated that 1.5 hours were saved on each task that would have otherwise required a site visit.
The Advanced Remote Engineering Platform (AREP) Vision
These initial approaches provided valuable learnings and were key in helping Elektrilevi in the development of a much more ambitious solution we refer to as the AREP (Advanced Remote Engineering Platform.)
The two initial efforts were primarily about enabling remote access to devices, thus avoiding the time and expense to travel to the field device. The AREP vision (Figure 8) was expanded to be much broader and more ambitious than these initial approaches. AREP is an engineering platform envisaged to provide a flexible, efficient, and secure way of working in heavily digitized critical infrastructure.
The overall objective of the AREP is to deploy a software system that enables remote engineering of operational technology (OT) devices used by the power distribution grid operators to monitor, update and automate management tasks of the respective devices.
For example: The solution defined requirements for the ability to automatically change passwords and retrieve Firmware Versions, Configuration Files, Event Logs and Fault Files on a user-initiated or scheduled basis.
Other use cases related to device passwords were also defined. These included managing IED Password policies, supporting a password checkout capability and performing automated IED password updates.
The intended users are engineers from Enefit group (a consortium of over 20 companies, including network owner Elektrilevi and maintenance service provider Enefit Connect), contractor companies who are servicing and building the grid, and IED vendors who are granted permission to access, monitor, and update their products.
The core of the AREP is based on a Privileged Access Management (PAM) software solution. The PAM solution is customized to support specific OT devices that are not available out-of-the-box.
To support contractors and vendors, the PAM supports a centralized work-order-based access management capability to provide them secure, temporary access to specific devices.
The AREP vision included European Union eID identification capability to identify the specific person performing work and eSignature for signing work-orders.
The AREP also provides capabilities for:
- Granular user profiles with grouping options
- Safe login for DSO internal users as well as authorized external users
- Enabling minimal necessary connectivity to IEDs, ensuring the secure virtualized connection to the IED is only opened for the duration required to perform the specific engineering task
By implementing the AREP, the need for physical travel to the IED on the grid is noticeably lower for the abovementioned parties, resulting in lower maintenance costs, quicker access to detailed network diagnostic data, and faster reaction on power distribution grid faults.
System Requirements Development and The AREP RFP
Elektrilevi developed a comprehensive set of business use cases and functional and non-functional requirements for the Advanced Remote Engineering Platform.
The business use cases allow for:
- Creating a secure way to access IED following the principle of least privileges
- Capability for granting work order-based access to IEDs to external parties
- User client access with application virtualization
- Changing IED passwords
- Verifying IED configuration version against baseline
- IED password checkout
- Fault event analysis
- Scheduled tasks (password change, event log retrieval, fault record retrieval, version checking, etc.)
- Extending AREP for new device types
- Reporting
To support the business use cases, functional requirements were specified for:
- Thirteen user-initiated actions
- Six scheduled actions
- Ten operational requirements
- Three security functions
- Eight analysis functions
- One architectural requirement
- 33 non-functional requirements (including security)
Beyond the Functional and Non-functional requirements, the AREP requirements specification also defined a large list of IEDs that needed to be supported by the platform. In developing the AREP requirements, it was identified that support is required for over 88 different device types from 24 different IED manufacturers.
The key technical requirements summarized above were described in detail in the AREP Request for Proposal (RFP) that Elektrilevi created. The AREP RFP included a detailed vendor evaluation scoring system to effectively evaluate the solutions’ ability to meet the AREP requirements.
The public tender also included system sizing requirements and a general architecture.
The initial production system is scaled to be 80 AREP users, 30 simultaneous sessions, 10,000 managed IEDs, and 70 IED management applications. Within 5 years the production system could be expanded to 300 AREP users, 50 simultaneous sessions, 100,000 managed IEDs, and 300 IED management applications.
Having gained an appreciation for the various complexities related to effective IED remote engineering from the initial two trial approaches, Elektrilevi elected to survey the market for IED management systems that would meet their functional requirements. Elektrilevi did some initial market research for an OT PAM solution. Many vendors had IT PAM capabilities, but Elektrilevi found that not many PAM vendors offered the specific OT device capabilities they sought. Several PAM vendors were contacted by Elektrilevi to discuss their AREP use cases.
A vendor was ultimately selected from the RFP process that satisfied the majority of their requirements and conformed closely to Elektrilevi’s proposed cybersecurity architecture. Elektrilevi entered into an agreement with SUBNET Solutions Inc. for deployment and development of their PowerSYSTEM Center product.
System Functionality Overview
The primary business motivation for the system is to reduce cybersecurity risks and to improve grid reliability by allowing faster collection and analysis of event and fault information. The entire system is designed within a cybersecurity framework that isolates the external users and corporate users from the OT network through use of DMZ and jump servers.
Functional requirements include the ability to remotely enable users to:
- Modify parameters and settings in an IED
- Retrieve IED firmware version
- Update IED firmware
- Retrieve event log
- Retrieve configuration file
- Update passwords
- Retrieve fault file
- Launch IED management tools in read-only mode
Automation requirements include:
- Configuration version management and baselining
- Firmware version management and baselining
- Event log management
- Fault record management
- Open COMTRADE files in analysis tools
Management requirements include:
- Create work orders
- Checkout a password (with expiry time)
- Password policy management
- Document management
- Audit / activity reports
- Managing access control
While a number of these functions are not directly tied to the primary use case of remote engineering, they are dictated by cybersecurity and architecture considerations.
All external connection is authorized through a work order system that limits access to the specific devices permitted in the work order and restricts this access to the duration specified in the work order.
All users are given specific permissions to allow them to perform specific functions with specific devices. For most users, the system automatically manages connection to the IEDs, removing the need for users to know passwords, network addresses, etc. For some user functions, such as fault record analysis, the system automatically collects information and delivers it to the user without them needing training in the use of the specific device vendor’s software tools, thus simplifying their work.
All actions by users and all automatic or scheduled actions are logged to provide full audit history of the operation of the system.
AREP Project Funding and Return on Investment
The Advanced Remote Engineering Platform project discussed in this article was co-financed by the Connecting Europe Facility of the European Union.
The digital part of the Connecting Europe Facility (CEF Digital) contributes to:
- Development of safe, secure and sustainable high-performance infrastructure, including Gigabit and 5G networks
- Increased capacity and resilience of digital backbone infrastructures
- The digitalization of transport and energy networks
The application process was successful and a report detailing the results of the solution will be completed following implementation of the AREP. One of the initiatives within this project is to disseminate the knowledge in EU countries about the secure remote engineering practices for critical infrastructures.
Elektrilevi has done some work to quantify the value of the operational efficiencies the remote engineering and automation capabilities would provide. These savings include the following aspects, calculated for a 7-year period:
1. Remote engineering resulting in reducing travel which covers ~20% of the project profit value
2. Reducing cyber security risks by decreasing potential attack surface. These cover ~80% of the project profit value
The global cyber threat level to critical infrastructure has increased significantly in recent years.
In the past the major threat may have had a cyber-crime focus, but now there is a very real, significant risk to critical infrastructure operations.
Other areas of potential savings that were not considered initially for this project, but may be realized going forward include:
1. Manual tasks that are reduced by automated jobs
2. Savings from avoiding early obsolescence of existing IEDs with weaker native cybersecurity capabilities
3. Other benefits from improved management and analysis of IED data (e.g. improved SAIDI)
AREP Deployment
Elektrilevi completed a Design Session for the solution in early 2022.
Following the completion of the Design Session, the solution was deployed in a lab environment at Elektrilevi offices to support testing the solution functionality with many of the required IEDs.
The system began proof -of -concept tests in Q4 of 2022. As every different device version is potentially unique, it is best practice that each unique device type is specifically tested. Fortunately, the solution allows uses to modify driver configuration to support device uniqueness, including future device changes throughout their product lifecycle. This methodical onboarding approach can be a bit time consuming initially, but the automated testing capabilities of the solution have significantly helped facilitate this process.
Also, during the testing process, some new key enhancements were identified as needed prior to AREP production deployment. One of the new enhancements identified was to provide specific IEC-61850 Relay Vendor Project file support. Traditionally, relay configuration files would be created and managed individually. However today, many IEC-61850 relay vendor configuration software tools define a single Project file to manage the configuration of a group of devices or substation. As a result, the configuration of many different devices is contained within one vendor project configuration file.
Changes to a Project file typically result in the need to deploy changes to some or all of the devices defined in the Project file.
With this enhancement, the AREP solution is able to support the secure remote access and device management on a Project file basis. This includes coordinating and managing the secure connectivity and changes to all the devices associated with the vendor Project file. To ensure they would continue to have a unified solution regardless of device vendor, support was implemented for the different relay vendor Project files utilized by Elektrilevi. A further enhancement was also implemented to aid in remote access management when different vendor applications cannot co-exist on the same computer.
Final preparation steps for the production deployment are currently being performed. Visibility of devices has already improved considerably as the setting up of task automation is being finalized, The process of redesigning the companies Remote Engineering and Access procedures documentation is also being completed.
The target date for production system is Q2, 2023 for Elektrilevi staff. The plan is to start onboarding external partners to be able to utilize the solution after the internal set-up is complete.Elektrilevi has expansion plans for the AREP system that include integrating other OT devices such as communications equipment and smart meters to the system.
Summary
Elektrilevi’s goals in implementing the AREP were:
- Upgrade the existing remote engineering solution by implementing a secure PAM system for OT devices that provides remote engineering access for all authorized users, with granular role-based and function-based access control of each user’s permissions and full audit trails of all manual and automated interactions with the OT devices
- To increase the number of supported OT devices and enhance remote engineering of configuration, logs and passwords
- Upgrade the existing IED Remote Engineering System terminal server with a new IED Remote ngineering System to increase security and run scheduled automated tasks such as passwords, logs etc.
- Add specific support for vendor specific IEC-61850 Project File management workflows
- To provide a solution that is easier to use and administer, ensuring high user utilization and effective system maintenance
- Provide remote engineering access to IEDs for contractors to reduce the need for travel, resulting in a reduction in CO2 emissions, and improved staff safety and staff efficiency. An important aspect of this is eID based authentication which will enable new efficient cross-border partnership between DSO and vendors or maintenance service providers
- Positively affect the DSO’s EBITDA
- Implement a state-of-the-art showcase system as an example for adoption by other utilities in the EU
The AREP system implements over 70 identified functional and non-functional requirements to support twelve business cases. The system is sized to support 300 users, fifty of whom can be simultaneously accessing Elektrilevi’s current fleet of over 10,000 IEDs. In final deployment, the system is required to support up to 100,000 IEDs.
Biographies:
Indrek Kunnapuu has a background in the Estonian Defence Forces and over a decade of experience in information technology and information security, with in-depth knowledge of implementing and maintaining communication infrastructure in critical services. He is currently the CISO at Elektrilevi, the primary DSO in Estonia. Indrek is also responsible for implementing an ISMS and a BCMS. Indrek holds a Bachelor of Arts in Social Sciences and Business Administration from Estonian Business School. Currently, he is working on a master’s degree in Cybersecurity.
Hando Luus – Innovative leader with engineering background and interest in novel technology. Currently IT product owner for Elektrilevi on Industrial Control Systems. In total 10 years of experience for the Estonian DSO in different roles in regards of substation automation and respective asset management. Passionate about smart solutions and solving the challenges in OT domain which goes hand in hand with cyber security.
Ameen Handon has a Bachelor of Electrical Engineering degree and began his career as an electric utility SCADA Telecontrol Engineer. Since 2000, Ameen has been President and CEO of SUBNET Solutions Inc. SUBNET provides software solutions and consulting services to electric utilities and other critical infrastructure industries around the world. SUBNET’s offerings provide a comprehensive, vendor agnostic solution for managing and securing the many different intelligent OT devices that are deployed to monitor and control critical infrastructure. SUBNET offerings also help securely integrate the data from these OT devices with mission critical operational control systems (SCADA/ADMS/DCS) and IT business intelligence systems.
Rene Voog is a business project manager at Enefit Connect which is a subsidiary of Eesti Energia. He has a background in project management from various fields, e.g., military technology, information- and operation technology systems and start-ups. In addition, Rene has previous experience lecturing higher mathematics and military technology at Estonian Military Academy. He has strong analytical and mathematical skills as he received a master’s degree in physics from the University of Tartu in 2017.
