Lessons Learned

Finnish Experiences in Building up Standardized PAC Schemes

by Patrik Lindblad, Mikko Holmgren, Kimmo Muttonen, FINGRID, Finland

Fingrid is the Finnish transmission system operator, transmitting electricity from producers to distribution system operators and major industrial users, as well as importing and exporting electrical energy to and from Finland’s neighboring countries. Fingrid owns most of the Finnish very-high-voltage grid (220kV and 420kV) and approximately half of the high-voltage grid infrastructure (123kV). Fingrid’s grid consists of 14,000 km of overhead lines and about 120 substations and more than 1000 feeder bays. The number of new substations to be constructed has lately risen remarkably due to a boom in the integration of renewable energy resources, mainly wind power.

The development of standardized protection and control (PAC) schemes at Fingrid has taken several years and these standardized PAC schemes are now, ultimately, implemented and taken into use in substation projects. Naturally, some finalization work is still needed and on-going. This article explains the needs, challenges and experiences from the whole step-by-step process, in which standardized PAC schemes have been built up.

Background:  Why do we think we need standardized PAC schemes? The Finnish substation and PAC system procurement and asset management has changed significantly during the latest decades.

Until mid-90’s, all substation construction work was done in-house with utility specifications and type approval testing of main protection devices (distance, current differential), although heterogeneous devices were used in the primary and secondary systems. As the power utility had the necessary resources, quite an effort was made to ensure correct operation of main protection. However, at that time restructuring of the electricity market took place and, thus, the grid owner started to purchase substation construction according to official public procurement regulation from the European Union. With this, the contractors started to offer PAC products that were new to Fingrid and that started an era of major development of the technical and functional specifications for PACS. Although some of the purchased new relay types seemingly fulfilled the owner’s requirements, there were several mistakes and lacking functionality in many of the first implementation cases. Therefore, all learnings from earlier projects (mistakes, compromises, technical restraints) needed to be updated in Fingrid’s technical specifications for PACS. At the same time, the digital protection devices became increasingly multifunctional and more complicated to configure for each implementation, which required more skills of the engineers and led unfortunately also to a slightly decreased quality of the PACS functionality.

After the millennium change, protection maintenance was outsourced step-by-step, which led to decreasing in-house testing skills and resources within the TSO. The number of substation projects started to increase significantly. So far, there was no notion of standard solution at the functional level: each PAC scheme was implemented in each substation project without systematically trying to reuse previous well-proven solutions or without any application type testing, also primary switchgear topology was non-standardized (i.e. with a lot of different variations). However, typical functional block diagrams for the main applications were used together with detailed specifications (text) for each project. Template (or standard) specifications and schematic drawings as appendices were continuously developed further based on experience from projects (e.g. due to misinterpretations and lack of details).

However, the quality of the implemented PACS functionality was still not acceptable due to too many human errors made in the PACS implementations. Therefore, some actions were taken:

  • Factory acceptance testing of each implementation project was introduced
  • Some new protection experts for supervising implementation projects and PACS maintenance were recruited
  • Type pre-approval process was already used for teleprotection devices only, but consisted mainly of EMC interference tests and a few selected functional tests
  • Unofficial “light” type pre-approval of RTUs was re-introduced, with thorough evaluation of the fulfilment of Fingrid’s technical and functional requirements including local support (evaluation on vendors’ documents and a Q&A lists), as well as some demonstrations of an actual built up RTU/HMI system (in FAT or substation)
  • Official type pre-approval of protection relays was introduced

The pre-approval of relay types was intended to solve the problem with varying quality and solutions by standardizing the configurations of the main protections. The main driver was the need to improve protection asset management, as we had version handling problems, differences in internal logics of same type of relays as almost same functionality could be engineered in different ways, errors found in recently commissioned relays (especially new types or versions), sometimes leading to unwanted trips.

Pre-approval process was adopted for the selected most complicated relay types (distance, current differential, busbar and breaker fail as well as directional EF relays) by developing standard solutions/configurations/connection diagrams for each relay type and for the main applications (400/220/110 kV, busbar topology, etc.). For such relay types that were already in use, the “standard configurations” were selected from some suitable earlier project with same/similar relays and applications. Each type of relay was finally FAT-tested in order to verify the fulfilment of the owner’s requirements. As deliverables, the owner received a description report, connection diagrams, standard configuration/setting files in vendor SW and OCC-file of verification testing.

The introduction of the type pre-approval process required some effort by Fingrid, even though the making of the standard configurations and verification testing was originally done by a service supplier (the most experienced relay test engineers of the main maintenance and project contractor). Fingrid experts evaluated the configurations made by the service supplier and witnessed the verification tests. However, as some detailed relay knowledge was lacking in-house, some issues were solved only after the following implementations of the new relays at site by other contractors or commissioning engineers. Also, the selected service supplier did not cope with the number of new relay types to be verified. After two years of very slow progress the vendors/contractors were required to take over the whole process of making the standard configurations.

Another problem seemed to be the fast development of versions, as the 1st versions of a new product often seemed to have some problems that had to be solved by introducing the next version of the relay firmware. Thus, the pre-approval process included some iterations and new versions needed to be adopted in next implementations, although version upgrades were not preferred. There was not sufficient co-operation between vendors, contractors and owner, in order to get the process smooth and efficient. Anyway, we gained a list of type approved PAC devices, which could be used in projects. Thus, the number of varying products was not wildly increased anymore. At that time, another internal development project was started, aiming at further standardizing protection and control systems within the utility. Joining the CIGRE working group B5.27 was part of this process, where we could collect experience from other utilities that already used standard schemes.

Fingrid learned a lot from participating in the WG work, like that it would be very advantageous to standardize its PAC schemes. Unfortunately, there were a few obstacles to overcome. The number of protection experts was too low to start a standardization process at that time and the variations of applications (primary topologies) had not decreased yet. Although the protection team started to grow, it happened slowly, and the new experts were soon involved in developing the migration of protection assets from an old obsolete asset management system to a totally new one with a lot of new features. Some arguments against further standardization came from the previous bad experiences from the start of pre-approval of relay types and from the risk of higher costs due to losing some competitors on the PAC market.

Finally, after some additional years of internal evaluations and studies, the decision was taken to go “all the way” with PAC scheme standardization in 2018. Having an “open” pre-approval process for IEDs was not efficient nor easy to manage. The responsibilities of the vendors seemed not to inspire them too much effort, they wanted to continue with the old products that were already used by Fingrid. However, there were several new products that were pre-approved, but the pre-approval process was too vague, full of surprises e.g. in time schedules and the need for expert resources. We also still struggled with version handling and quality challenges, especially after starting to adopt IEC 61850 solutions about ten years ago, meanwhile the number of substation and PAC renewal projects was strongly increasing. The challenges with version handling gave problems in spares management, as we could not be sure to have a fully working spare for each different application.

Furthermore, the first years of using IEC 61850 based PAC system was a painful experience, as we trusted the standard too much and we were not able to specify the solutions sufficiently in detail nor identify all the parts/details that should have been utility standardized already from our first implementations. Moving from mainly hardwired solutions (and some minor IED-RTU signaling via IEC-103 protocol) to mainly LAN-bus solutions was not easy. By strengthening the PAC expert teams even more, there was ultimately sufficient in-house knowledge and manpower to be able to develop the standardized PAC schemes and to renew the maintenance procedures.

The long “road” of development stages that ultimately led to the type approval project (shown in red) and finally to the implementation of standardized PAC schemes is shown in Figure1.

The Selected Standardization Approach

What do we consider a standard PAC scheme? Well, Fingrid’s interpretation of a “standardized PAC scheme” is a PAC scheme that is a comprehensive solution comprising of both standardized modules and a standardized concept, including not only the IEDs and computers but also the substation LAN topology, redundancy protocol and configuration of switches. The standard PAC schemes comprise of a standardized PAC solution for each type of feeder bay, compared to the earlier “loose” pre-approval concept, in which standardized configurations were built separately for each IED or relay type. Naturally, the standardization of the IEDs include their types, order codes and versions, but now also the full IED configurations for standardized control and protection functions as well as all needed communication.

Fingrid decided that the manufacturer’s offered system shall include at least the following devices: distance, line differential, bus bar protection with integrated breaker failure protection, transformer differential, 400 kV autoreclose and bay control unit IEDs and, also, RTU as a gateway to remote control center. The following major advantages were seen that lead to this decision to require the whole PAC system from the same manufacturer:

  • Clear indication that systems specially designed to the transmission system are wanted
  • Clear responsibility and straight forward trouble shooting in case of device or configuration problems: one manufacturer instead of separate protection and automation manufacturers
  • Interoperability interfaces are at minimum when there is one manufacturer that provides the entire protection and control system. If control systems and protection systems would be divided into separate type approvals or separate purchasing/frame contracts, it would lead to excessive extra testing
  • Supports the trend where protection and automation systems are integrated more and more to only one complete system.

Fingrid also wanted to standardize the substation LAN topology and redundancy protocol. Chosen topology was a double star connection with PRP redundancy protocol between IEDs and switches. Ring topology is used between switches with RSTP redundancy protocol. Standardizing substation LAN topology was seen as beneficial for enable in-depth learning of the chosen topology in every step of a substation life – from designing to constructing and commissioning to maintaining.

How did we develop it?  To start with, Fingrid introduced a future roadmap for how to manage the in-house skills of the overall control and protection systems. The question was what level of skills we want to have in-house and what level of outsourcing (i.e. what services) we want to rely upon in the future. The utility decision was to re-organize and further increase in-house expert resources significantly, especially in the area of PAC testers and telecom bus experts. This roadmap also included our development project for the implementation of type approved standardized PAC schemes. It was generally understood that the type approval process will demand a rather high effort from the expert resources of both Fingrid and the manufacturers.

One feature of Fingrid’s type approval process was the seeking of easy-to-implement solutions, which led to the decision not to allow multi-vendor solutions. This leads to single vendor-based systems, for which interoperability certainly can be granted and of which only a few devices are manufactured by other manufacturers than the main control and protection devices. The target was to have at least three different types of approved systems. With only one or even two approved solutions, there could be additional risks, such as, delivery problems, market availability and insufficient competition. Another decision to make the solutions easier to implement was to make the PAC schemes simpler by removing some of the factors that had given variations in the earlier protection principles and functionalities. With this in mind further functional integrations were introduced to the protection IEDs. One good example of a well succeeded integration is the 400/110/20kV power transformer protection which was previously made with nine separate IEDs and from now on with only three IEDs and a lot less of relay cubicle internal wiring and logics.

Type Approval Process

A cyclic project model was created for the type approval process. The approval process consisted of five main phases: designing, applying to type approval process, comparison and pre-selection, testing and creating configurations and, finally, starting the type approval period. The project phases with rough timelines are indicated in Figure 2. The type approval was open for all manufacturers. When the type approval opened, the manufacturer could enter their product families and products to the approval process. The manufacturer needed to point out the requested information and fill in detailed requirements forms per each device they offered. Comparison of products and the pre-selection was made based on the given information. The target was to select at least three manufacturers who were able to provide complete product-families including both control and protection devices.

A typical PAC system general overview solution that the standardized packages should comprise of is shown in Figure 3.

The pre-selected products were tested, and standard configurations were developed for them. The protection devices were tested separately to verify validity of the protection functions and together with the RTU and HMI devices to check the interoperability of the different devices. Standard configurations were made for every voltage level and for every main application. The version locked standard configurations will be finalized, where necessary, during the first implementation projects. The approval period was chosen to be four years and it can be extended by three option years. Thus, the start time of the next round of the type approval process can be flexible and adjusted according to Fingrid’s needs.

In the design stage of the type approval process, Fingrid developed the process descriptions, minimum requirements for qualification and technical specification lists that defined the pre-selection rules. To involve the manufacturers and to make the process realistic and feasible, Fingrid took contact with the vendors to seek for sufficient interactions with them, so that their point of views could be included from the beginning, as applicable, and their engagement could grow to the process early enough.

The vendors were able to take part in the type approval process freely, based on pre-qualification (PQ) process according to public procurement regulations within EU, although the result of this process does not include any purchasing of products belonging to such public procurement regulations. In the PQ-phase, Fingrid’s requirements were built up in several pre-specified minimum requirement (criteria) layers so that it was possible to determine the level of each of the products from all interested vendors. The concept is shown in Figure 4. It was important that these pre-specified requirement levels were well-defined, open, transparent and fair to all participants. They also needed to be clearly measurable and easy to evaluate from the submitted technical descriptions and product data.

The different requirement levels incorporated not only the degree of fulfilment of Fingrid’s technical specifications and offered product package comprehensiveness, but also such items as version handling issues, busbar protection solutions, easy-to-use implementation possibilities for required functionality, inter-compatibility, MTTF/MTBF, redundancy features, availability of local skills and support for projects (configuration and testing), CVs of PAC experts and testers of the type approval process participants, open competitiveness for implementation projects and relay model availability of Fingrid’s relay asset management systems.

As a result of this PQ-stage, three worldwide PAC vendors reached a specific minimum requirement level concerning both control and protection, and these three were selected and approved to take part in the next step, i.e. the full type approval process.

The three vendors participating in the type approval process were obligated to build up a product package solution in co-operation with Fingrid, which had to comply with all specified use cases and specifications and which would be possible to implement later on in actual substation PAC projects. Product families and models were originally offered by the vendors already in the PQ-stage as an overall concept, but later on they were further developed to include suitable IED order codes, IO extent, fixed hardware, software and firmware versions, configurations, connection diagrams and signal tables to comply with all relevant Fingrid’s requirements for functionality and performance. After the PQ stage, it was also discovered in the more comprehensive evaluations that a few originally offered products could not comply with all of Fingrid’s requirements and these were replaced with other products that were found to meet all the requirements.

Fingrid aimed to have a limited number of different models and order codes per each manufacturer. It was noticed that a detailed engineering of physical IO was needed to reach a level where the different control voltages from different batteries and MCBs were taken into account. IOs rarely have independent voltage contacts, but often a grouping of inputs and outputs can be used. This means that one group of inputs or outputs can be used for one dedicated control circuit only. It was also a target to minimize the external wiring and use of auxiliary relays in the relay cubicles during the IO engineering. All protection applications are designed without any auxiliary trip relays in the trip circuits i.e., IED contacts goes directly to the trip coils of the breaker.

During package development and especially when approaching approval stages, the IEDs and products were undergoing an extensive verification testing. With the help of comprehensive tests, the standard configurations of IEDs were often further developed and ultimately verified. This stage was very resource demanding for Fingrid and to the manufacturers.

As a lesson learned, it can be easily stated that constructing standardized PAC schemes with several sets of standard configuration templates and verifying the performance and functionality of the IEDs would have been impossible to realize without the additional recruits made to Fingrid’s PAC expert team.

When the verification tests were successfully passed by a vendor, the corresponding part of the package was approved. However, the final approval was still made for the entire protection and control system and it was given only after all major applications were tested and verified. The target was that with the intensive testing all critical shortcomings in the product itself or in the configuration would be revealed. Of course, something will for sure come up with the first projects, but these should, most probably, be fixed with configuration changes. It is also acknowledged and accepted that there will be some minor short comings in the products and configurations or even bugs and they will be tolerated without version upgrades. Of course, protection related critical bugs will need version upgrades but, luckily, they seem to be rare.

In this first type approval process, the process bus parts were dropped out of the scope, because Fingrid didn’t see it as realistic to take this step yet. However, many steps to learn from the fundamentals of a digital substation have been put into use such as precise time according to IEEE 1588 and seamless redundancy protocol of the substation LAN in form of PRP.

Implementing the Standardized PAC Schemes?

Fingrid made an implementation plan of how to use the type approved PAC solutions in actual projects.

From now on the type approved PAC packages can be used in each substation project by the main contractor (winning bidder), which may be the same as one of the vendors of a type approved PAC solution or any other vendor-independent 3rd party entrepreneur. The contractor is obligated to select from one of the type approved systems according to his preferences. However, the manufacturer is not allowed to make exclusive rights agreements with only one of the contractors or to mix and match the IEDs and products between the type approved systems.

The implementation plan included a comprehensive training for in-house PAC engineers and experts as well as those of the contractors and vendors. The training sessions consisted of:

  • One common two-day training, in which Fingrid’s PAC and test experts presented its PAC concepts and requirements in a rather detailed way and who clarified and highlighted the newest modifications that had been made to the PAC requirements and specifications when using the type approval concept.
  • Three-day trainings, one for each vendors’ type approved system, in which the vendor experts instructed the PAC experts of both 3rd party contractors and Fingrid on how to work with and configure their IEDs and systems
  • In-house training program is launched where Fingrid’s specialists, who participated in the FAT tests, will train in-depth the other PAC specialists of Fingrid.

It is recognized that maintaining the achieved standardization level will require good instructions and plans how to work with the standardized PAC systems. It is also understood that a full commitment and co-operation is needed between all parties to sustain the high level of standardization in the future also. This can be achieved with open and honest discussions and trainings with contractors to increase the understanding of the standard configurations and willingness to use and maintain the made configurations.

At the time of the writing of this article, first projects incorporating the type approved standardized PAC schemes are out for bidding. As a result of the type approval process, the full protection and control system of each manufacturer are approved to be offered to Fingrid’s substation projects by the contractors.

It will be interesting to see how much effort saving the PAC implementations will give when adopting these solutions compared to the earlier projects demanding rather high workload for engineering and supervision of each different solution by each different contractor.

Experiences: What are our main experiences of the PAC standardization process so far?

  • Needs fulltime commitment from all parties
  • The development and implementation process was very successful
  • Commitment by the vendors and contractors was excellent.

Model and version lock: Fingrid chose to freeze the models and versions of the products that were tested. This means that primarily all substation projects are made with the defined order codes and versions. Of course, some exceptions will come due to some specific reasons and special cases, but a rough target is that at least 80 % of the projects would be implemented with the pre-defined standardized order codes.

Long approval period: Fingrid will aim to have a long approval period, four to seven years, with these approved systems so that the benefits can grow in time. A long approval period will ensure good benefit-cost ratio as the project was so resource intensive. The in-built flexibility with several option years is deliberate in order to be able to optimize the time schedule of the next type-approval project.

Fingrid has a strong confidence that the new type approval process helps in the up-coming energy sector transformation. Fingrid is facing its highest rate of investments soon and we have a strong belief that the work and money invested in the PAC standardization project will pay off in the following way:

  • In-house skills enhancement
  • Good standard configuration basis of PAC systems for the contractors
  • Leaner quality control in substation projects
  • Clear responsibilities when trouble shooting
  • Easier maintenance and spares management.
  • Less product and version variation
  • Improved level of functional integrations
  • Less devices, internal relay panel wirings and auxiliary relays.

Lessons learned / Key factors to success

Support and willingness of top management: The project was given a status as a company level strategic project which ensured sufficient resources and prioritization among many other projects. Standardization is a huge project and needs fulltime commitment from many specialists.

Decision to have broad and deep inhouse knowhow: Full commitment and leading role from Fingrid’s specialists to assure high quality and Fingrid specific operation of the applications. Fingrid’s existing strong FAT testing culture and knowhow was a necessity for finalizing the applications.

Fully committed manufacturers:  Skilled engineers and strong R&D support from manufacturer’s side and good co-operation with Fingrid ensured successful utilization and configuration of the products. Participated companies committed to the timetable and scope fully and saw the benefits and opportunities that will come during approval period.  

Biographies:

Patrik Lindblad (M.Sc. El.Eng. from Technical University of Helsinki) has more than 30 years’ experience of control and protection in different positions (among others as leader for the protection team) and is working at Fingrid (Finnish TSO) since 1997. Now, he is a senior expert of HVDC and FACTS C&P systems. 

He has participated in several CIGRE working groups for Study Committees B3 „Substations”, B5 „Protection and Automation” (Finnish national member 2012-16) and B4 „DC systems and power electronics”, being co-writer of several CIGRE Technical Brochures. He has participated in ENTSO-E task forces relating to IEC 61850 and HVDC reliability issues.

Mikko Holmgren (M.Sc. El.Eng. from Technical University of Helsinki) has been working at Fingrid since 2014 with protection and automation systems. Currently, he is working as an expert in the PACS unit in Fingrid. He has led Fingrid’s type approval project for the last three years. He is currently the Finnish national member in CIGRE B5. Prior to Fingrid, he worked in the power generation sector as an engineer and consultant for generator and transformer systems for six years.

Kimmo Muttonen (M.Sc. El.eng. from LUT University and eMBA from Aalto University)

Experience in electrical systems for more than a decade in different roles. Worked at Fingrid past four years as manager of the control and protection systems unit with main responsibility of secondary systems development, management and maintenance. He is also the electrical work supervisor at Fingrid. Prior to Fingrid, he has over ten years of experience in the oil and gas sector, in various electrical engineering roles like maintenance manager, project manager, operations manager and expert. Tasks ranged from secondary systems to grid operation and procurement.”