Cover Story

Where has the Process Bus got us so far?

by Thomas Charton, National Grid, UK, Dieter Binon, Elia, Belgium and Rene Troost, Stedin, The Netherlands

It is no secret that Protection, Automation and Control Systems (PACS) are slowly evolving and adopting digitalisation in more and more areas. The pace reflects the critical nature of these systems and their requirements with regards to operational security and dependability. However, it feels like our industry is reaching a distinct point on the journey where a significant number of trials and pilot implementations have been completed and the digital substation is becoming a part of “Business as Usual” delivery options.

Where has the Process Bus got us so far?

It is a timely opportunity to review and consolidate the collective experience and learning within our industry with regards to the adoption of IEC 61850 Process Bus as part of digital substation solutions. CIGRE working group B5.69 is attempting this task and has asked its members and stakeholders to provide feedback to a survey earlier this year.

Whilst a significant number of Process Bus projects have been described and discussed in publicly available literature and at industry conferences, there is also a large number of projects where little or no information is available in the public domain. Publications also tend to focus on specific aspects of a project in line with the objectives of the authors and focus on the chosen conference, journal or target audience.

The CIGRE B5.69 working group therefore launched a survey with the aim to also collect some information about these unreported projects and unreported aspects of published projects.

The survey was conducted between September 2020 and January 2021. It should be noted that this article provides a summary of the experience from industry stakeholders. Any recommendations should be critically evaluated and may not be applicable or valid in every context but will hopefully stimulate some discussion when finalizing designs for digital substation rollouts.

Survey Respondents

Overall, 33 responses by 32 industry stakeholders from 29 companies, departments or academic research institutes were analyzed in the survey. The respondents’ background is mainly technical (83%) whilst the remaining (17%) of respondents are providing feedback from a technical management perspective.

Geographically, the majority of responses stem from Europe, however contributions from Asia, Oceania and America have also been received. A wide range of organizations have provided responses including utilities, suppliers, industry, and academia as shown in Figure 1.

A Variety of Project Types

The projects covered a broad range of installation periods from 2008 to the near future, however an increase in the number of projects can be observed from 2018 onwards. This reflects the improving functionality and availability of compliant equipment, engineering tools and test tools for process bus applications as well as improved interoperability and compliance with edition 2 of IEC 61850. Process Bus projects were fairly evenly spread over the range of 35 kV to 400 kV.

Most projects feature some level of interoperability requirement. Protection IEDs from more than one manufacturer are used most frequently which may be driven by internal technical redundancy policy. Merging Units and Bay/Switchgear Control Units from multiple vendors are also used to demonstrate interoperability and validate the engineering process for multi-vendor solutions. Network switches are in most instances sourced from a single manufacturer as there are fewer concerns about interoperability and common modes of failure. Figure 2 shows the split between single vendor solutions and multi-vendor solutions for each of the categories of equipment used in the reference projects.

Drivers for Process Bus Deployment

The drivers and motivations behind the development of Process Bus technology can be varied but do in general include an economic component.

This will include reduced purchase cost, installation costs, operation and maintenance as well as total cost of ownership including end of life replacement. The development of process bus solutions and other one-off costs such as training and test tools will require some initial investment, but a majority of projects (73%) aim to achieve a saving on the total cost of ownership. Around two thirds of projects are aiming to demonstrate reduced installation, operation and maintenance costs (64% and 67% respectively). A smaller part of the projects (33%) is aiming at reduced purchase costs (see Figure 3).

In many cases the cost of LPITs, merging units, IEDs, network switches and cyber security measures may not currently translate into a saving for a first deployment, however there is potential for savings as volumes increase. It will also depend on what is included in the purchase cost in terms of engineering and installation where savings can be demonstrated. Other economic drivers that respondents mentioned were improved lifecycle management (including obsolescence) and efficiencies as a result of process improvements such as remote testing and commissioning as well as improved remote diagnostics and audit.

Other, non-economic, drivers for Process Bus (PB) trials and deployments were spread fairly evenly over the proposed options shown in Figure 4.

Drivers listed under “other” in the survey were in many cases indirectly linked to economic factors such as reduction of wiring, design and engineering work. Interoperability and efficiencies as a result of digitalization were also mentioned as well as the convenience of testing PB solutions making use of simulation and the test modes defined in edition 2. The reduction of resources used is also mentioned, which is an important environmental benefit, i.e., reduced footprint, copper wiring and cubicle space.

Challenges and Learning Points

Some of the main challenges for the deployment of PB solutions do clearly stand out (see Figure 5). The introduction of PB technology requires a new set of testing and diagnostic tools. This is a major cultural shift for many organizations and the staff that will require training often have very low availability.

Apart from the concerns around training, testing, maintenance and tools, respondents see challenges in the transition to industrial deployment, data modelling, changes to internal processes and internal acceptance as key obstacles. To maximize the value of PB solutions, standardization in design and data modelling is important, as well as the support within each organization to make upfront investments into this technology.

Some respondents have highlighted concerns regarding the economic viability and business case (15% and 9% respectively) whilst regulatory issues are only seen as an obstacle for around 12% of projects. Some additional concerns have been raised under the “Other” category.

The introduction of merging units may add several milliseconds to the overall response time in a functional chain. In some instances, this may be mitigated by the use of heavy-duty contacts for trip outputs but needs consideration for functions where these delays make a significant impact. The reliability of PB solutions in particular with regards to the use of outdoor cubicles and reliance on GPS and network switches is another concern as well as interoperability and compatibility with future standards over the lifetime of the equipment.

Design Considerations

Time synchronization is an important requirement for PB implementations and particularly important for applications with multiple Sampled Value (SV) stream inputs. Many designs include PTP, and various scenarios for failure, best master algorithm, transparent and boundary clock capability, networking as well as VLAN design need to be considered.

Reliability, availability, maintenance and replacement need to be built into the design, particularly where functional integration and the engineering process are concerned. Networking devices that have previously been hard wired can also make it more difficult to split responsibility between suppliers and can make testing interfaces more challenging. It is also useful to consider standard interfaces to primary plant that can ensure quick replacement of failed merging units without lengthy system outage requirements.

Maintenance and replacement also depend on suitable strategies for patch management, availability of spares and management of obsolescence which has been a challenge in some projects.

A modular approach to design based on standard templates with clearly defined interfaces has been mentioned as a benefit and helped optimize engineering effort. However, the creation of new modular PB solutions has also required additional engineering time which needs to be accounted for during the initial deployment, which is a general issue with a modular approach. It is also important to be aware of the challenges to utility-specific design templates due to the fact that first deployments will typically lead to significant new learning which will need to be built into future designs making stable design templates a longer-term goal.

Bottom-up design, where the design starts with the configuration of IEDs, has been found by some respondents to make testing and monitoring more difficult and more dependent on specific manufacturer’s tools. The majority of projects was based on this approach with some signaling an intention to move to a top-down approach in future.

Top-down design, starting with the System Specification (SSD file), including virtual IEDs has perceived benefits of being able to include testing and monitoring requirements as well as cyber security. It has been noted however that it is important to ensure that all requirements are captured clearly in the SSD file and that all interfaces including those to conventional equipment are covered.

As expected, one of the key benefits consisting of significantly simplified cabling requirements by removing most copper connections has been confirmed by many of the respondents.

The growing number of possibilities for network design including redundancy, flow control and VLAN design are making design choices for utilities more complex. Inter-bay process bus and full redundancy have also been found to create complexities in some projects. Network architectures with PRP have been reported as easy to maintain though.

Development Process

Some feedback has been provided where a significant in-house engagement in the development was possible. The development of PB solutions has required in some instances iterative testing of individual device configurations to reduce the complexity of troubleshooting on the complete system. Being able to test smaller subsystems separately has been noted as a benefit for future projects.

The development from the initial proof of concept to the fully standardized solution has been significant in some cases and the varying availability of non-mandatory parts of the IEC 61850 standard between equipment suppliers needs to be well managed as part of the development.

A key success factor has been the engagement of the end users at an early stage in the development process.

Testing Methods

In some instances, testing similar to the process for conventional solutions was carried out using secondary injection to the merging units. This can be done where conventional instrument transformers are used.

Making use of digital injection requires a step change in terms of tools and procedures for testing. Tools and skilled resource to inject simulated values and monitor PB network traffic will be required. Successful rollout requires careful planning and training. Some additional tests were suggested such as merging unit process delay.

Test and simulation mode are not always available at functional level (mainly at IED level). This needs to be considered when developing test plans and could impact the functional allocation of the design because of testing requirements. Clear naming conventions, well-structured and clean files and traffic management are noted as factors making testing easier. However, testing additional bays when extending a substation has been noted as a challenge.

Automated test procedures have been demonstrated successfully and will contribute to the overall benefits of PB deployment. Likewise, opportunities for remote testing have been mentioned as an important feature by some stakeholders.

Testing of LPIT

Feedback on testing of LPIT including the MUs echoes to a certain extent the comments made above on testing in general. The function of LPITs is generally tested as part of factory and laboratory tests to validate and tune measurements.

The respondents noted the lack of standardized LPIT testers though. In some projects, complete functional chain tests were included with injection on the primary side. However, HV injection can be difficult to carry out. This also includes the acceptance testing for transient behavior which might only be possible in a laboratory setup.

Test Tools

The importance of using the right tools for testing and commissioning is highlighted by several respondents. Users are looking for tools that:

  • Are user friendly and easy to use,
  • Are based on the features of IEC 61850 and can understand SCL files
  • Can create meaningful error messages based on observed traffic and cross check against SCL files
  • Implement a wide range of IEC 61850 features for interoperability
  • Provide protocol analysis for the relevant messages on the PB network
  • Offer a simple licensing model

The respondents found that further progress with test tools would be beneficial. The need for staff training is also flagged up again in this context.

Factory Acceptance Tests (FAT)

The learning points from FATs are confirming that a well conducted FAT can help reduce the required testing on site (SAT). With hindsight some respondents recommend spending more time on the FAT and involving all key stakeholders including site and maintenance staff, thus achieving full system “sign-off” prior to site installation.

The FAT has highlighted configuration issues in some instances such as “Peer to Peer” settings for PTP, VLAN configuration, behavior during loss of time synchronization, and system integration issues. Some users have also found that equipment did not behave as expected and some development requests have been fed back to suppliers.

Allowing for a prototyping phase and being able to amend the solution based on the results from the initial FAT is therefore recommended by some.

Network traffic load has been reported as an important test and some respondents therefore recommend testing the complete system using actual devices or simulating traffic. On the other hand, the benefits of separately testing individual devices or subsystems/modules have also been mentioned and the overall feedback is suggesting that the testing strategy should include both.

Site Acceptance Tests (SAT) and Commissioning

There is a consensus within the responses that the successful FAT has enabled spending a significantly shorter time on the SAT. Pre-developed, automated test procedures could be deployed (online) from a central point on the network in some cases.

Feasibility depends on network architectures and data flow control but should at least be possible within each bay or subsystem depending on the test features considered in the design.

Implementation

Whilst at a functional level the implementation delivered identical or enhanced functionality, the network architecture, design and control of data flows are now key features of process bus implementations. (Figure 6). The learning points and approach taken by the respondents was as follows:

  • Some respondents had a high priority to monitor all devices in terms of health, enable alarm reporting via MMS and distribute PTP timing signals directly from the station master clock (without Boundary Clock) which led them to provide station bus connectivity to all devices, including process level devices. In this context respondents also felt that a PRP architecture offered some benefit over HSR in terms of design and troubleshooting
  • Whilst there were no particular challenges with redundancy boxes reported, it was noted that, from an asset management point of view, these devices are seen as unnecessary additional points of failure and overheads
  • Process level equipment installed in field marshalling kiosks requires higher specification levels to meet the environmental requirements. It was mentioned that rear access panels offer benefits over swing frame designs as they create less of a risk to damage fibers
  • In some instances, further opportunities for more functional integration, reduction of IED count, more opportunities to provide redundancy/resilience and for simplified network architectures have been identified
  • Some limitations with regards to IED capability were reported such as lack of support for HSR and separate port allocation for station bus and process bus. Respondents also reported some challenges with interoperability

Cyber Security

Some respondents mentioned cyber security features that were used to secure networks and devices. Network traffic monitoring and intrusion detection systems were mentioned as well as a strict approach to zoning. Securing process traffic such as (R-)GOOSE that is transmitted between substation sites or to third party sites needs particular attention to cyber security. Some issues relating to immature standards were also reported but the responses reflect the general trend that this area is rapidly evolving.

Operation

Some projects have not been in service for long enough to provide significant operational experience, but some learning points could already be fed back as follows:

  • Communication channel monitoring has been provided as a permanent feature on the SCADA HMI and has been found to provide useful information regarding the system status and health
  • File version management, patching and updates have been challenging in some instances. This has been raised by several respondents
  • EMC issues may arise where equipment is installed in a near process environment
  • Good project management and documentation are essential for lifecycle management

Asset Management

The responses show that this area is as expected still developing as stakeholders are gaining experience with in-service PB solutions. The asset management metrics provided in the standard such as health, quality and performance indicators could not easily be linked to triggers for maintenance or replacements.

However, the ability to conduct system audits and reporting of alarms and errors has been welcomed.

Procedures for replacement of failed devices and validation of redundancy concepts have not been mentioned but will play an important role as older process bus solutions will approach their end of life.

Conclusions

The survey revealed that a lot of experience with Process Bus solutions has been accumulated in the industry. The conclusions we draw from this will partly be generic and partly reflect our bespoke priorities, policies and constraints and will provide us with the opportunity to optimize PAC designs and practice. It also shows that our industry is technically ready for wider deployment of digital solutions and digitalization.

However, a different mindset, skills, supporting tools and an efficient engineering process from specification to configuration are required to obtain the full benefits of a full digital substation. With the increasing urgency of addressing climate change, delivering the required energy system transition in an affordable way will be a priority.

Process Bus and digitalization might be just a small part of this but will certainly be a key technology on the journey.

Biographies:

Thomas Charton is a Senior Innovation Engineer at National Grid Electricity Transmission (NGET) in the UK. He joined National Grid in 2010 as SCADA and Communications Engineer and now leads NGETs Protection, Automation, Control, Digitalization and Operational Cyber Security Innovation. Enabling the transition of our energy networks to facilitate a net-zero emissions economy is his main priority. Thomas is a chartered electrical engineer, a member of the IET and received his M.Eng. in Electrical Engineering in 2000 followed by a Ph.D. in 2005.

Dieter Binon (dieter.binon@elia.be), graduated as an Electrical Engineer, and in 2009 Dieter started his career in smart grid applications at Eandis, a Belgian DSO. In 2019 Dieter joined Elia, the TSO for Belgium. Where he is currently a member of the asset management department as an expert in the field of digital substations. He as experience in the field of IEC61850 applications for over 10 years and is an active member of the CIGRE B5-69 working group.

Rene Troost(rene.troost@stedin.net), graduated as an Electrical Engineer, and started his career in Telecommunications. In 2014, Troost joined Stedin, the DSO for the South-West area of The Netherlands, including the Port of Rotterdam. Troost is currently responsible for substation automation policy in Stedin. He chairs the Dutch technical working group designing a DSO/TSO-DER interface, is an active member of the Dutch Committee of the IEC TC57 (NEC57) and the Technical Committee TC57 WG10 that deals with power system intelligent electronic devices (IEDs), communication and associated data models and chairs the Dutch CIGRE B5 study committee.