By Herbert Falk, Outside the Box Consulting Services, LLC, USA
There are several processes involved in providing a solid stance for cyber security.
The simplistic view of a cyber security cycle is:
- Risk Assessment: This is where utilities/companies evaluate the cyber threat vectors and what mitigations or procedures should be adopted. The IEC/ISA 62443 series provides guidance
- Design: Based upon the Risk Assessment, cyber security mitigations, architectures, and designs are developed. This is particularly important for a Digital Substation (e.g. IEC 61850) with or without externally routable connections
- Procedure Development: Based upon Risk Assessment and Design, procedure development to provide regulatory documentation, cyber security best practices, testing, and incident responses all need to be developed. Some specialized procedures may need to be developed for Digital Substations
- Testing and Deployment: As IEC 61850, or digital substations, the cyber security of those needs to be tested in addition to the normal substation function. This testing needs to be considered at all test stages before the installation and deployment of the equipment in the field. Of importance is to test that cyber security does not harm resiliency and minimizes the impact on availability, as well as establishing that the cyber security measures mitigate the risks for which they were designed
- Monitoring: The best cyber security systems, without monitoring, provide no capability to respond to a cyber event. Therefore, monitoring of the system is important
The cycle repeats periodically or based on an event. It is a process that is never-ending and should be consistently improving.
Collectively cyber security for IEC 61850 is based upon a combination of the IEC 61850 and IEC 62351 standards. These standards cover:
- IEC 61850 Client/Server security: Providing tamper prevention, authentication, and optional encryption (e.g., confidentiality) for Clients/Servers. Authentication is based upon X.509 digital the exchange of certificates
- Layer 2 GOOSE, Routable GOOSE, and Routable Sampled Values (e.g., for synchrophasors) security: These are multicast protocols and there is a standard that specifies how to negotiate a group key (for a specific IEC 61850 publication). Authentication to participate in the group, and obtain the required keys and policies, is relies on the exchange of X.509 digital certificates. The policies provide tamper prevention, authentication, and optional encryption
- Layer 2 Sampled Values security: Uses the same mechanisms as the other multicast protocols, but due to performance, encryption is not recommended
- Guidance and standards for X.509 certificate distribution and life cycle management: IEC 62351-9 provides information related to X.509 certificate acquisition and life cycle including protocols that can provide this service continuously. The lifecycle management is influenced by what is known as a Public Key Infrastructure (PKI).
- Guidance on how to monitor cyber events: There are several standards for this purpose which are based on Syslog or SNMP
The requirement to utilize X.509 certificates, and their lifecycle, for the IEC 61850 protocols means that PKI is foundational to the cyber security of IEC 61850. This means that testing of an IEC 61850 system also needs to test cyber security and the cyber security infrastructure.
There are various stages of testing involved in the deployment of IEC 61850 systems. These stages become even more important when implementing IEC 61850 cyber security as the cyber security infrastructure of each utility may be different. Therefore, testing needs to verify interoperability with the utility PKI and monitoring infrastructure as well as the other IEC 61850 devices in the system being deployed. The stages are fairly well defined:
- Purchase Orders (POs) and Request for Quotations (RFQs): Due to the options that utilities use within their specific PKI infrastructure the utility needs to specify the PKI options that they expect to be implemented. This is the foundation of testing the cyber security aspects of IEC 61850 applications and devices.
- Unit testing: validation that the IEC 61850 device/application can interoperate with the target PKI infrastructure technology
- Factory Acceptance Test (FAT): Offsite testing of an IEC 61850 system including cyber-security.
- Site Acceptance Test (SAT): On-site testing of an IEC 61850 system including cyber-security.
- Commissioning: Field deployment and testing of an IEC 61850 system including cyber-security.
Supporting Cyber Security
Several different general requirements need to be understood to fully test IEC 61850 cyber security. These include:
- Testing and integration with the utility Public Key Infrastructure
- Testing of security event monitoring
- Testing of utility cybersecurity-related procedures
PKI Infrastructure: All current IEC 61850 cyber security technologies as specified by IEC 62351-6 utilize X.509 Digital Certificates (specified by ITU SERIES X: DATA NETWORKS, OPEN SYSTEM COMMUNICATIONS AND SECURITY Directory X.509 : Information technology – Open Systems Interconnection – The Directory: Public-key and attribute certificate frameworks) for various aspects. A PKI infrastructure manages the issuance and revocation of such certificates.
Figure 1 depicts the two deployment options that utilities tend to utilize. One is a wholly owned internal PKI infrastructure where the Root Certificate Authority is owned and operated by the utility itself. The other is an infrastructure whose Root Certificate Authority is owned and operated by a trusted third party (e.g., Verisign, etc.). The risk of exposure of the private keys of the Root Certificate Authorities means that the governance and protection of the Root Certificate may be beyond the capabilities of many utilities and thus the use of a trusted third party.
Root Certificate Authorities should never interact directly with IEC 61850 devices or applications. Instead, the Root Certificate Authorities are used to establish trust with Intermediate Authorities (a.k.a Regional Authorities) by signing the Intermediate Authority (IA) public identity X.509 certificate thus establishing a chain of trust. Intermediate, or Regional Authorities, can extend the chain of trust by signing other public certificates of underlying Intermediate or Regional Authorities.
At some point in the PKI infrastructure, an IA will be responsible for issuing and maintaining X.509 Identity and Attribute Certificates (RFC 5755: An Internet Attribute Certificate Profile for Authorization) that are utilized by IEC 61850 cyber security. The purposes of these two certificates can be simplified to:
- Identity Certificates are utilized to establish trust between IEC 61850 devices and applications relating to an authorized identity. Additionally, such trusted certificates allow the secure transfer of public key information that may be utilized by IEC 61850 cybersecurity-related protocols
- Attribute Certificates are utilized to authorize specific actions and privileges between two or more IEC 61850 devices or applications. Eventually, these certificates will be utilized for Role Based Access Control as specified by IEC 61850-90-19: Using Role Based Access Control (RBAC) and IEC 61850
The types of certificates are signed by the IA and thus, as long as the chain of trust is not interrupted, the certificates can be trusted.
The typical way PKI generates a certificate can be simplified for IEC 61850.
Figure 3 depicts a simplified process for issuing an IEC 61850 X.509 Identity certificate. The process starts with the device generating a public/private key pair. Information encrypted with the Public Key can only be decrypted by the holder of the private key. Public keys can be used to verify signatures generated with the private key thereby providing authentication to the identity of the originator of the signed information. To obtain an Identity Certificate, it can be self-signed or signed/generated by a PKI Intermediate or Regional Authority.
The generation of the PKI-issued Identity certificate typically requires a PKCS #10 (RFC 2986: PKCS #10: Certification Request Syntax Specification Version 1.7) based Certificate Signing Request (CSR). This request contains the public key and has a signature signed by the private key. The PKI Authority is provided approval to issue the certificate either through human interaction or policy. All certificates, whether self-signed or PKI Authority generated, contain an expiration date. Beyond this, the certificate should no longer be utilized in new transactions.
The IEC 61850 device/application then imports the certificate and can determine if the Identity certificate belongs to it by comparing the public key embedded in the certificate. There are two other primary functions that an IA provides related to Identity certificates:
- Renewal of certificates. Certificates have an expiration date and need to be renewed before expiration to allow continuous operation of an IEC 61850 system
- Revocation of compromised certificates. This typically means that a private key has been compromised and therefore its equivalent public key, and therefore its Identity certificate, can no longer be trusted. Revocation removes the capability of a device or application to operate within the managed system until new private/public keys and certificates can be generated
There are several different mechanisms (and options) that utilities may choose to utilize to address these functions. Due to the PKI choices of the utility, it is important that RFQs, or POs, specify the options that IEC 61850 devices and applications must support.
Issuing of Original Identity Certificates: Three mechanisms can be utilized to acquire an original Identity Certificate, from an IA, using an IEC 61850 device internally generated keys.
Manual Certificate Signing Request (CSR): A file can be exported from the device that contains a rudimentary certificate and public key. This file represents a Certificate Signing Request (CSR) and should be of the PCKS#10 format. The IA imports the CSR and the IA generates, either with or without human intervention, an Identity Certificate. This certificate is typically provided in a textual file format referred to as a PEM format.
This file is then imported by the device that generated the CSR. The file is validated based on trust (e.g. configuration) of the IA and the public key contained within the PEM file.
Although manual operation is feasible on a small scale, it would prove cumbersome at a large scale or to utilize for renewal.
Protocol Based: There are a couple of protocols specified by IEC 62351-9 that allow for certificates to be acquired and managed. Utilities may choose to support either SCEP, EST, or both.
Simple Certificate Enrollment Protocol (SCEP): SCEP (RFC 8894: Simple Certificate Enrolment Protocol) is a standard that provides the capability to acquire and renew Identity certificates based on a pre-existing key pair. It can also provide the entire chain of trust (e.g., all the IAs in the PKI chain to the Root CA). It can also provide Certificate Revocation Lists (CRLs) (specified by RFC 5280: Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile).
The configuration of the Trust Anchor and the IP address of the SCEP endpoint are needed to perform these functions.
SCEP is encrypted at the application layer and does not require a secure transport layer (e.g., Transport Layer Security).
Enrollment over Secure Transport (EST): EST provides similar capabilities to SCEP but requires the use of Transport Layer Security. IEC 61850 is transitioning to the use of TLS 1.3 (RFC 8446:
The Transport Layer Security (TLS) Protocol Version 1.3) for Client/Server services. Therefore, most IEC 61850 secure Client/Server implementations would have TLS already in the device.
Renewal of Identity Certificates: Most utilities will choose to implement SCEP and/or EST for renewal.
Revocation of Identity Certificates: Most utilities will choose to utilize the Online Certificate Status Protocol (OCSP) (RFC 6960: X.509 Internet Public Key Infrastructure Online Certificate Status Protocol – OCSP) instead of SCEP, EST, or manual distribution of CRLs. OCSP support is becoming more important as TLS 1.3 has introduced the concept of OCSP Stapling.
OCSP stapling can be utilized to enhance the security stance of IEC 61850 applications by providing the entire chain of trust on connection/protocol initiation.
Testing PKI: The majority of PKI testing would involve X.509 certificate issuance, expiration, renewal, and revocation.
Certificate Issuance, Expiration, and Renewal: X.509 certificates all have an expiration date. IAs typically have a policy that can set the determination of this date by setting the number of days, or other time delta, from the date of issuance. Typically, utilities for operational systems would set the expiration to be no less than two (2) years.
Additionally, two other metrics determine when certificates can be renewed: the IA configuration of when actual renewal is allowed (typically a percentage of the overall lifetime/validity of the certificate) and the time at which an IEC 61850 application determines to start asking for renewal (typically 30 days).
Figure 2 shows the relationship between the interval that an application begins requesting renewal and the IA allowing renewals. During the overlap certificates will be renewed. Before the overlap, the same certificate may be issued (e.g. serial number, etc.) or there may be a denial of renewal or no response. During the overlap, the IA will provide a new certificate with a new expiration date and may supply the certificate with a new serial number.
Consider the example of an IA that is configured to allow renewals at 20% of the lifetime of a certificate and issues a two (2) year certificate. Unit, FAT, and SAT testing cannot afford to wait 584 days prior to testing for a successful renewal. Therefore, the IA’s used for the various testing stages need to have different policies. It is suggested that:
- Unit testing uses an IA that has the minimal renewal period possible, and that the expiry be set small (e.g. 1 day if possible)
- FAT or SAT utilizes a longer certificate lifetime that is less than the anticipated duration of the testing campaign. This will allow renewal and expiration detection and renewal impact on the System Under Test (SUT) to be determined without placing an undue burden on the SUT
To accomplish these objectives, utilities should consider a PKI architecture that facilitates this objective while being able to test the integration with the utility’s PKI infrastructure.
Figure 4 proposes that utilities stage at least one IA for each stage of testing. This allows for different policies to be set to enable more efficient testing while protecting the operational system. The IA for FAT may need to be commissioned and placed in a Demilitarized Zone (DMZ) so that it can be remotely accessed or be detachable from the infrastructure so it can be shipped to the FAT. Virtualization of all of the non-operational IAs may prove beneficial.
Recommended high-level test cases, per stage, are shown in Table 1.
Certificate Revocation: An analysis of the two prevalent revocation mechanisms (e.g. CRLs and OCSP), there are implications to testing revocation depending upon the mechanism being used by devices and applications to track revoked certificates.
Figure 5 provides simplified ASN.1 syntax for the delivery of revocation information via CRLs or OCSP. Implementations receiving this information have two choices to track revoked certificates. If an implementation tracks the revocation based on the combination of issuer/serial number (CRL) or the issuerNameHash/serial number (OCSP), then using different IAs for the various phases of testing would allow the same equipment to be utilized since new certificates would have different issuers and therefore the new certificates from the different IA could be utilized for testing. The IA architecture shown in Figure 4 would facilitate this type of migration between the various testing stages.
Testing revocation at specific stages will provide utilities with further information relating to the device interpretation of a revoked certificate as some revocation of the private key and any new certificate with the equivalent public key would not be allowed. Testing procedures would need to be modified based on the implementation strategy of the devices in the IEC 61850 system.
However, in the instance where a certificate is revoked due to the private key being compromised, the utility must have a procedure to remove and replace the device whose private key compromise was the reason for the revocation. It should be noted that the regenerating of public/private key pairs may require that the device be returned to the device vendor. Utilities need to understand this process on a vendor-by-vendor basis and write their procedures accordingly.
Monitoring the Cyber Security Event
Utilities may, or may not, have centralized event monitoring capabilities from IEC 61850 systems. Many utilities utilize IEC 61850 within a substation but communicate via serial communications externally to the substation due to lesser cyber regulatory requirements. However, without the ability to monitor, notify, and respond to cyber events, a good cyber security stance will not be achieved. More advanced utilities have realized that IP-encapsulated serial communications are not serial and should be treated as if the external communications were not serial.
For utilities with external non-serial connectivity, it is possible to deploy centralized, or de-centralized, security event monitoring. Integration into these types of systems is facilitated for IEC 61850 security via IEC 62351-7 (Simple Network Management Protocol (SNMP) version 3 ) and/or IEC 62351-14 (Syslog). It is recommended that utilities specify the support of Syslog/IEC 62351-14 mandatory support for IEC 61850 applications as well as a requirement that cyber events be placed into a least one user accessible Sequence of Event (SOE) logs.
The type of events that could be generated, besides PKI-related events, will vary based on the IEC 61850 services (e.g., Client/Server, GOOSE, R-GOOSE, SV, or R-SV). These IEC 61850-specific test cases will be a topic of the relevant service section.
These services need to be tested for PKI testing as part of IEC 61850 system testing. To be discussed in another article.
Cyber Security Procedures
Utilities will have regulatory, general, and substation-specific cyber security policies. However, there may need to be additional procedures defined that:
- Specify the procedures to enable/disable unused Ethernet ports so that test equipment (e.g., Transient Cyber Assets (TCAs)) can be utilized. This will be discussed in a follow-on article
- Procedures related to scheduling and recovering from penetration testing. Ideas for execution of penetration testing for IEC 61850 systems will be covered in a future article.
These procedures need to be executed, and validated, as part of FAT and SAT.
The philosophy should be that the procedures do not disable cyber security unless required and that manual re-enabling of cyber security is minimized.
The reason for this is if cyber security is disabled, and somebody forgets to re-enable it, the security stance and the reliability of the grid will be at risk. To be discussed in another article.
Testing of IEC 61850 Cyber-secure Protocols
As mentioned previously, IEC 61850 combined with IEC 62351 standards have cyber-secure exchanges for Client/Server, GOOSE, Routable GOOSE, Sampled Values, and Routable Sample Values. The test campaigns for these will be discussed in another article.
Biography:
Herbert Falk is an IEEE Fellow and has been involved with cyber security since 2002 through his involvement with the EPRI Utility Communication Architecture (UCA), EPRI Intelligrid, NIST Smartgrid, and IEC TC57. He is the US Technical Advisory Group lead to IEC TC57 WG15 which is responsible for cyber security within IEC TC57. He is an editor of IEC 61850 and was responsible for several of the security standards associated with IEC 61850 and is actively working with the DNP Security Task Force. He actively participates in IEEE Power Systems Communication and Cyber Committee (PSCCC). He has performed several cyber risk/cost assessments around the world and has assisted many companies in improving their cyber footing and products. He is currently a co-chair of ISA 99 WG14 working on standardization of defense in depth profiles for the Energy OT sector.
