Building Synchrophiezed Measurement Systems for Future Grid Operations

Authors: David Schooley, ComEd, Yi Hu and Damir Novosel, Quanta Technology, USA

In the past twenty years, ComEd has witnessed major changes not only in its own territory but across the entire Eastern Interconnection. The noticeable changes include the increasingly high penetration of inverter-based resources, resulting from public-policy driven increases in the amount of wind and solar generation and the retirement of multiple coal generation plants. The pace of change is anticipated to quicken in the foreseeable future as many states in the US are moving towards a low or no carbon electrical power supply future. As a result, the operating conditions of the current ComEd have been changed and will continue to change. The operating conditions of the future ComEd system grid will be very different from today’s operating conditions. Reduced system inertia, decreased short circuit current, and faster system dynamic response are already happening, and will become more pronounced in a future power grid.
Many on-going initiatives are aimed to continue improving the reliability performance metrics through modernize ComEd’s distribution systems. ComEd envisioned that it could become a network service provider/integrator (NSPI) for distribution systems to provide a variety of services to existing and new types of customers and partners as part of its Utility of the Future roadmap. Realizing this vision would require a much greater visibility and controllability of its T&D systems.

In response to and in anticipation of continued operation condition changes in the ComEd and the Eastern Interconnection control areas, ComEd has embarked on many initiatives to modernize its transmission grid and distribution networks. Adopting new technologies, such as synchronized measurement technology, to improve and enhance ComEd operations is one of these major initiatives. For example, as inverter-based generation does not have inertia, any changes in the system happened faster and require better visibility across the interconnected grid. Synchronized measurements, with their 1 microsecond accuracy allow for much faster tracking of the events than conventional EMS/SCADA and through GPS synchronization visibility of the events across the grid. Furthermore, data sharing across neighboring utilities is important to address fast grid events. Exelon companies (ComEd, BGE, PECO, and PEPCO Holdings) are working together to both build a robust, productized system and to allow for data sharing across the grid.

Deploying synchrophasor technology for future grid operations: ComEd’s early involvement with the synchrophasor technology started after the 2003 blackout through participation in the Eastern Interconnection Phasor Project (EIPP) and later in the North American Synchro-Phasor Initiative (NASPI) activities.
ComEd’s experience with synchrophasor technology deployment and application stems back to 2009, when it first joined a PJM-led project co-funded by the U.S. Department of Energy through the Smart Grid Investment Grant (SGIG).  Through the PJM synchrophasor project and ComEd’s own follow-on efforts, ComEd has acquired a great deal of knowledge and proficiency with the deployment of synchrophasor systems and the use of the valuable information they provide.  For example, synchrophasor data greatly improves the accuracy and speed of the post-event analysis and the information has been found valuable in identifying events and issues that were not visible before with other tools/systems. It is also part of the data sets required for meeting NERC PRC-002-2 requirements.
This positive experience, coupled with the knowledge gained through active participation at NASPI, NERC, and IEEE activities, has shown that synchrophasor technology could play a major role in economical and reliable operation of the future ComEd transmission grid and distribution systems.

For offline applications, in addition to post-event analysis, model validation ensures that system operating safety margins are based on accurate generator, load, and system models that allow for the maximum capacity of the transmission grid to be utilized.
Real-time synchrophasor applications that enhance ComEd’s transmission grid operations include: Linear State Estimator for real-time data validation, and Online Model Monitoring and Adjustment for keeping the models updated in real-time; Fast System Condition Assessment after forced outages; Voltage Stability Analysis, Island Operation and System Restoration Support; Wide-Area Voltage/VAR Control; and Wide-Area System Integrity Protection Schemes.
Synchrophasor could also be applied to improve ComEd’s distribution operations in five functional areas through (1) improve/enhance classical and modern distribution functions with the use of synchrophasors, and (2) implement advanced and future distribution functions that depend on synchrophasors. The five functional areas are: distribution system operations, distributed energy resource (DER) integration, system monitoring, protection, automation and control, asset management and reliability, and planning and analysis.
However, integrating synchrophasors into ComEd’s day-to-day operations requires sizable investment to deploy the technology in ComEd’s transmission grid and distribution systems. For example, to achieve a full observability of the transmission grid and increase the number of practical applications, it will be needed to substantially increase the number of PMUs, improve data quality, increase redundancy, and ensure the deployed synchrophasor systems to comply with NERC CIP standards.

ComEd synchrophasor technology deployment roadmap:  Given the sizable investment required and many activities involved for transitioning synchrophasor from pilot to production use, a synchrophasor deployment roadmap was developed. The objective of the roadmap is to provide a high-level guidance for the investment and activities required for successfully deploying synchrophasor technology within ComEd’s service territory on its transmission grid and distribution system.

The synchrophasor roadmap development has considered the following interdependencies (see Figure 1)
1. The synchrophasor applications selected for deployment are highly dependent on key business needs and drivers that various stakeholders desired to address, and
2. The required supporting infrastructure (i.e., PMUs, PDCs, Phasor Gateways, data storage systems, communication channels and networks, analytical applications, system design choice) are dependent on the synchrophasor applications being selected.

Although realizing the full benefits would require the development of proper operation processes and procedures and the necessary user training, this generally is done by operation units and not part of a high-level deployment roadmap.
With the above considerations, the roadmap development has gone through the 1) information collection and analysis, 2) identification of ComEd key business needs and drivers and the applications that address them, 3) business case development, 4) deployment readiness assessment, and 5) synchrophasor application deployment prioritization processes.
The business case development was a cost-benefit analysis based on potential benefits and benchmark cost estimations. The results of the analysis are positive that in a ten-year period the benefits would outweigh the deployment cost.
A qualitative deployment readiness assessment was also performed for synchrophasor deployment in transmission grid and distribution system. The assessment takes into consideration of several areas including required PMU installation, available communication infrastructure, CIP Compliance requirements, application maturity, product availability, and deployment efforts required.

Considering that there are hundreds of “synchrophasor-ready” devices in ComEd’s transmission grid and an extensive optical telecommunication network already installed on the ComEd transmission system, there is already a high level of readiness for a more wide-scale production use of synchrophasor technology by ComEd. However, considering the difference in CIP compliance requirements for synchrophasor applications deployed for non-real-time operation use and for real-time operation use, the following deployment roadmap for transmission operations has been selected, which includes:
1. Enable PMU functions in digital relays at all 345kv and select 138kV transmission substations
2. Deploy a real-time synchrophasor system, i.e. a wide-area situational awareness system (WASAS), for real-time operation use at the transmission grid control center
3. Deploy a non-real-time synchrophasor system, i.e. a wide-area measurement/monitoring system (WAMS), for all non-real-time operation use by authorized ComEd users
4. Set up communication network to support secure and quality of service assured data transfer between substations and the real-time and non-real-time systems

As shown in Figure 2, the deployment involves three stages, the jump-start stage (year 1 and 2), the full deployment stage (year 3 through 5), and the integration and enhancement stage (beyond year 5). Making production WAMS operation ready is the focus of the jump-start stage and deploying a production WASAS will be the focus of the full deployment stage.
Taking into consideration that for distribution systems, CIP compliance is not a requirement, synchrophasor has not been piloted before, generally communication infrastructure for synchrophasor is not readily available, and the diversity of different types of distribution circuits, the deployment roadmap selected for distribution system is as shown in Figure 3.
In the same five-year period, the deployment in distribution systems will go through three stages: the piloting stage, the initial deployment stage, and the expanded deployment stage with continued enhancement/evolvement beyond the five-year period.

ComEd transmission grid synchrophasor systems deployment: ComEd first implemented PMUs as a pilot project in conjunction with the DOE ARRA along with PJM and other PJM member companies. In 2015, ComEd decided to expand the initial pilot project into a full-scale deployment. Using the roadmap described earlier to justify the costs and to provide the overall architecture, ComEd initiated projects to build the IT infrastructure necessary for a large PMU deployment while also budgeting for 5 years of PMU deployments. The planning process for transmission projects has been updated to include PMUs as part of other reinforcement projects.

To account for future cyber security requirements and to separate real-time functions from engineering applications, ComEd’s IT infrastructure is divided into two subsystems as shown in Figure 4. The first, WASAS, or the Wide Area Situational Awareness System, is intended to support real-time control-room applications as well as secure, CIP compliant communications to the EMS and PJM. The WASAS system is upgradable to meet future CIP requirements as they arise. All synchrophasor data coming from transmission PMUs is sent directly to WASAS.
The second system, WAMS, or the Wide Area Measurement System, supports engineering and synchrophasor data analytics applications and long-term storage. Data and applications used for off-line and after-the-fact analysis have different security requirements than do real-time data and tools, so it makes sense to separate these functions to maintain security requirements where necessary and ease-of-use and accessibility where possible. Distribution synchrophasors have different cyber security requirements than transmission PMUs for data access and security in the field, so the distribution PMUs send their data to WAMS. WASAS forwards transmission PMU data to WAMS for use in engineering analysis and for long term storage.

Redundancy was one of the goals for the production PMU data system. Redundancy is achieved through local redundancy of the PDCs and by utilizing duplicate data centers. The PDC servers consist of 2-machine clusters of one active server and a hot inactive server. If the active server fails or needs to be taken out of service for maintenance, then the cluster will fail over to the inactive server within a few seconds.
To further enhance redundancy, the WASAS and WAMS systems are duplicated at two geographically separated data centers. The installations at the data centers are identical for the production components of the WASAS and WAMS systems. Each of the two-data center WASAS installations will eventually send data to PJM, providing duplicate streams and redundancy at the ISO level. Auxiliary components, such as the development and quality assurance environments, do not require the same level of redundancy and are only implemented at a single data center. An example of how the data-center redundancy works with the real-time RTDMS application is shown in Figure 5.

OSISoft’s PI Historian is the long-term data storage solution chosen for the synchrophasor data. The PI enables high-availability access to data with support for multiple servers. The high availability function of the PI automatically allows for selection of an active server without a need for user intervention.

ComEd utilizes substation PDCs as part of its standard design for transmission and PMU deployment. Each substation PDC sends data to both data centers at the same time. If one data center loses connectivity, the PMU data will still be accessible at the other data center. As with the data center design, the substation PMU/PDC implementation is designed to be upgradable as CIP requirements change in the future. The goal in the design is to not have to remove any equipment as requirements change, only to add equipment such as card readers and firewalls as necessary. The substation configuration makes a substation “PMU ready,” which enables future PMUs to be added for a low cost.
The WAMS is in use today for engineering and event analysis. The WASAS is implemented but not entirely separated from WAMS, with the real-time tools in use for evaluation of capabilities such as oscillation detection and analysis. The WAMS and WASAS design has allowed ComEd to increase its PMU deployment from 12 PMUs at 7 substations to 44 PMUs at over 20 substations in less than two years. ComEd’s goal is to install 50 PMUs per year in a combination of planned projects and as additions to other reinforcements. Following WASAS applications are planned to be deployed in next 3 years: linear state estimation, on-line model calibration, fast contingency analysis, voltage stability monitoring, and islanding and system restoration support.

ComEd distribution networks synchrophasor technology deployment:  ComEd has initiated a pilot program to begin evaluating distribution synchrophasor application for selecting high-value applications for pilot implementation and to support existing projects such as microgrids and voltage optimization. The roadmap effort identified a large number of possible distribution synchrophasor applications. The distribution organization has gone through a rigorous evaluation process and selected a subset of these applications for pilot evaluation in the near future. Some of these applications, such as verification of dynamic load models will incorporate both transmission and distribution PMU data. The pilot program is already providing opportunities for staff to gain experience working with large data sets.
The distribution pilot is currently sharing the infrastructure and applications put in place to support the transmission synchrophasor deployment project, but the distribution synchrophasor systems will eventually migrate to use their own PDCs and applications. The distribution PMU data resides within the WAMS area of ComEd’s transmission grid synchrophasor system where it is available for analysis with fewer cyber security restrictions.

Deployment activities in other Exelon sister companies: Given the success of ComEd’s synchrophasor deployment, the next step is for other Exelon companies in the mid-Atlantic region (BGE, PECO, and PEPCO Holdings) to build a similar system. Exelon is currently consolidating its EMS at two data centers for these mid-Atlantic utilities and these data centers will also support the synchrophasor system infrastructure. Data sharing between the mid-Atlantic utilities and ComEd will be incorporated into the final design. ComEd will take advantage of the lessons learned in data sharing with its sister utilities as a path toward implementing similar data sharing with its neighbors who are not members of PJM.
While the IT systems will be identical to what was installed to support PMUs at ComEd, the geographical separation between the utilities in the mid-Atlantic will pose some challenges. Engineers in the mid-Atlantic utilities will be able to access the ComEd data and applications in preparation for the eventual deployment of the mid-Atlantic synchrophasor systems for use by these utilities.

Concluding remarks:  As the energy supply is quickly moving towards a low/no carbon future, it is expected that new challenges in operating such future power system will continue to emerge. Synchronized measurement technology should be an integral part of overall solution in addressing these challenges.
It is clear that integrating synchrophasors into real-time and non-real-time operations is a major undertaking that requires substantial investment and a careful planning to ensure a successful deployment for realizing the intended benefits.
ComEd’s roadmap development exercise and the deployment experience so far has indicated that it is extremely important to start the planning of such deployment for future grid operation as early as possible, as it will take at least 4-5 years before the deployed systems become operationally ready.  



David C. Schooley is a Principal Engineer in the Transmission Analysis department at ComEd (Exelon Utilities.) His Bachelor of Science and Master of Science degrees in Electrical Engineering are from Oklahoma State University, and his Ph.D. degree in Electrical Engineering is from the Georgia Institute of Technology, where he studied the integration of large amounts of wind and solar energy into the generation mix. His job responsibilities have included work in both transmission operations and planning. David is the Exelon lead for PMU deployment and applications.

Yi Hu Director of Quanta Technology, has over 30 years’ experience and is a leading expert in the synchronized measurement based Wide-Area Monitoring, Protection, automation, and Control systems deployment and applications.
Dr. Hu is an IEEE fellow who had contributed to the development of several synchronized measurement technology related IEEE standards. He currently chairs an IEEE working group to develop an “IEEE Guide for Engineering, Implementation, and Management of System Integrity Protection Schemes.” Yi holds 13 U.S. patents, and has published multiple technical papers and articles in Refereed Journals and Conference Proceedings.

Dr. Damir Novosel is president and founder of Quanta Technology, a subsidiary of Quanta Services, a Fortune 500 company.  Previously, he was vice president of ABB Automation Products and president of KEMA T&D US. Dr. Novosel is member of US National Academy of Engineers and served as IEEE PES President and Vice President of Technical Activities. He is also a member of the CIGRE US National Committee and received the CIGRE Attwood Associate award.  Damir holds 17 US and international patents, published over 140 articles, and contributed to 5 books.