The Guru

Interview with PACWorld guru Herb Falk

PAC World: When and where were you born?

H.F.: I was born in Greensboro, NC, USA in August 1956

PAC World: Where did you go to school?

H.F.: I was a boarding student at Deerfield Academy in Deerfield, MA from 10th-12th grade from 1972-1974.

PAC World: Did you have any specific interests while you were in school?

H.F.: Science and computer programming. Deerfield had a teletype connection to the Dartmouth time-sharing system, and we could write Basic programs. In one of my classes, we needed to write a game, and I chose to write a scrabble like game – RSVP, where a person played against the computer.

PAC World: Who do you think is the person that had the most influence on you when you were growing up?

H.F.: When I was younger, my father, and his Father, provided guidance about the need for education and work ethic. My grandfather Falk instilled the knowledge of civil rights and marched with Martin Luther King. My grandfather on my mother’s side (Nathan) was always tinkering with mechanical things as he owned a boat and that is where I gained the curiosity about how things work and how to design them. Nathan was also a prohibition rumrunner through him and my mother I inherited the predilection for jokes and critical thinking.

PAC World: Do you remember anything from your childhood that contributed to you becoming an engineer?

H.F.: Due to Nathan, I had a toolbox at a very young age and tried to take apart things which was mostly successful and then put them back together which typically failed. From there, my parents knew I was interested in science, so they bought me a chemistry set as well as microscopes and a small telescope through which I became interested in Astronomy.

PAC World: Why did you decide to continue your education?

H.F.: Education is the one thing that people can’t take away from you and my parents emphasized that getting a college degree was almost mandatory to succeed. They also wanted me to become a doctor or a lawyer.

PAC World: How did you choose the university to go to?

H.F.: My Deerfield college advisor suggested five colleges to which to apply and visit. The two colleges that were my primary choices were Duke and Northwestern University. Both had something called early acceptance which allowed me to apply to but hear about acceptance in early March. I heard from Northwestern but not from Duke. Thus, I took the bird-in-the-hand and accepted Northwestern. The acceptance letter from Duke arrived one week later.

PAC World: Did you study electric power systems while in college?

H.F.: The simple answer is no, except for one basic course. With my parent’s push for me to become a doctor, I was EE-Biomedical Engineering in undergraduate. EE-Biomedical provided extensive education in system analysis and mathematical modeling. Due to my early completion of the undergraduate degree, I decided to continue at Northwestern for a master’s degree in EE primarily in Communication Theory and Systems as well as Quantum Physics (I know it is a Quarky thing).

PAC World: Did you have any other interests while in university?

H.F.: I was very active in Athletics and tried to walk-on for the Northwestern Football team and was told I was too small (good thing). I had made a promise to play rugby if I didn’t make the team and had a great 12-years of playing rugby as a tight-head prop at Northwestern and other teams/venues around the world. There was a very active intramural sport program at Northwestern, and I enjoyed floor hockey as well as 12-inch fast-pitch softball.
Outside of classes Northwestern offered a wide range of research opportunities, and I had a wide range of interests but couldn’t do everything I wanted to research. But I did get to help design the hardware and software for one of the first sip-and-suck computer interfaces that could allow quadriplegics type and control their wheelchairs. I also got to do base research related to Reagan’s Star Wars initiative as well as hydrogen cells.
Those who know me, know that I like a good alcoholic drink, but the city in which Northwestern was located, Evanston, was dry (no alcohol for sale unless you bought a meal).
We spent a lot of time in Chicago as well as trying to convince Evanston restaurants of the definition of a meal. We were able to convince one that a hot-dog was a meal and that became our favorite watering hole.
I enjoyed city politics (trying to get alcohol allowed) as well as politics involving the Vietnam war and the draft

PAC World: What was your first job?

H.F.: I went to work for Westinghouse in what they called a graduate rotation program. My first rotation was to Westinghouse Numa Logic that designed programmable controllers and I/O. I then was rotated to a defense-oriented activity called the Advance Technology Lab and I am still not allowed to discuss what went on in that facility. I couldn’t imagine working a lifetime and not being able to tell anybody what I did, so I rotated back to Numalogic where I was given all types of interesting projects. My first project lead was what was called a Mini-Loader. (see page 60).
The purpose of the Mini-Loader was to replicate the functionality of a CRT based unit that had 10-man years of software development in 8085 assembly language. I was given one year to develop all of the hardware (an 8085 in a handheld would have caused too much heat). So, the CPU chosen was a CMOS 1802 and we got 90% of the 8085 code to execute in the Mini-Loader through the development of an 8085 interpreter in 1802 code.
There were other projects that required unique solutions, and Westinghouse gave me most of them.

PAC World: What What made you join SISCO?

H.F.: Westinghouse decided to terminate the department known as Numa-Logic and move manufacturing and key personnel to Pittsburgh. Very few of the NumaLoids took the move and thankfully, Jon Martin (product planning manager at Numa-Logic) had seen the writing on the wall and had founded SISCO prior to the Numa-Logic move. He offered me, Ralph Machiewicz, and Mike Eick a method to join SISCO as minority owners. Who wouldn’t take such an offer.

PAC World: What made you interested in communications?

H.F.: At Northwestern, to earn some money, I worked in a mini-computer lab consisting of Digital PDP-8 machines that all had to be programmed to talk to each other in assembly code. We actually designed our own protocols there and the rest is as you say is history as I have been designing communication systems and protocols since.
One lesson learned from Northwestern in communication theory was that the least amount of data can convey the most information. As an example, when you go and flip the light switch and the lights come on, that tells you that your fuse/breaker is OK and that there is power to your building. Obviously, more information is needed if the lights don’t come on (e.g. for diagnostic purposes), but at a meta-level the assertion is true. Why do I raise this, I have had projects where all data is instrumented and that creates a torrent of data that may or may not be converted into information. With the advent of AI and LLM, I fear that the trend of not analyzing data versus informational needs will accelerate.

PAC World: What was the most challenging task at the beginning?

H.F: There could be a book written about the development, and relationship, of MAP/TOP, UCA, UCA2, IEEE TR 1550, and IEC 61850.
In the early 1980’s General Motors and Boeing recognized that they needed a set of common, and “standardized,” protocols for the integration of the processes on the manufacturing floor as well as the backend business processes. The result was the development of the set of standards that became known as the Manufacturing Automation Protocol (MAP) and Technical Office Protocol (TOP) protocols. It is the MAP initiative from which the Manufacturing Automation Protocol (MMS) was developed. Since I worked for Numa-Logic, I was assigned to assist in the development of the precursors of MMS as well as the standardization of MMS itself.
MMS was one of several protocols that were the foundation of the US Government Open Systems Interconnection Profile (GOSIP). GOSIP did not address the specific requirements of the utility industries (e.g. water, gas, and electric) and that was the foundation of UCA1. Since I had been involved in MMS and GOSIP, the Electric Power Research Institute had Author Anderson Consulting hired me as a subcontractor to assist in mapping GOSIP to UCA. There were many entrenched Internet Engineering Task Force Protocols (IETF) that were not in GOSIP, and the market used the IETF standards instead of the GOSIP specified ones. The surviving protocol was MMS and that became the focus of UCA2 to provide more semantic interoperability.
Bill Blair, of EPRI, hired a small group of industry experts, including me, and consultants, to help develop UCA2 and there were two basic issues: semantic interoperability as well as “high-speed” automation. Semantic interoperability was determined to be more than the high-level objects of MMS and this resulted in the definition of “bricks” that could be conveyed by MMS objects. The need for high-speed automation caused an EPRI funded experiment, at SISCO, to determine which was better: Profibus or Ethernet. That experiment provided the foundation for GOOSE as well as many of the industrial automation protocols today.
UCA2 was not a standard and thus IEEE was used to start the standardization process as IEEE TR 1550. The reason it is a TR was an agreement between IEC TC57 and IEEE to attempt to produce a single global standard which became IEC 61850 as we know it today.

PAC World: What is the most satisfying moment during that time?

H.F.: There was a meeting in Chicago of 7 people where a use case for distributing trip information amongst 100 IEDs within 4 msec. The analysis of the use case determined that TCP/IP based protocols could not. George Schimmel (Tamarack) and I worked on the concept of using Ethernet for multicast messages. This became the UCA GOOSE (e.g. GSSE) and eventually the foundation for the IEC 61850 GOOSE as we know it today. There was much scepticism of using a network versus hardwired for protection signals and Kay Clinard (UCA Users Group) provided a beanie baby award known as the “Lucy” GOOSEy award to manufacturers that proved interoperability and that GOOSE was faster than hardwired. (see page 60).

PAC World: You were also actively involved in the development of routable GOOSE and sampled values communications. What is their impact on PAC systems?

H.F.: I personally believe that this opens up, and simplifies the deployment, of using GOOSE outside of the substations. Many IT staffs refuse to support and maintain a flat L2 network across multiple geographical sites and that is what is created when using L2TP/VPNs to tunnel L2 GOOSE. Additionally, as cyber security moves to looking at Zero Trust frameworks, the L2TP/VPN combination does not provide that capability. Therefore, I believe R-GOOSE is the mechanism that should be used for outside the substation. Furthermore, R-SV was designed to secure the delivery of synchrophasor information and there still is no standard profile for C37.118.2 that includes the needed cyber security for Zero Trust. Therefore, as synchrophasors become more prevalent in operational decisions, there will probably need to be a migration to R-SV.
Now for the controversial opinion. I believe that R-GOOSE will eventually replace L2 GOOSE within a substation. There are network maintenance reasons to do this, but there is a use case to think about: If you need to send a DataSet via GOOSE inter and intra substation would you do this using two GOOSE control blocks (one for L2 and the other for R-GOOSE).
I believe that in the near future, the answer will be one control block using R-GOOSE. Once this starts to occur, there is no real performance difference between L2 and R- GOOSE performance. I suggest people think about this.

PAC World: IEC 61850 has been around for 20 years. It is clear that digital substations offer significant advantages. Why do you think it is taking so long for this technology to be applied everywhere?

H.F.: As with everything else, change is hard to accept. We had the same issue moving from electromechanical to digital protection. And we are in the transition to digital substations, and it requires changes to utility processes and procedures as well as intensive training. Additionally, the skill sets of the protection engineers, and testers, need to include a knowledge of networking.

PAC World: What made you interested in cyber security?

H.F.: Back in the early 1990’s there was a nuclear plant that sent its emergency event tones via radio “in the clear.” A HAM radio operator captured the signal and replayed it on the same frequency every 30 minutes for 3 days before they found the operator and stopped the replay. The nuclear plant had approximately 144 unannounced nuclear events that needed documentation and responses to. The overhead cost to do this was large and could have been prevented with a couple of $1000 appliances. That event perked my interest, and I wrote the first security for UCA and ICCPs.

PAC World: What do you think are the biggest cyber security threats and how should we deal with them?

H.F.: There are several different levels of threats evolving, I think the most egregious issue are utilities working to satisfy regulatory requirements (e.g. US NERC CIP) instead of applying best security practices. As an example, US NERC CIP specifies if the utility does not use an externally routable protocol to ingress/egress a substation, certain requirements need not be met. This lends itself to utilities routing/tunneling serial protocols except for entry/exit into a substation. Incurring ALL of the security concerns of a routable protocol but because of the last “50 feet” they are not subject to some cyber requirements. Utilities should be focused on best security practices and NOT just regulatory requirements. It has been my experience that best security practices typically meet or exceed regulatory requirements.
The new buzz word in cyber is post-quantum. The IT world is embracing quantum resistant cypher suites without regards to the impact on OT systems where bandwidth and computational resources may be limited. The next “IT/OT” battle will probably be in this arena.
The basic cyber threats are the same, but the technologies to launch those threats is evolving. I believe AI may provide ways to elevate some threats/information disclosure mechanisms to become easier.

PAC World: Virtualization is a hot topic today. What do you think about its impact on future PAC systems?

H.F.: There is no doubt that virtualization of protection is on the horizon as there is a lot of effort going into the promotion of this technology. I am personally involved in base research regarding the critical metrics of virtualization which will be required in order to have protection engineers considering the technology. As with electromechanical to digital protection transition, change is going to be slow, and education and proof of capability will be required.

PAC World: What do you think is most important for a protection engineer’s development today?

H.F.: A protection engineer will need to understand networking technologies and cyber security impacts not just protection settings. Training and education will be needed.

PAC World: You have been actively involved in the IEEE, IEC and CIGRE. What do you think is their impact on our industry?

H.F.: I embraced working with all the organizations and they have been good for the industry in general. With that said, the cost of acquiring IEC standards, especially the IEC 61850 series is becoming an impediment to smaller companies embracing the technology. The other issue is there are work items that overlap, and this causes constrained resources to be stretched (e.g. people). I personally have to determine which group is more beneficial for the industry to work within on a given topic to choose where I spend my time.

PAC World: After 33 years with SISCO you started your own business. Why?

H.F.: There is always a backstory behind the story. With that said, there was no way to monetize my knowledge of IEC 61850, CIM, and cybersecurity as long as I worked at SISCO. In general, companies are very concerned about hiring consultants from vendors. I really wanted to be able to make a difference in cyber security, so I left.

PAC World: What do you believe is the best way to share your knowledge and experience with the new generation of protection engineers?

H.F.: I have always been a proponent of trying to assist universities with their education. I would love to be able to do more, in this area but finances often get in the way. Education, training, and seminars are best. I will probably be adding short videos to my YouTube channel in the future.

PAC World: What is the advice that you would give when you are in front of an audience of young people?

H.F.: Believe in yourself and do not be afraid to ask questions if you don’t know something. Asking questions is a mechanism to learn and not a weakness.

PAC World: What do you consider the biggest challenge in your professional career?

H.F.: To stay focused on the task at hand is often difficult.

PAC World: What do you consider your biggest professional accomplishment?

H.F.: I don’t believe that there is a single “biggest” accomplishment as there are different metrics that I could use to choose different accomplishments. However, the research regarding Ethernet versus Profibus moved the electric utilities and industrial controls in the correct direction. That probably had the widest impact. The development of the foundational GOOSE/SV multicast has probably had the most impact on the electric utility industry.

PAC World: You have received different awards throughout your career. Is there a specific one which is the most important to you?

H.F.: First, I don’t do this work for awards. The one that is most special is becoming an IEEE Fellow as my peers thought I was worthy.

PAC World: What do you think is your most important personal achievement?

H.F.: With the help of my wife, Lydia, raising my children with a moral compass and a thirst for success and knowledge.

PAC World: You have traveled to many countries. Do you have a favorite place to visit?

H.F.: My favorite city to visit is Brussels. It is multi-cultural, has great food, and the people are friendly.

PAC World: Do you have a hobby or something that you like to do when you are not working?

H.F.: I used to be into model rocketry and remote-controlled aircraft. I still have a couple of rockets to build in my basement and hope to get to fly them with my grandchildren. I just can’t get into drones as a hobby. I read a lot and am a proficient producer of divots at golf (means I am not very good at golf).

PAC World: How do you mix your professional and your personal life?

H.F.: My wife, Lydia, attempts to remind me to have balance and occasionally forces me to stop work. On many meetings now, besides my dog needing relief, we have date nights that stop me from overworking.

PAC World: Do you ever consider retiring?

H.F.: As long as I have interesting work/projects that allow me to learn, probably not.

PAC World: What is your favorite music?

H.F.: 70’s Rock and Roll. Favorite songs to drive to is from Lynyrd Skinner: Freebird or Allioto, Hanes, and Jerimiah: Lake Shore Drive.

PAC World: What is your favorite form of entertainment?

H.F.: Reading books.

PAC World: Do you have any favorite food?

H.F.: Being from North Carolina, I absolutely adore good, shredded pork barbeque (or most smoked meats.)

PAC World: Do you have a motto?

H.F.: I borrow from Mark Twain: “Learning is what you have left after you forget all that you were taught.”

Biography:

Herbert Falk graduated with BSEE and MSEE degrees from Northwestern University. He spent most of his career as project manager and system architect at SISCO and is currently Managing Director of Outside the Box Consulting Services, LLC.
Herbert is IEEE Fellow and an active member of the IEEE Power Systems Communication Committee (PSCC), IEEE Power System Relaying Committee (PSRC), and the DNP Security Task Force.
He is the Vice President of Testing for the UCA International User Group (UCAIug) and is responsible for managing the Conformance and Interoperability Test Programs within UCAIug. He has managed IEC 61850 Interoperability Tests bi-annually since 2011.
He is an editor of IEC 61850 and the US Technical Advisory Group (TAG) lead on Cyber Security to IEC TC57 WG15.