Grid Security
Securing power grid systems during a live modernization project
100%
NERC CIP compliance maintained
2,400+
OT assets monitored
Zero
Operational disruptions
A regional utility with 1.4 million customers was two years into a grid modernization program when its internal security team raised concerns about the pace of connectivity being added to operational technology systems. Smart meters, distribution automation equipment, and grid sensors were being connected to communication networks as part of the modernization — each one a potential entry point into systems that controlled physical infrastructure. The utility's NERC CIP compliance program covered its highest-voltage transmission assets but left significant coverage gaps in the distribution systems where most of the new connectivity was being added.
The outcome
We built a security program that moved at the same pace as the modernization, adding visibility and controls to each system as it was connected rather than trying to retrofit security across all systems at once.
NERC CIP compliance is not the same as being secure
The utility's NERC CIP program was well-run. Its high-voltage transmission assets — the substations and transmission equipment that NERC CIP primarily covers — were inventoried, assessed, and monitored. The compliance team maintained the documentation required for audit and had a track record of clean audit outcomes. The security problem was in the systems that NERC CIP doesn't comprehensively address: the distribution automation equipment, the smart meter communication networks, and the SCADA systems managing distribution-level switching. These systems were being connected to IP networks as part of the modernization program. Most of them had no built-in security logging. Many were running operating systems that hadn't received security updates in years because the vendors hadn't provided updates and the utility hadn't replaced equipment that was still functionally operational. The attack surface was growing faster than the security program's coverage.
A visibility-first approach to active infrastructure
The first principle we established with the utility's security and operations teams was that we would add visibility before we added controls. In an operational technology environment, a security control that fails in an unplanned way can have consequences that range from a service interruption to a physical safety incident. The risk of a poorly-deployed security control was, in some configurations, higher than the risk from the security gap the control was intended to address. Passive network monitoring — listening to network traffic without inserting any device into the communication path — was the starting point for every system. We deployed passive sensors on the communication networks serving the distribution automation equipment, the smart meter head-end systems, and the SCADA historian network. The sensors recorded all communication patterns without affecting them. Within eight weeks, we had a comprehensive traffic baseline for each network segment and a complete device inventory derived from the communication patterns — more complete than the utility's existing asset inventory, which had been built manually and was approximately eighteen months out of date.
Adding controls without stopping the grid
Security controls were introduced to each system in a sequence designed to minimize operational risk. The first controls added to each network segment were detection-only: rules that generated alerts when communication patterns deviated from the baseline, but didn't block any traffic. Detection-only controls introduced no operational risk because they couldn't affect the network traffic they monitored. The maintenance team evaluated each alert category during a six-week period and confirmed which deviations represented real anomalies and which represented normal operational variation that the baseline hadn't captured. Only after the detection rules were calibrated to an acceptable false positive rate did we introduce blocking controls — and even then, blocking controls were introduced one system at a time, with operations teams present for the initial deployment period on each system. This approach was slower than deploying controls across all systems simultaneously would have been. It was the approach that the operations team would accept, and the operations team's acceptance was the prerequisite for any security improvement in an environment where their cooperation was required for the controls to be correctly scoped.
Evidence packages that reflect how the systems actually work
A recurring problem in utility NERC CIP compliance programs is that the documentation required for audit doesn't match how the systems actually operate. Access control lists in the audit documentation don't match the access control lists deployed on the equipment. Network diagrams don't include systems that were added after the diagram was last updated. Asset inventories don't include equipment installed during maintenance activities that didn't go through the formal change management process. Auditors see clean documentation; the actual systems have gaps. We built the compliance documentation program from the monitored data rather than from manual documentation processes. The network diagrams were generated from the passive monitoring data: every device that communicated on a monitored network segment appeared in the diagram automatically. The asset inventory was maintained by the monitoring system: a new device appearing on a network segment triggered a review and, if confirmed as an authorized asset, was added to the inventory. The access control lists were validated quarterly by comparing the rules deployed on equipment against the rules recorded in the compliance documentation — discrepancies triggered a reconciliation before the next audit cycle.
Where the program stands today
Thirty months after the program began, the utility's security coverage includes 2,400 operational technology assets across its transmission and distribution infrastructure — significantly more than were in scope when the program started, because the modernization program has continued to connect additional equipment. The NERC CIP compliance program has received clean audit outcomes in each of the two audit cycles completed since the program began. More practically, the security operations team has detected and responded to eleven genuine security events in the monitored infrastructure during this period — seven of which were contractors accessing systems outside their authorized windows, three of which were misconfigured devices attempting to communicate with addresses outside their expected scope, and one of which was a confirmed attempt to access the SCADA historian from an external IP address that was blocked automatically by the network controls and reported to law enforcement. None of these events resulted in an operational disruption.
Facing a similar infrastructure challenge?
We're happy to have a technical conversation about your specific environment — no commitment required.