Protecting Generating Networks with Unidirectional Security Gateways

There was a day when the most common question heard while demonstrating a Unidirectional Security Gateway at an electric-sector trade show was "What's that?” Today, most electric-sector security practitioners are aware of unidirectional gateway technology and its advantages over firewalls in one circumstance or another, but many are still unclear as to what a unidirectionally-protected "big picture” network looks like. In this article, we describe the network architecture we see deployed routinely to protect power plant control networks. A simplified version of the network is illustrated below.

The design strategy behind this architecture is simple - replace at least one layer of firewalls in the layered, defense-in-depth network with unidirectional gateways, thus protecting our most important control system components from network attacks originating on less-trusted networks, such as corporate networks and the Internet.

Safe IT/OT Integration

The vast majority of power plants choose the IT/OT firewall layer as the layer to replace with stronger unidirectional protections. Cross-zone communications at this layer serve many functions. In the discussions below, each IT/OT integration function is considered independently. In practice, many or all of these functions are combined into a single set of Unidirectional Security Gateway equipment. Let's look at each of the functions in turn.

Primary IT/OT Interface: The primary IT/OT interface is used to monitor real-time operations continuously for purposes such as predictive maintenance, operations optimization and widespread awareness of key performance indicators. The majority of unidirectionally-protected power plants deploy Unidirectional Security Gateways at this interface, but newer installations also use the Waterfall FLIP at this interface, and it is a FLIP that is illustrated in high-level diagram above. For readers unfamiliar with gateways or the FLIP:

• A Unidirectional Security Gateway is hardware that is physically able to transmit information in only one direction, coupled with software that makes copies of industrial servers, most often process historian databases, or various types of OPC servers. Users and applications on the corporate network needing access to real-time data can query the replicas for that data.
• A Waterfall FLIP is a kind of unidirectional gateway whose orientation can reverse on a schedule. A FLIP can physically send information in only one direction at a time, and makes copies of industrial servers just like the gateways do.

At unidirectional-gateway sites, security updates are communicated to plant networks on removable media - most commonly write-once CD's. At FLIP sites, the FLIP is configured to pull such updates automatically, on a schedule.

Vendor Monitoring: The majority of coal, gas and hydro plants have remote connections to their turbine vendors for vendor monitoring and diagnostics, and many have similar connections to other vendors as well. These vendors monitor physical equipment or control-system components continuously, and adjust system parameters from time to time. Unidirectionally-protected plants meet the monitoring need with server replication; the vendors monitor the replica servers. Most unidirectionally-protected plants meet the "occasional adjustment" need with remote screen view technology. Remote screen view makes a real-time feed of screen images available to vendor support personnel, while site personnel operate control system equipment. The vendors generally advise plant personnel over the phone throughout these adjustments.

Generation Dispatch Center: Most power plants need to report power production and other parameters to a generation dispatch center continuously. A minority of plants has production schedules set far into the future; such plants can deploy unidirectional gateways to replicate ICCP servers to permit continuous monitoring by dispatch centers. The majority of plants not only report their status continuously to a dispatch center, but receive second-by-second updates of power production set points from that center as well.

Some such plants lease serial lines to exchange ICCP information with dispatch centers, but a more secure solution is a pair of "inbound/outbound" unidirectional gateways replicating ICCP servers. Inbound/outbound gateways are two independent unidirectional gateway deployments, each replicating an ICCP server in one direction. The inbound gateway polls the dispatch center’s ICCP server and emulates that server to plant systems. The outbound gateway polls the plant ICCP server and emulates that server to the dispatch center EMS. These two channels are generally deployed on different sub-networks and are unable to communicate with each other, though both are able to communicate with control center and plant ICCP servers.

Cloud Providers: Communications with cloud services is similar to communications through the primary IT/OT interface, and is called out separately in the diagram because of the high degree of interest in this emerging field for power plants. Communications with cloud service providers may be via unidirectional gateways, or if occasional updates from cloud applications are necessary, may be via the FLIP as well.

Safety and Protection Systems

In some cases, owners and operators decide they wish to continue to use firewalls at the IT/OT interface, but still need safety instrumented systems and protective relays protected unidirectionally. Such plants always deploy unidirectional gateways at the interface between control networks and the safety & protection networks. Unidirectional gateways poll safety and protection devices and emulate those devices to control systems. This design provides for continuous monitoring of the status of safety systems and equipment protection systems without any risk of a network attack pivoting through intermediate systems and networks in order to reconfigure, reprogram, disable or otherwise compromise safety systems and protective relays.

Benefits of Unidirectional Protections

The primary driver for deploying a unidirectionally-protected network is improved security and cyber-threat risk reduction. The biggest cyber threat to power plants is not random, accidental side-effects of infections by high-volume malware, but targeted attacks. When an individual or group deliberately targets a site, they design their attack to bring about as much damage as possible, at a time as inopportune as possible for the targeted site. By far the biggest attack vector for these modern, targeted attacks is interactive remote-control attacks, across the Internet, pivoting through intermediate networks, connections and systems and breaching intermediate firewalls. Unidirectional Security Gateways and related technologies defeat these interactive remote control attacks, as well as more conventional network virus, worm and insider attacks.

A secondary driver in North America, France, Israel and other jurisdictions with cyber security regulations in place, is reduced compliance costs. The NERC CIP V5 and proposed V6 standards for example, have 37 additional, costly “external routable connectivity” rules that large medium-impact power plants must follow when they use firewalls, rules that do not apply to plants protected exclusively by Unidirectional Security Gateways.

An additional driver for unidirectional gateway deployments is operating cost reductions. Power plant IT/OT firewalls tend to be complex: many streams of data, applications and users are typically configured to reach through the firewall to request data from plant systems. Maintenance of these complex firewall configurations is costly, because of the risk that any error in configuration might relax firewall protections unacceptably. Monitoring of firewall logs and communications through firewalls is even more costly, but such monitoring is essential, because of the very real risk that some attack will breach the firewall.

In contrast, Unidirectional Security Gateways and related technologies are generally deployed to replicate a number of servers in whole or in part, and these configurations rarely change. Furthermore, even if unidirectional configurations change over time, unidirectional gateway technologies do not forward messages, and the physical, unidirectional, remote-control-defeating nature of the gateways makes them intrinsically safer than firewalls. As a result, gateway deployments do not warrant the same degree of continuous, costly scrutiny that is mandatory for firewalls.

Looking Forward

Unidirectional communications technologies are an idea whose time has come. Control system security standards all over the world are being updated to reflect the strength of unidirectional protections. Industrial control system owners and operators in all industries are considering and deploying unidirectional gateways to dramatically improve the security posture of their industrial control system networks.

Waterfall Security Solutions has published a whitepaper describing this power plant network architecture in detail, and has network architecture whitepapers in progress for other parts of the electric sector, and for other industries as well. For the full whitepaper, please contact me at andrew.ginter@waterfall-security.com.

Steering Away From IT Security’s “Gold Standard”

This post is authored by Paul Feldman, Chairman, Midwest ISO & Independent Director, WECC.

For the first decade of industrial control-system cybersecurity, IT security practices were held up as the gold standard for control system security. Yes, certain IT practices amounted to constant, aggressive change to “keep up with the bad guys,” such as constant updates to anti-virus signatures and security updates. While these practices were recognized as a poor fit for the engineering change control discipline fundamental to safety and reliability, IT experts kept telling us that if we could just somehow invent a way to apply standard IT security practices to control systems, then all would be well.

This expert consensus is shifting. The IT “gold standard” has been found inadequate to the needs of protecting control systems. How can this be? Well, let’s look at what is the “IT way.” IT security starts at the perimeter with a layer or three of firewalls between the open Internet and the corporate network. These firewalls are assumed to be porous; after all, they forward messages from the Internet into the corporate network, including millions of email messages each day for large organizations, and a comparable number of Web pages. Some of these messages contain attacks. Firewall vendors and security practitioners do what they can to filter out the attacks, but no filter is perfect. Some attacks get through.

Inside the network perimeter, what do we find? Software: countless computers running all manner of software, including security software. The problem is that all software has bugs and some bugs are security vulnerabilities. In practice then, all software can be hacked, even security software. For proof of this, we need look only as far as every security software vendor's website and count the security updates posted last month.

All of this is why the pinnacle of every modern, defense-in-depth, “gold-standard” IT security program is intrusion detection. We put “eyes on glass,” we pit “our experts against theirs,” we assume we have been compromised and we systematically hunt down the equipment our attackers have taken over. We isolate that equipment, erase it and restore it from a pre-compromise backup.

Control system security is different

How does this work for control system security? Firewalls at the control system perimeter are just as porous as firewalls at the corporate perimeter. Firewalls are routers after all, routers with filters. Firewalls forward messages from less-trusted networks into control-system networks, and the filters do what they can to separate “bad” messages from “good” messages. No filter is or can ever be perfect, though. From time to time, all control-system firewalls forward attacks into control-system networks.

Inside every control system network, we find just as much software as we find in corporate networks. Control systems generally have a little less security software deployed than do IT systems, and they are generally a little more out of date than are IT systems. This means that just like IT systems, control-system software can be hacked; the interior of control system networks is generally an even softer target than the interior of IT networks. At first glance then, all the preconditions seem identical, and so intrusion detection systems seem just as essential to ICS networks as to IT networks.

The problem with intrusion detection is that it takes time. In June of 2015, Tripwire published survey results of 400 critical infrastructure executives and IT professionals: 86 percent of the respondents were confident that they could detect compromised equipment on their control-system networks within a week of the compromise. Other studies suggest this confidence is misplaced. A 2014 Ponemon study showed that the average time from infection to detection was 170 days, and a 2014 Verizon study showed that the average time from infection to remediation was 200 days.

Whether the time to detect and remediate compromised equipment is a month, or a week, or an hour is immaterial. For all of that time, however long it is, a remote attacker has control of equipment on our reliability-critical and safety-critical control-system networks. Control system practitioners always regard such unauthorized operation of their equipment as an unacceptable risk. The IT “gold standard” has failed control-system security practitioners. Control-system security must be based on a much more thorough foundation of attack protection than is possible on IT networks.

Revising control-system security standards

Control-system security standards are being revised and updated all over the world, and are evolving away from this IT approach to security. For example, France's 2014 ANSSI regulations for control-system security identify three types of control-system networks, depending on the societal impact of the networks. Class 1 networks are expendable; society suffers minimally when such a network is compromised. Class 2 networks are important to society, and the compromise of class 3 networks has serious consequences. For class 2 networks, ANSSI states that connections to less- trusted networks “should be unidirectional” toward the less-trusted system. For class 3 networks, “The interconnection of a class 3 ICS with an ICS of a lower class shall be unidirectional towards the latter.” The recently-updated NIST 800-82r2, NERC CIP V5 and V6 standards, and IEC 62443-3-3 all position unidirectional gateways within control-system defense-in-depth programs, as well.

To be fair, many elements of the IT gold standard are still applicable to control system security; it is the emphasis that is shifting. The top priorities on control system networks are not availability or integrity after all, but safety and reliability.

The ANSSI classification is instructive. The control networks most important to society must be protected unidirectionally, but there are no such demands of networks French society considers expendable. Few businesses operating large industrial sites, though, will regard their industrial operations as expendable to the business, however expendable society may deem those operations.

With this new understanding of control-system security being codified in updated standards and advice, we all need to start asking, “Which of our operations are expendable enough to be protected by firewalls?”

September news roundup: Exploring the threats to critical infrastructure

September was marked by ongoing exploration and discussion of the very real threats to U.S. critical infrastructure. From successful cyberattacks against U.S. Department of Energy computer systems to a malicious phishing scheme targeting IT workers at critical infrastructure companies, these are the industrial security stories that captured our attention.

U.S. Critical Infrastructure under Cyberattack (Network World, Sept. 29, 2015)

Recent research from ESG reveals that 68 percent of U.S. critical infrastructure organizations have experienced one or several security incidents within the past two years. And 67 percent believe the threat landscape is more dangerous and getting worse than it was two years ago, leading some experts to predict a “cyber Pearl Harbor” in our future.

Cyber Risk Isn’t Always in the Computer (Wall Street Journal, Sept. 24, 2015)

When people think about industrial control systems, they don’t often consider equipment such as backup generators, thermostats and air conditioners, but they should. These components support data-center networks, and due to decades-old technology and communication standards, they are vulnerable to cyberattacks that could take down an entire operation.

Work Needed to Secure Power Grid, Experts Tell House Committee (USAToday, Sept. 11, 2015)

The power grid faces a host of threats, according to witnesses speaking to the House Committee on Science, Space and Technology’s oversight and energy subcommittees. Ranging from natural to physical to cyber, threats to the grid could result in a catastrophic outage, and this possibility should encourage the industry to address vulnerabilities with all possible haste.

Records: Energy Department Struck by Cyberattacks (CNBC, Sept. 10, 2015)

Serving a harsh wake-up call to critical infrastructure companies everywhere, USAToday learned there were 159 cyberattacks that compromised U.S. Department of Energy (DOE) computer systems from 2010 to 2014. Records show that DOE components reported 1,131 total cyberattacks in a 48-month period ending in October 2014, demonstrating a consistent and alarming onslaught of attacks, as well as numerous security vulnerabilities within the department’s cyber defense strategies.

Phishing Schemes Target IT Workers at Critical Infrastructure Companies (Wall Street Journal, Sept. 8, 2015)

The Department of Homeland Security Industrial Control Systems Cyber Emergency Response Team reported the use of a malicious phishing scheme targeting IT workers at critical infrastructure companies. Considered the first stage of a cyberattack, phishing emails are intended to target a critical infrastructure operator’s business network, and from there, its control systems.

For more cybersecurity news, check out last month’s news roundup.