Lessons Learned from the Power Outage in Ukraine and How the Electric Grid of the Future Will Reduce Cybersecurity Risk

First of its kind

From the vantage point of the 225,000[1] Ukrainian customers who lost power on December 23, 2015, it was an ordinary outage.  Customers routinely lose power in Ukraine, particularly in the cold winter months, and since electricity was returned to most customers in less than six hours, there was no reason to suspect extraordinary circumstances surrounding this particular outage.  But there was something different, something exceptional about this outage—these customers did not lose power due to the failure of aging equipment or as a result of a severe weather event.  Three Ukrainian electric utilities were unable to deliver power to their customers on December 23rd due to the effects of a destructive cyberattack executed against these utilities, the first publicly acknowledged power outage resulting from a cyberattack.

How did the attack unfold?

The investigation of the incident is still underway, but publicly available reports from the U.S. Department of Homeland Security (DHS)[2] and the Electricity Information Sharing and Analysis Center (E-ISAC) and SANS[3] provide insight into how the attack unfolded.  At least six months prior to the events of December 23rd, the threat actor first compromised the electric utilities through a spear-phishing campaign that targeted individuals with access to their IT (business) networks.  The phishing emails included Microsoft Word and Excel attachments containing BlackEnergy 3 malware.  When a recipient of the phishing message clicked on one of the attachments and followed a prompt to enable macros within the document, BlackEnergy 3 was installed, providing the adversary backdoor access to the infected system.  After establishing this initial foothold, over the next several months, the actor conducted extensive reconnaissance, compromising user credentials (user names/passwords), escalating privileges, moving laterally throughout the utilities’ IT networks, and ultimately gaining access to their control system (Supervisory Control and Data Acquisition or SCADA) networks used to manage and monitor grid operations.

Leveraging the knowledge and access gained during their recon activities, the adversary launched the highly coordinated attack on December 23rd.  First, they took control of grid operator workstations and opened circuit breakers at approximately 30 substations, taking them all offline and causing the power outage.  Next, to hinder the recovery effort by the utilities, the actor disabled the uninterruptible power supplies (battery backups) for two control centers and disabled remote control of many of the substations to prevent grid operators from sending remote commands to re-close the circuit breakers and restore power.  They also ran a customized version of the KillDisk malware to erase and corrupt various systems at the utilities.  To further complicate recovery efforts (and frustrate customers), the actor also launched a telephone denial-of-service attack against customer call centers to prevent customers from reporting the outages or from gaining clarity on when their power would be restored.

What does this attack mean?

The possibility of a cyberattack causing physical damage to electric system equipment first received widespread news coverage in 2007 when Idaho National Laboratory performed the Aurora generator experiment,[4] in which a diesel generator was destroyed by a simulated cyberattack.  Over eight years later, the attack on the three Ukrainian utilities represents the first publicly acknowledged power outage resulting from a cyberattack.  Although this is significant in its own right, the overall impact of the attack was relatively low when gauged by the number of customers impacted (225,000 out of a Ukrainian population of over 40 million[5]) and the duration of the power outage (less than 10 hours).

Lessons learned

That said, for owners and operators of critical infrastructure in the United States, there are many lessons learned from the incident in Ukraine, a few of which are summarized below.

As has become commonplace, the initial attack vector was through spear-phishing.

1. From a behavior management perspective, to reduce the risk of employees clicking on malicious links or attachments, organizations should implement a phishing simulation program through which they send faux phishing messages to employees to test their ability to recognize such messages. Employees who click the link or attachment in the message are immediately presented with a brief online training vignette that reinforces common indicators of malicious messages.
2. From a technology perspective, organizations should deploy a sandboxing solution that inspects inbound emails for malicious links or attachments before allowing the email to land in an employee’s inbox.

Once the adversary breached the utilities through a phishing attack, they were able to compromise valid user credentials, move throughout the utilities’ IT networks, and ultimately gain access to control system networks used to manage and monitor grid operations.

3. Organizations should implement a robust network security architecture, including proper segregation and segmentation between the IT and control system networks using firewalls and intrusion prevention/detection tools. Organizations should also perform continuous network security monitoring in order to baseline and understand normal activity, thus enabling the identification of abnormalities on the network.
4. Organizations should limit remote access to control system networks to the full extent possible. If remote access to networks is unavoidable, the connections should be time limited and controlled using two-factor authentication where a user can access systems only after entering two forms of credentials, such as a password (something the user knows) and a one-time code texted to the user’s smartphone (something the user possesses).

Lastly, the threat actor employed a variety of techniques to delay the utilities’ recovery efforts, including disabling control center battery backup power, preventing grid operators from sending remote commands to re-close the circuit breakers and restore power, erasing and corrupting various systems at the utilities using a customized version of the KillDisk malware, and the execution of a telephone denial-of-service attack against customer call centers to prevent real customers from being able to reach the call centers.

5. Organizations should document a detailed incident response plan and test the plan on a regular basis. These drills should include active participation from all team members who play a role in the incident response effort, including technical (Information Technology, Operations Technology, and Cybersecurity) and Operations staff, as well as key partners, such as control system vendors and forensics providers.  The Ukraine attacks underscore the importance of stretching the incident response team during test exercises with a wide range of scenarios of varying levels of complexity and sophistication.

Additional mitigation strategies are outlined in the reports issued by DHS² and E-ISAC/SANS³.

Other factors to consider

Contrary to the media hype associated with this subject, DHS stated in a recent report[6] that they believe the threat of a damaging or disruptive cyberattack against the United States energy sector is low.  Most experts agree that although several actors possess the capabilities to execute such an attack, the motivation is less prominent since a widespread attack on the electric grid would have a devastating impact on the world economy, not just the U.S. economy.  The likelihood of a destructive cyberattack is much higher when two nations are entering an armed conflict, in which case cyber warfare would be used to complement conventional warfare.

That said, there is no debate that the threat to the power sector is real and that the adversary is becoming increasingly sophisticated. The key for electric utilities in the short- and medium-term is to remain agile, continuously evolve, and strengthen both their defenses and response capabilities.

Reducing risk by transforming the architecture of the electric grid

For our nation, perhaps the ultimate long-term mitigation against a widespread blackout is the fundamental transformation that is already underway related to the architecture of the U.S. electric system.  In its simplest form, the current power delivery system in the U.S. involves the central generation of electricity at large power plants, the transmission of power from where it is generated to the area where it will be consumed by customers, and finally distribution of the power to the end customer, including residential households and commercial and industrial facilities.  Since electricity is generated and transmitted at high voltage levels and consumed by customers at much lower levels, large transformers located at substations sit in between power plants and customer sites to increase or decrease (transform) the voltage to the appropriate levels.

Large power plants and large substations present an easier target for threat actors who wish to cause a widespread, cascading blackout by disrupting the flow of electricity between the generation source and the end customer (through a cyberattack, a physical attack, or a combination thereof).  The transformation that is underway in the U.S. electric system involves two fundamental shifts, both of which are altering the architecture of the grid—first, the decentralization of power generation (think rooftop solar) and second, the introduction of more advanced technologies like battery-based energy storage that provide additional resiliency to the grid by allowing energy to be charged (stored) and discharged at a moment’s notice.  Both of these factors also enable increased deployments of microgrids, local energy grids that connect to the traditional macro electric grid, but have the capability of disconnecting and operating autonomously.

Adoption of these newer (and cleaner) technologies is driven by:

• Rapid declines in the cost of the underlying technologies—For example, the installed price of solar PV systems has fallen over 55 percent in the past five years[7];
• Expanding environmental policies—At COP21 in Paris, for example, nearly 200 countries made historic commitments to significantly reduce carbon emissions; and
• Increased customer empowerment—More than 50 companies, including Google, Nike, Starbucks, and Walmart, are part of a global campaign of businesses committed to 100 percent renewable electricity.[8]

As technology prices continue to fall and new market mechanisms are designed to more appropriately value resources like solar and energy storage, the transformation will accelerate.  The electric grid of the future is cleaner, smarter, and more flexible.  In addition, its architecture is more distributed due to technologies like rooftop solar and other forms of distributed generation and more resilient thanks to technologies like energy storage.

The electric grid of the future will not be realized overnight, but all of these changes will dramatically alter the risk profile of the electric delivery system in the U.S., making it significantly more difficult for threat actors to plan and execute a cyberattack that causes a widespread, cascading power outage in our nation.

References

[1] 225,000 endpoints (homes/facilities) were impacted, so the number of individuals who lost power is larger.

[2] “Cyberattack Against Ukrainian Critical Infrastructure,” ICS-CERT, U.S. Department of Homeland Security, IR-ALERT-H-16-056-01, Feb. 25, 2016, https://ics-cert.us-cert.gov/alerts/IR-ALERT-H-16-056-01.

[3] E-ISAC/SANS, Analysis of the Cyberattack on the Ukrainian Power Grid, SANS ICS, March 18, 2016, https://ics.sans.org/media/E-ISAC_SANS_Ukraine_DUC_5.pdf.

[4] “Mouse Click Could Plunge City into Darkness, Experts Say,” CNN, Sept. 27, 2007, http://www.cnn.com/2007/US/09/27/power.at.risk/index.html.

[5] “World Fact Book: Ukraine,” CIA, last updated May 6, 2016, https://www.cia.gov/library/publications/the-world-factbook/geos/up.html.

[6] “DHS Intelligence Assessment: Damaging Cyber Attacks Possible but Not Likely Against the US Energy Sector,” U.S. Department of Homeland Security, Jan. 27, 2016, available at https://publicintelligence.net/dhs-cyber-attacks-energy-sector/.

[7] “Where are Solar Costs Headed?,” Presentation by MJ Shiao, GTM Research, at the Solar Goes Corporate conference,Feb. 2016.

[8] “Companies,” RE100, http://there100.org/companies(list of RE100 companies).