Report on Cyber Tempest

Regional Cyber Exercise
Dec. 4-5, 2006

Maeve Dion, Legal Research Associate. Email.
December 2006.

It started as a typical day: software vendors released routine patches for minor, unexploited vulnerabilities; there existed the standard vague and unconfirmed foreign-source threats against government and financial service networks; and there was some increase in underground chat activity, but nothing extraordinary. The US CERT issued an alert on unexploited vulnerabilities in a common Internet browser; mitigation procedures were provided.

A little while later, a malware blog posted unconfirmed reports that the browser vulnerabilities were being exploited, and that other browsers may have similar vulnerabilities.

Around the same time, an electric utility’s helpdesk sent out a company-wide email advisory: its telephone network (PSTN) was suffering intermittent failures.

Meanwhile, media traffic reporters announced emergency maintenance closures along major urban areas of Interstate 95. The reporters had gleaned this information from state department of transportation websites. Unbeknownst to the reporters, the DOT websites of five northeastern states had been defaced, and the emergency closure warning was fraudulent. The states’ cybersecurity offices began fielding phone calls from governors’ offices. The misinformation and confusion created traffic congestion at unparalleled levels.

In another part of the economy, websites of hospitals, state Red Cross offices, and hospital associations were also defaced, replaced with statements decrying the cost of healthcare, and calling for action.

In the financial sector, numerous institutions were receiving an unusually high volume of customer complaints that they had been locked out of their online accounts. ... The electric utility’s helpdesk was similarly inundated with customer “lock-out” calls. ... Government employees in the five northeastern states were locked out of government networks. ... Healthcare providers contacted the state departments of health to report an inability to access their Health Provider Network accounts.

Back at the electric utility, in addition to the telephone network failures, sporadic DDoS attacks began to significantly slow the utility’s email service. Concurrently, the same DDoS problem was occurring in financial and healthcare institutions, and the state governments.

The press began reporting the account lock-out problems at the financial institutions. Unfortunately, some of the financial institutions had also begun experiencing intermittent failures of their telephone networks (PSTN). ... Three of the northeastern state governments encountered similar PSTN problems. ... Various healthcare facilities experienced telephone network failures, which seriously impeded investigation of Statewide Planning and Research Cooperative System reports of an increase in chicken pox.

In the meantime, a malware blog announced new, more complex and aggressive variants of Bifrose (a backdoor Trojan which provides unauthorized remote access to an infected computer).

And a major hardware manufacturer issued a security advisory regarding a serious flaw in the operating system of its popular routers. The flaw could be exploited by a remote, unauthorized user, who could create a condition of sustained denial of service. The manufacturer recommended upgrading the router software to a new version, and disabling the Dynamic Host Configuration Protocol service.

As part of the Cyber Tempest exercise, these events were played out in four rooms at a state conference center outside of Albany, New York. The approximately 100 participants represented federal, state, and local agencies; businesses and associations in the financial, electric, healthcare, information technology, and telecommunications sectors; the MS-ISAC, FS-ISAC, Communications ISAC, and IT-ISAC; and observers from academia and the Canadian government.

In addition to the above scenarios, the participants also had to address unreliable T1 lines and frame relays; unreliable ATM communications which required additional cash to be sent to branches / institutions; intermittent 911 service; escalation of the Bifrose trojan problem and the DDoS attacks targeting email servers; discovery of physically-installed keyloggers; widespread local and long-distance telephone outages in the northeast; large-scale botnet attacks against northeast financial institutions; communications (from trusted sources / channels) that induced certain information sharing and installation of software patches ... and later discoveries that these communications were fraudulent and that the “patch” had corrupted and prevented performance of equipment that relied on the software; detection of a week-old theft of credit card information from three states’ EZ-Pass systems; increasingly high volumes of customer complaints regarding utility billing errors; Health Provider Network posts (via authorized accounts) warning of e-coli infected salad greens ... and a subsequent denial by account owners that they had sent the postings; unauthorized access and corruption of a utility’s customer meter reading database; corruption of the northeastern states’ Criminal Justice Information System databases; and extortion demands and threats of additional damage.

All this on only the first day of the exercise.

The second day heralded additional problems, including release of a previously unknown internet worm; failure of the elevators and heating, ventilation, and air conditioning (HVAC) systems in healthcare and financial institutions, which caused the financial institutions to evacuate and relocate their employees / services to areas outside of the affected region; escalating reports of insufficient funds in commercial customers’ financial accounts; widespread water purification problems due to corrupted process control systems for purification applications; discovery of significant discrepancies between the database inventory and physical inventory of blood supplies; lock-out from the Hospital / Health Emergency Response Data System; failure of electric utilities’ energy management systems; failure of electric utilities’ frame relays; wide area rolling electrical power disruptions in the northeast; and complete loss of power in northeast healthcare and financial institutions.

This two-day Cyber Tempest was organized by the New York State Office of Cyber Security and Critical Infrastructure Coordination (CSCIC) and the Multi-State Information Sharing and Analysis Center (MS-ISAC), and was supported by the Department of Homeland Security’s National Cyber Security Division (NCSD).

As stated by William F. Pelgrin (speaking above, center), Director, New York State CSCIC and MS-ISAC Chair, the primary goal of Cyber Tempest was “to exercise the interaction (e.g., information-sharing, coordination, etc.) and consequences of regional cyber network disruptions, as well as explore the vast complexities of interrelated effects.” Pelgrin continued, “It’s a great collaboration between levels of government and with the private sector. It’s not about how good we are -- it’s about how good we can be.”

The exercise was similar to a war game, in that the participants’ responses, decision-making, and information sharing affected the exercise. Cyber Tempest focused on a wide area of cyber disruption from a regional perspective, and thus was artificially bounded to avoid addressing the scope of events resulting from a declared Incident of National Significance or a Federal Emergency Declaration. The participants were also instructed to discuss only the cyber implications, and not the technical causes of the events or the possible physical consequences.

The participants were informed up front that the events were not terrorism related, because that would involve different levels of activity and groups of people not participating in this regional exercise.

Throughout the exercise, participants focused on how to (1) gain and maintain situational awareness; (2) develop strategy / actions with an integrated response; (3) mitigate consequences; (4) allocate limited resources; and (5) collect, analyze, formulate, and disseminate information to stakeholders (including the media). The participants also developed recommendations for sector regulators. The control group facilitated the exercise and created a process to record the inter-group communications (who initiated the communication; to whom they communicated; content of the communication; and response / outcome).

Cyber Tempest was structured so that there were four separate groups during the gaming sessions -- Government, Financial, Healthcare, and Utilities (IT/Telecom, and Electricity). The groups communicated with each other via couriers. Members of the Control Group monitored the gaming groups to ensure that the participants adhered to the scenario injects. The Control Group also acted as “extras” (software vendors, Federal intelligence organizations, etc.) to respond to the participants’ queries. During the gaming sessions, there were also periodic ISAC meetings, drawing participants from each group. After each gaming session, there was an outbrief attended by all participants.

In each outbrief, it was interesting to note the different “alert levels” the various groups had reached. The exercise exposed the diverse decision-making thresholds among industries (e.g., when to contact an ISAC; when to assume certain problems were correlated; when to report to law enforcement; when to ask for government help; etc.). As the events escalated, law enforcement was the first to completely disconnect from the Internet, followed to some degree (e.g., from disconnecting most systems, to disconnecting only non-critical systems) by state governments, electric, health, and finance.

During the first outbrief, participants wondered if, in the normal workday setting, they would have correlated the events, and if so, when (i.e., too late?). As one participant commented, “if we were not all in the same room, would we have reached these conclusions? Would we have asked, ‘is this happening somewhere else?’”

Confidentiality, reputation cost, and other trust issues were another big factor. Not only did institutions have to consider obligations and consequences of reporting to regulators and law enforcement, but information sharing with ISACs carried different implications depending on the sector. One ISAC operated so that the institutions could share information on new and ongoing incidents (helping to identify patterns of behavior / problems), but another ISAC operated so that the member institutions only reported incidents after they were resolved.

The participants also had to face problems of lack of public and consumer confidence, and the participants’ own lack of trust in communications. Discovery of fraudulent communications injected a hesitance in accepting all later communications at face value. The groups had to develop practices to reasonably verify the sources of later communications.

The exercise showed that of the four groups, the Government group was the first to ask “Who is doing this? Why? and What will they do next?” While all the groups were actively involved in containing and mitigating the problems, the Government group showed early leadership and creative thinking regarding prediction of future events, expending resources to ask (if not answer) these questions. In the second day of the exercise, as the escalating events highlighted various interrelationships and interdependencies, both the private and public sectors responded with an excellent level of information sharing, volunteering and requesting information.

Cyber Tempest also provided opportunity for both public and private sector participants to understand just how quickly a regional problem could outstrip the coordination and analysis resources of law enforcement. Similarly, the law enforcement participants observed the different industries’ thresholds and concerns for involving law enforcement when the institutions may need to rebuild their networks on a priority basis. As one law enforcement participant said, the exercise provided “excellent insight into the critical thinking and decision-making that occur in businesses before law enforcement gets involved.” This understanding helps the state police to respond better to businesses’ concerns.

One industry participant stated that if he reported the attack, law enforcement may want to shut the network down for forensics, which the company could not afford. However, Cyber Tempest gave the law enforcement participants the opportunity to share some of the more modern practices of some state police cyber offices: they commented that (1) it is a recognized bad business practice to demand a shut down during a critical incident, and (2) the police often do not have the facilities or resources to run a proper investigation with the police forensics labs (which are often busy with work related to many other every-day criminal prosecutions), so a demand to shut down the network would be unlikely in such situations.

As a regional exercise, Cyber Tempest caused the participants to examine the prioritization and allocation of resources. Some Disaster Recovery (DR) and Business Continuity plans were serviced on a “first come, first served” basis. Some DR facilities may have faced the same vulnerabilities as the main network / systems, since the DR centers used the same (flawed) software or connected back to the same (infected) network.

Some of the participants noted that when it comes to cyber events, a regional response may not be practical. For example, once an event is beyond the control of a financial or telecommunications institution, it likely will require a national response (e.g., failure of the frame relay; the need to provide and disperse cash after the ATMs and regional financial facilities went down, etc.).

However, this regional exercise did result in questions that were new to some of the participants. For example, who coordinates prioritization of restoration in a region? Each state government would have its own plan, but if an event is regional (yet not national), should there be a regional restoration plan?

As the Control Group announced at the end of Cyber Tempest, the “bad guys” were two hacker groups who were competing with each other to gain prestige in the black market economy. The goals were to remain below the level of a cyber Incident of National Significance and to keep their identities secret, and the group with the most money “won.” As the Control Group leader explained, “it was about money, not about killing people.” The hacker groups had begun their assault six months earlier, by buying insiders at various institutions; however, as the game progressed, the cyber incidents got out of the hackers’ control.

As explained by Glenn Fiedelholtz, Deputy Director of Exercises, NCSD, “Cyber Tempest was a unique cyber exercise in that it was one of the first Northeastern regional exercises in the United States, which tested the information sharing and communication path capacities of the public and private sectors -- IT, communications, utilities, finance / banking, health, and government -- to respond to a cyber event. Additionally, the exercise examined interdependent responses and their cascading effects within and across sectors.”

Although the gaming aspect of Cyber Tempest is complete, the exercise itself is not yet over. The New York CSCIC expects to have a Cyber Tempest after-action report out to all participants before the end of December. Then, during the month of January, CSCIC will organize several conference calls among designated leads in each participating group, to analyze the report and discuss actionable items. Finally, these designated leads will meet in February, at the GMU Law School, to finalize action items and recommendations that will be distributed to all participants. Where possible, the CIP Program will share additional results and information via the CIP Report and our website.

***

Cyber Tempest was Sponsored by the New York State Office of Cyber Security and Critical Infrastructure Coordination, and the Multi-State Information Sharing and Analysis Center, and was supported by the Department of Homeland Security, National Cyber Security Division.

Jeff Wright, Deputy Director for Strategic Initiatives, and Director for Exercises, NCSD, gave the opening address at Cyber Tempest.