Introduction
The United States economy, as well as large portions of the critical infrastructure that supports it, now depends on a working Internet for normal operations. The Internet has enabled an always-on, always-connected infrastructure that society depends on daily. Though the Internet seems omnipresent in modern life, it has its own physical infrastructure that is critical to its operations and is not well-understood by public- and private-sector decision makers. Internet infrastructure is primarily composed of data centers of varying size and type and the fiber-optic connections that connect these centers to each other. Data centers and fiber lines are generally designed to be invisible to the casual passerby but are increasingly present in communities nationwide, and their concentration can itself become a vulnerability to Internet resilience.
The cluster of data centers in the greater Ashburn area in Loudoun County serves as the primary global Internet traffic hub on the East Coast owing to the presence of major Internet exchange points (IXPs). With the unique concentration of both fiber and power, 50 to 70 percent of all Internet traffic flows through the greater Ashburn-area data centers on average. These facilities contain information technology (IT) infrastructure supporting governmental agencies and private companies that, in turn, supply day-to-day services to critical utilities and the public.[1]
Through the Regional Resiliency Assessment Program (RRAP), the U.S. Department of Homeland Security (DHS) launched a project in 2016 to study the physical vulnerabilities of the regional Internet infrastructure located in Ashburn, VA (in the suburbs of Washington, D.C.).[2] Within this study, DHS, supported by Argonne National Laboratory (ANL), evaluated both upstream and downstream dependencies of data center and fiber providers in the region and the potential cascading consequences of outage events to both key nodes in Internet infrastructure and their dependencies.
Types of Internet Infrastructure
When generically referencing “data centers,” several types of facilities can fit within this categorization (Figure 1).
For illustration purposes, the following types of Internet infrastructure can help frame discussions about Internet resilience and related dependencies.
- Tier 1 Network Provider Data Centers – Network provider data centers that have access to every other network on the Internet without paying for transit. Tier 1 network providers have their own data centers where content providers can rent space to deliver increased performance to a variety of customers.
- Content Provider Data Centers – Data centers operated by major content or cloud providers (Google, Microsoft, Facebook, etc.)
- Internet Exchange Points (IXPs, also known as Network Access Points [NAPs]) – IXPs offer peering and transit interconnections. These are essentially buildings where different networks come together to talk to each other.
- Colocation/Real Estate Data Centers – Colocation data centers are built by companies that typically consider themselves to be in the real estate market. These companies sell power, cooling, and/or network in addition to physical space. They generally view their relationship with the customer as a traditional landlord/tenant relationship.
- Internet Service Provider Data Centers – Although many customers connect directly through tier 1 providers to the Internet, tier 2 and tier 3 Internet Service Providers (ISPs) often have data centers of their own, as well. A tier 2 network is an ISP who engages in the practice of peering with other networks, but who also purchases Internet protocol transit to reach some portion of the Internet. Tier 2 providers are the most common ISPs, as it is much easier to purchase transit from a tier 1 network than it is to peer with them and attempt to become a tier 1 carrier. Tier 3 networks purchase Internet protocol transit solely from other networks to reach the Internet. These data centers are smaller and often only house services for their direct customers.
- Fiber – Fiber optic cables are typically run underground or underwater. They often follow road or railways and have maintenance vaults where signals can be amplified, split, retransmitted, etc. Fiber routes and vaults are typically minimally protected by traditional physical security methods.[3]
Vulnerabilities of Internet Infrastructure
The findings outlined below were identified through the Ashburn RRAP project but apply more generally to Internet infrastructure as a whole.
Internet resilience is contingent on a limited number of centralized Internet exchange points (IXPs).
“Peering” is a critical function that allows interconnections between separate networks and allows traffic to flow across them, ultimately enabling the Internet to operate effectively. North American Internet infrastructure developed around centralized peering points, frequently called IXPs.[4] Peering is defined as an agreement between two networks to accept each other’s data packets and forward them. IXPs act as major data exchange hubs, where hundreds (and through them, millions) of participants exchange Internet traffic. The centralized nature of IXPs is largely market driven. It is suspected that the networks and servers within these facilities are largely resilient against cyber-attacks because of the presence of a wide variety of companies, hardware, software, etc. However, the existence of a single physical structure that houses so many important connections represents a vulnerability with the potential for serious cascading consequences for global Internet routing.
Transparency in both network and data center infrastructure would enhance resilience planning.
Customers and providers alike have an interest in data center resilience. Many data center providers are very transparent about resilience and preventive maintenance with customers and audit this information. However, sufficient shared vocabulary does not exist for customers and network/data providers to discuss resilience requirements and options. Cloud services (i.e., elastic computing resources housed in data centers and depending on network and Internet infrastructure) complicate this challenge considerably, as some providers may not have complete visibility into the underlying operating infrastructure.
Law enforcement does not have adequate training or information to recognize suspicious activity around unprotected fiber routes and vaults.
Local law enforcement is often the first line in defending critical infrastructure against malicious activity. Law enforcement at the local and state levels are typically not trained on how to identify suspicious activity around fiber routes and vaults. Law enforcement personnel do not know where critical nodes are located, nor do they know how to authenticate legitimate maintenance personnel.
Conclusions and Future Work
Internet infrastructure is highly distributed among different private- and public-sector entities. Many companies own different parts of the infrastructure, and Internet traffic can travel on many paths. The Transmission Control Protocol (TCP) that is one of the primary communications protocols used on the Internet is designed as an end-to-end protocol, meaning that it attempts to guarantee message delivery from sender to receiver despite changes or outages in the intervening network nodes. This protocol allows networks using TCP to be highly resistant to failures when multiple paths from sender to receiver exist. TCP’s resilience has the potential to fail if a concentration of high-capacity routes between a particular sender and receiver become unavailable. The geographical locality of such a large quantity of data centers and network routes in the Ashburn area makes this concentration a concern. However, even a single fiber cut can cause major outages, so awareness and protection of physical Internet assets must be a higher National priority, not only in Ashburn.[5] The New York Times recently published an article on the vulnerabilities and importance of the physical infrastructure that the Internet comprises.[6] As Internet access increasingly becomes a critical dependency, the public is becoming more aware of its physical vulnerabilities. This recognition has made it imperative to study these vulnerabilities and develop strategies to improve the infrastructure’s resilience.
In the wake of the Ashburn RRAP project and its findings, a study should be conducted that simulates the outage of an IXP. Such a simulation would include modeling the traffic and TCP congestion during an outage of IXP facilities in the greater Ashburn area. In addition, a workshop should be conducted on the development of cloud and data center service taxonomies and assessments. Such taxonomies and/or assessments should allow for equal comparison of resilience features across providers and empower customers by fostering open, honest competition. Also, law enforcement personnel should engage industry stakeholders to facilitate training and education on fiber routes and suspicious activity. This training should also address how and when to approach maintenance personnel and how to confirm that they are authorized to work in a given area.
Additional data need to be gathered about routes, capacity, and throughput. Though network and data center providers generally run detailed failure scenarios against their own infrastructure, the Internet is an interconnected network by its very nature.[7] At a minimum, these types of scenarios should be done involving multiple providers. With adequate provider involvement, more conclusive analysis could be conducted, allowing for better resilience planning.
These problems are not unique to the Internet. Large critical infrastructure owners and operators and government entities often purchase or deploy their own private managed networks. However, many of these private network owners may not realize that their networks may run over the same physical fiber routes as Internet routes. Thus, although they may be insulated from Internet congestion, they are not insulated from physical damage to Internet infrastructure, whether accidental or—as has become more and more commonplace—intentional.[8]
The submitted manuscript has been created by UChicago Argonne, LLC, Operator of Argonne National Laboratory (“Argonne”). Argonne, a U.S. Department of Energy Office of Science laboratory, is operated under Contract No. DE-AC02-06CH11357. The U.S. Government retains for itself, and others acting on its behalf, a paid-up nonexclusive, irrevocable worldwide license in said article to reproduce, prepare derivative works, distribute copies to the public, and perform publicly and display publicly, by or on behalf of the Government. The work presented in this paper was partially supported by Argonne National Laboratory under U.S. Department of Energy contract number DE-AC02-06CH11357.
If you would like more information regarding the information presented in this paper, please contact Michael Thomson at thompsonm@anl.gov.
References
[1] Regional Resiliency Assessment Program Fact Sheet (U.S. Department of Homeland Security, 2015), https://www.dhs.gov/publication/rrap-fact-sheet.
[2] Johnathan O’Connell, “Data Centers Boom in Loudoun County, but Jobs Aren’t Following,” The Washington Post, Jan. 17, 2014, http://www.washingtonpost.com/business/capitalbusiness/data-centers-boom-in-loudoun-county-but-jobs-arent-following/2014/01/17/b4a704c8-7f0e-11e3-93c1-0e888170b723_story.html.
[3] Ramakrishnan Durairajan, et al., InterTubes: A Study of the US Long-Haul Fiber-Optic Infrastructure (2015), http://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p565.pdf.
[4] Andrew Blum, Tubes: A Journey to the Center of the Internet (New York: HarperCollins Publishers, 2012).
[5] Chris Talbot, “Amazon ‘Outage’ Actually ISP Fiber Line Cut,” Talkin’ Cloud, Jan. 7, 2014, http://talkincloud.com/public-cloud/010714/east-coast-blizzard-takes-down-amazon-customers.
[6] Kate Murphey, “The Cyberthreat Under the Street,” New York Times, Nov. 7, 2015, http://www.nytimes.com/2015/11/08/sunday-review/the-cyberthreat-under-the-street.html.
[7] Tekla S. Perry, “Facebook Engineers Crash Data Centers in Real-World Stress Test,” IEEE Spectrum, Sept. 2, 2016, http://spectrum.ieee.org/view-from-the-valley/computing/it/facebook-engineers-crash-data-centers-in-realworld-stress-test.
[8] Ed Payne, “Vandals Cut San Francisco Area Fiber Optic Lines for 11th Time in a Year,” CNN, July 1, 2015, http://www.cnn.com/2015/07/01/tech/california-fiber-optic-cable-cuts/.