As computer technology has advanced, federal agencies have become dependent on computerized information systems to carry out their operations and to process, maintain, and report essential information. Virtually all federal operations are supported by computer systems and electronic data, and agencies would find it difficult, if not impossible, to carry out their missions, deliver services to the public, and account for their resources without these cyber assets. Information security is thus especially important for federal agencies to ensure the confidentiality, integrity, and availability of their systems and data. Conversely, ineffective information security controls can result in significant risk to a broad array of government operations and assets, as the following examples illustrate: Computer resources could be used for unauthorized purposes or to launch attacks on other computer systems. Sensitive information, such as personally identifiable information, intellectual property, and proprietary business information, could be inappropriately disclosed, browsed, or copied for purposes of identity theft, espionage, or other types of crime. Critical operations, such as those supporting critical infrastructure, national defense, and emergency services, could be disrupted. Data could be added, modified, or deleted for purposes of fraud, subterfuge, or disruption. Threats to systems are evolving and growing. Cyber threats can be unintentional or intentional. Unintentional or non-adversarial threats sources include failures in equipment, environmental controls, or software due to aging, resource depletion, or other circumstances which exceed expected operating parameters. They also include natural disasters and failures of critical infrastructure on which the organization depends but are outside of the control of the organization. Intentional or adversarial threats include individuals, groups, entities, or nations that seek to leverage the organization’s dependence on cyber resources (i.e., information in electronic form, information and communications technologies, and the communications and information-handling capabilities provided by those technologies). Threats can come from a wide array of sources, including corrupt employees, criminal groups, and terrorists. These threat adversaries vary in terms of their capabilities, their willingness to act, and their motives, which can include seeking monetary gain, or seeking an economic, political, or military advantage. Table 1 describes the sources of cyber based threats in more detail. Cyber threat adversaries make use of various techniques, tactics, and practices, or exploits, to adversely affect an organization’s computers, software, or networks, or to intercept or steal valuable or sensitive information. These exploits are carried out through various conduits, including websites, e-mails, wireless and cellular communications, Internet protocols, portable media, and social media. Further, adversaries can leverage common computer software programs, such as Adobe Acrobat and Microsoft Office, as a means by which to deliver a threat by embedding exploits within software files that can be activated when a user opens a file within its corresponding program. Table 2 provides descriptions of common exploits or techniques, tactics, and practices used by cyber adversaries. Reports of successfully executed cyber exploits illustrate the debilitating effects they can have on the nation’s security and economy, and on public health and safety. Further, federal agencies have experienced security breaches in their networks, potentially allowing sensitive information to be compromised and systems, operations, and services to be disrupted. These examples illustrate that a broad array of federal information and critical infrastructures are at risk: In August 2015, the Internal Revenue Service (IRS) reported that approximately 390,000 tax accounts were potentially affected by unauthorized third parties gaining access to taxpayer information from the agency’s “Get Transcript” application. According to testimony from the Commissioner of the IRS in June 2015, criminals used taxpayer- specific data acquired from non-IRS sources to gain unauthorized access to information; however, at that time, he reported that approximately 100,000 tax accounts had been affected. These data included Social Security information, dates of birth, and street addresses. In July 2015, the Office of Personnel Management reported that an intrusion into its systems compromised the background investigation files of 21.5 million individuals. This was in addition to a separate but related incident that affected personnel records of about 4 million current and former federal employees, which the agency announced in June 2015. In September 2014, a cyber-intrusion into the United States Postal Service’s information systems may have compromised personally identifiable information for more than 800,000 of its employees. The Federal Information Security Management Act of 2002 (FISMA 2002) was enacted into law to provide a comprehensive framework for ensuring the effectiveness of information security controls over federal information resources. The law required each agency to develop, document, and implement an agency-wide information security program to provide risk-based protections for the information and information systems that support the operations and assets of the agency. Such a program includes assessing risk; developing and implementing cost- effective security plans, policies, and procedures; plans for providing adequate information security for networks, facilities, and systems; providing security awareness and specialized training; testing and evaluating the effectiveness of controls; planning, implementing, evaluating, and documenting remedial actions to address information security deficiencies; procedures for detecting, reporting, and responding to security incidents; and ensuring continuity of operations. The act also assigned oversight responsibilities to the Office of Management and Budget (OMB) and gave the National Institute of Standards and Technology (NIST) responsibility for developing standards and guidelines that include minimum information security requirements. The Federal Information Security Modernization Act of 2014 largely supersedes FISMA 2002. This law retains the requirements for agencies to develop, document, and implement an agency-wide information security program, as well as OMB oversight and NIST development of standards and guidelines. Its changes include requiring DHS to assist OMB with providing oversight by administering the implementation of information security policies and practices for information systems. DHS responsibilities include developing and overseeing the implementation of binding operational directives requiring agencies to implement OMB’s information security standards and guidelines; operating a federal information security incident center (previously OMB’s responsibility), which has been established as the DHS United States Computer Emergency Readiness Team (US-CERT); deploying technology, upon request by an agency, to continuously diagnose and mitigate against cyber threats and vulnerabilities; and conducting targeted operational evaluations, including threat and vulnerability assessments, on agency information systems. In January 2008, the President issued National Security Presidential Directive 54/Homeland Security Presidential Directive 23. The directive established the Comprehensive National Cybersecurity Initiative, a set of projects with the objective of safeguarding federal executive branch government information systems by reducing potential vulnerabilities, protecting against intrusion attempts, and anticipating future threats against the federal government’s networks. Under the initiative, DHS was to lead several projects to better secure civilian federal government networks, while other agencies, including OMB, the Department of Defense, and the Office of the Director of National Intelligence had key roles in other projects, including monitoring military systems and classified networks, overseeing intelligence community systems and networks, and spearheading advanced technology research and development. The initiative’s projects can be grouped into three focus areas: Establishing front lines of defense. This includes projects intended to protect the perimeter of federal networks, such as consolidating connections and deploying intrusion detection and prevention systems. Defending against full spectrum of threats. This includes physical and cyber projects intended to protect national security and intelligence- related information and systems across the federal government. Shaping the future environment. The initiatives in this area are focused on expanding cybersecurity education and research and development efforts for future technologies and cybersecurity strategies. As required by FISMA (both the 2002 and 2014 laws), NIST has developed standards and guidelines for agencies to develop, document and implement their required information security programs, select controls for systems, and conduct risk-based cyber threat mitigation activities. For example, NIST’s Special Publication 800-37 recommends cost-effectively reducing information security risks to an acceptable level and ensuring that information security is addressed throughout an information system’s life cycle. In addition, NIST Special Publication 800-94 establishes guidance for federal agencies to use when designing, implementing, and maintaining the systems they deploy to perform intrusion detection and prevention. DHS designated the National Protection and Programs Directorate to lead the national effort to strengthen the security and resilience of the nation’s physical and cyber-critical infrastructure, including supporting federal agencies in securing their information systems and information. Specifically, the directorate is responsible for enhancing the security, resilience, and reliability of federal agencies in the protection of the “.gov” domain of the federal civilian government. Within the National Protection and Programs Directorate, the Office of Cybersecurity and Communications, among other things, operates the National Cybersecurity and Communications Integration Center (NCCIC) that is to serve as a 24/7 cyber monitoring, incident response, and management center and as a national focal point of cyber and communications incident integration. The US-CERT, one of several subcomponents of the NCCIC, is responsible for operating the NCPS, which provides intrusion detection and prevention capabilities to covered federal agencies. The Network Security Deployment (NSD) division of the Office of Cybersecurity and Communications is responsible for developing, deploying, and sustaining NCPS. For example, the division is to deliver NCPS intrusion detection capability directly to federal agencies through Trusted Internet Connection Access Providers or through Internet service providers at Managed Trusted Internet Protocol Service locations. NCPS is an integrated system-of-systems that is intended to deliver a range of capabilities, including intrusion detection, intrusion prevention, analytics, and information sharing. The NCPS capabilities, operationally known as the Einstein program, are one of a number of tools and capabilities that assist in federal network defense. Originally created in 2003, NCPS is intended to aid DHS in its ability to help reduce and prevent computer network vulnerabilities across the federal government. Its analysts examine raw and summarized data from a wide variety of information sources to make determinations about potential attacks across the network traffic of participating federal agencies detected by NCPS. Table 3 provides an overview of the enhancements DHS has made to the original iteration of Einstein as well as the corresponding objective of NCPS the functionality supports. NCPS is intended to build successive layers of defense mechanisms into the federal government’s information technology infrastructures. When NCPS intrusion detection sensors are deployed at a Trusted Internet Connection location, the system monitors inbound and outbound network traffic, with the goal of allowing US-CERT, using NCPS and its supporting processes, to monitor all traffic passing between the federal civilian networks and the Internet for malicious activity. Figure 1 illustrates how Trusted Internet Connection portals interact with the NCPS intrusion detection sensors and the Internet. For more detailed information about NCPS’s development and functionality, see appendix II. As we reported in April 2015, DHS spent over $1.2 billion on the NCPS system through fiscal year 2014. Figure 2 below depicts the funds spent for NCPS over the past 6 budget years. NSD plans to use the fiscal year 2015 funding to sustain currently deployed capabilities and expand intrusion prevention, information- sharing, and analytics capabilities of NCPS. As of April 2015, the projected total life-cycle cost of the program was approximately $5.7 billion, through fiscal year 2018. The overarching objectives of NCPS are to provide functionality that supports intrusion detection, intrusion prevention, analytics, and information sharing. While NCPS’s ability to detect and prevent intrusions, analyze network data, and share information is useful, its capabilities are limited. For example, NCPS detects signature-based anomalies, but does not employ other, more complex methodologies and cannot detect anomalies in certain types of traffic. Further, the intrusion prevention capabilities can currently mitigate threats to a limited subset of network traffic. Regarding NCPS’s analytics function, DHS has deployed aspects of this capability and plans to develop more complex tools. However, information sharing, which is the fourth stated objective, has only recently been approved and funded for development; thus, current information sharing efforts are manual and largely ad hoc. In addition, although DHS established a variety of NCPS-related metrics, none provide insight into the value derived from the functions of the system. Developing such metrics poses a challenge for the agency, according to DHS officials. Until NCPS’s intended capabilities are more fully developed, DHS will be hampered in its abilities to provide effective cybersecurity-related support to federal agencies. NCPS’s intrusion detection capability is intended to provide DHS with the ability to scan network traffic for signs of potentially malicious activity. Effective intrusion detection provides an organization with the ability to detect abnormalities within network traffic and can be accomplished through the use of multiple types of intrusion detection methodologies. In order to more comprehensively and accurately detect malicious activity, NIST recommends using a combination of three detection methodologies: signature-based, anomaly-based, and stateful purpose analysis. Signature-based intrusion detection is able to detect malicious traffic by comparing current traffic to known patterns of malicious behavior, also referred to as signatures. This method is considered effective at detecting known threats and is the simplest form of intrusion detection, because it can only match against known patterns of malicious traffic. The anomaly-based and stateful purpose detection methodologies are more complex approaches, which involve comparing current network activity to predefined baselines of “normal behavior” to identify deviations which could be indicative of malicious activity. These approaches to intrusion detection are more effective than signature- based detection at identifying previously unknown threats, such as “zero-days,” as well as variants to known threats and threats disguised by the use of evasion techniques. NCPS uses only a signature-based methodology for detecting malicious activity. According to US-CERT officials, NCPS’s intrusion detection capability is supported by 228 intrusion detection sensors placed throughout the .gov network infrastructure. The sensors provide a flow of network traffic to be analyzed. Officials added that there are over 9,000 intrusion detection signatures deployed within NCPS, with approximately 2,300 that are enabled and being used to evaluate traffic at any given time. A majority of the signatures are available through commercially available products, though a portion is custom developed. According to DHS documentation and NSD officials, NCPS was always intended to be a signature-based intrusion detection system, and thus it does not have the ability to employ multiple intrusion detection methodologies. Further, NSD and US-CERT officials stated that NCPS is just one of the many tools available to federal agencies to help enhance their cybersecurity posture. They stated that it is the responsibility of each agency to ensure their networks and information systems are secure while it is the responsibility of DHS to provide a baseline set of protections and government-wide situational awareness, as part of a defense-in- depth information security strategy. By employing only signature-based intrusion detection, NCPS is unable to detect intrusions for which it does not have a valid or active signature deployed. This limits the overall effectiveness of the program. Moreover, given that many federal agencies use commercially available signature- based intrusion detection systems to support their information security efforts, the addition of another signature-based intrusion detection system may do little to provide customer agencies with a baseline set of protections. DHS officials acknowledged that the intrusion detection systems used by many federal agencies likely have more signatures deployed than NCPS. Thus, the agencies’ intrusion detection systems would be able to compare their network traffic against a larger set of potential exploits, such as exploits that US-CERT determined no longer needed to be scanned by NCPS. In other cases, US-CERT officials stated, some agencies do not possess their own robust intrusion detection capability and thus rely more on the intrusion detection functionality provided by NCPS. Regarding zero-day exploits, US-CERT officials stated there is no way to identify them until they are announced. Once they are announced, US- CERT can develop a signature, as was the case with Adobe Flash exploits that were recently publicly announced. While there are sources that can be used to buy zero day exploits, officials stated that DHS does not pay for zero days. Occasionally, US-CERT will receive notifications of exploits from partners before they go public, but these are mostly malware notifications. While we acknowledge the challenge of developing signatures for zero-day exploits, enhancing NCPS’s current intrusion detection approach to include functionality that would support the development of a baseline of network behavioral analysis, as described in NIST 800-94, would enhance DHS’s ability to combat such threats. According to NIST, many intrusion detection products have the ability to detect attacks carried out through various types of network traffic, such as traffic related to network browsers, e-mail, and file transfer, as well as traffic related to supervisory control and data acquisition (SCADA) control systems. In addition, intrusion detection systems should also have the ability to detect malicious activity across multiple layers of network protocols, including Internet Protocol Version 6 (IPv6). Further, NIST states that some intrusion detection products have the ability to detect characteristics of encrypted traffic (i.e., whether encryption had been applied) but not evaluate the traffic itself. Adversaries will often use encryption to mask malicious traffic to help better facilitate the successful execution of cyber-exploits, such as zero-day attacks. However, NCPS is not currently evaluating all types of network traffic. NSD and US-CERT officials stated there are currently no signatures deployed with NCPS that will detect threats embedded in certain types of network traffic. US-CERT officials stated that they have not deployed signatures related to these additional types of network traffic for various reasons. They stated that NCPS customer departments and agencies have not been clear on the details of the specific types of network traffic present within their organizations or the amount of traffic allowed to pass through their network gateways. According to an NSD official, they initially collect such data and hold meetings with officials from customer departments and agencies to exchange technical information, but the departments and agencies are responsible for routing network traffic to the NCPS sensors and not required to keep DHS abreast of changes. In addition, US-CERT officials stated that they have not conducted a detailed analysis of customer departments’ and agencies’ traffic to gain this understanding. Further, US-CERT officials stated that they were not equally concerned with the risk posed by all types of network traffic. Without an ability to analyze all types of traffic, DHS is unable to detect threats embedded in such traffic and increases the risk that agencies could be negatively impacted by such threats. According to NIST, signature-based intrusion detection systems depend on the quality of the signatures contained within them, and thus need to be updated to reflect new vulnerabilities and exploits that emerge. Organizations can purchase signatures from commercial vendors, custom develop them, or obtain them from open sources. NIST maintains the National Vulnerability Database (NVD), which is an open source of information that can influence many information security activities, including the development of intrusion detection signatures. Federal agencies are encouraged to use the information contained within the database as part of their information security efforts. In addition, US-CERT has acknowledged the importance of incorporating the use of common vulnerabilities and exposures (CVE) information in information security activities. In April 2015 US-CERT issued an alert which stated that cyber threat adversaries continue to exploit unpatched software products from vendors such as Adobe, Microsoft, and Oracle. Vulnerabilities in these products are often a common vector for spear phishing attacks. The alert stated that as many as 85 percent of these attacks are preventable through the implementation of patches. Accordingly, the bulletin contained 30 of the top targeted vulnerabilities and associated CVE information that security officials could use to implement within their organizations. However, the signatures supporting NCPS’s intrusion detection capability only identify a portion of vulnerabilities associated with common software applications from vendors such as Adobe, Microsoft, and Oracle. Specifically, we found that NCPS had limited coverage of vulnerabilities associated with 10 common client and server applications we evaluated. At the time of our review, NCPS intrusion detection capability signatures provided: partial coverage for 7 vulnerabilities, and no coverage for 2 vulnerabilities. reasonable coverage for 1 vulnerability, Further, for the 12 advanced persistent threats we evaluated, NCPS’s intrusion detection capability had signatures that at the time of our review provided: partial coverage for 4 advanced persistent threats. reasonable coverage for 8 advanced persistent threats, and More specifically, for the five client applications we reviewed (Adobe Acrobat, Flash, Internet Explorer, Java, and Microsoft office), the NCPS intrusion detection signatures provided some degree of coverage for approximately 6 percent of the total vulnerabilities selected for review (i.e., 29 of 489), with coverage for specific applications ranging from 1.2 to 80 percent of vulnerabilities identified in CVE reports published during 2014. Further, it is unknown how, if at all, US-CERT plans to leverage vulnerability data from other DHS sources to influence the development of intrusion detection signatures. For example, the Federal Network Resilience division is responsible for managing the Continuous Diagnostics and Mitigation program. The vulnerability information garnered from this program could be used to develop signatures that would target exploits that are affecting many federal agencies. US-CERT officials stated that they plan to use this information to influence NCPS, but could not provide specific details as how they plan to accomplish this due to the relative immaturity of the Continuous Diagnostics and Mitigation program. One reason that the signatures did not cover all identified vulnerabilities is that the current tool DHS uses to manage and track the status of intrusion detection signatures deployed within NCPS does not have the ability to capture CVE information. US-CERT officials stated that when they developed the Signature Management System tool, they were not required to create a link between a signature and a published CVE data. However, US-CERT has acknowledged this deficiency and stated this is something it plans to address in the future. In addition, US-CERT officials agreed with the results of our analysis of client vulnerabilities, but reiterated that the goal of NCPS was not to protect against all vulnerabilities. US-CERT officials stated that agencies with their own internal intrusion detection systems would likely be able to comprehensively address the common client vulnerabilities we selected. US-CERT officials stated that the overall intent of the system was to protect against nation-state level threat actors who often leverage “zero- day” exploits which may not have had a known mitigation or specific CVE assigned. Accordingly, officials stated, they must consider input from a variety of classified and unclassified sources, in addition to open source data such as CVEs, when developing their intrusion detection signatures. However, NCPS did not possess intrusion detection signatures that fully addressed all the advanced persistent threats we reviewed, which are often a type of exploit leveraged by nation-state actors. US-CERT officials added that they must consider a variety of factors when deciding which specific signatures to deploy and the length of time they keep the signatures active. For example, the current version of the software managing the intrusion detection function does not allow for custom rules at each sensor. As a result, the signatures deployed must be uniform across all sensors and cannot be tailored to a specific agency. This adds an additional layer of complexity when deciding how long to deploy signatures. For example, a smaller agency may be unaware of a particular threat or associated signature, and thus could benefit from having that signature deployed longer than a larger agency, which may view it as potentially duplicative of signatures employed by its own internal intrusion detection tool. Officials stated that they expect this issue to be addressed when they upgrade to the next version of the software that manages the intrusion detection function. We acknowledge that NCPS’s intrusion detection capabilities draw on many sources of vulnerability information and it should not necessarily duplicate capabilities that agencies already possess. However, updating the tool used to manage NCPS signatures to draw on and more clearly link to publicly available, open-source repositories of vulnerability information, such as the NVD, and using data from the Continuous Diagnostic and Mitigation program as they become available as an input into the development and management of signatures could add value by minimizing the risk that known vulnerabilities will be exploited. NCPS’s ability to provide intrusion prevention is another key objective of the system. Intrusion prevention is an additional technique recommended by NIST in support of effective information system monitoring. When fully developed, NCPS will have the ability to proactively mitigate threats across multiple types of network traffic. This is important because malicious actors can propagate threats across multiple vectors and types of network traffic. NCPS’s intrusion prevention capability provides DHS with the ability to proactively address network-based threats before they can potentially cause harm to federal networks. This is accomplished by monitoring network traffic to and from a customer agency’s network and taking some action to stop traffic (e.g., blocking an e-mail) that has characteristics matching pre-defined indicators of malicious traffic. NCPS has the ability to prevent intrusions in near real time, but is currently only able to proactively mitigate threats across a limited subset of network traffic (i.e., Domain Name System, or DNS, blocking and e- mail filtering) at a selected group of customer agencies. Consequently, there are other types of network traffic (e.g., web content), which are common vectors of attack not currently being analyzed for potentially malicious content. NSD officials noted that initial capabilities for intrusion prevention were intended to be more robust, but were scaled back due to a change in the program’s approach. Specifically, these officials stated that the original intent of the intrusion prevention deployment was to protect all types of network traffic with classified indicators. Further, the solution was supposed to provide government-furnished equipment to Internet service provider networks as the backbone of the intrusion prevention function of NCPS. However, NSD officials stated that, due to the excessive costs of operating and maintaining the original solution, the agency decided in May 2012 to change approaches. The new approach follows a managed service model, where the Internet service providers would receive classified indicators within their appropriate facilities and manage the prevention for their particular customer agencies. Because of this managed service model approach, NSD officials stated that the first set of prevention capabilities was based in part on the existing solutions provided by commercial service providers under the Defense Industrial Base Opt-In pilot (later renamed the Enhanced Cybersecurity Services). Officials also added that another motivation for the new approach was to create a cybersecurity marketplace where the various Internet service providers would compete with each other to provide better cybersecurity solutions for federal customers. DHS officials stated that they are developing prevention capabilities for other types of network traffic. Specifically, NSD officials stated that they plan to introduce the ability to filter web content by January 1, 2016. A review of a recent monthly report from one of the Internet service providers supporting NCPS intrusion prevention indicated that the contractor has begun work on web content filtering and provided DHS with a draft report on the indicators and overall process. Another key objective of NCPS is to provide DHS with an analytics capability. NIST recommends that organizations take a variety of actions with respect to analytics, including analyzing and correlating audit records across different repositories to gain organization-wide situational awareness, correlating information from nontechnical sources with audit information to enhance organization-wide situational awareness, analyzing the characteristics of malicious code, and employing automated tools to support near real-time analysis. The functionality deployed in support of NCPS analytics capability developed to date is in accordance with recommended standards. For example, the security information and event management solution, which has been operational since February 2012, simplifies cyber analysis by providing a centralized platform in which the log data from similar events can be aggregated, thereby reducing duplication. The tool also facilitates analysts’ ability to correlate related events that might otherwise go unnoticed and provides visualization capabilities, making it easier to see relationships. Additionally, NSD has established functionality that enables the analysis of the characteristics of malicious code. For example, the Packet Capture tool enables US-CERT analysts to see “inside” the packet, and inspect the payload to analyze a specific cyber threat. Further, the Digital Media Analysis Environment (Forensics) and Advanced Malware Analysis Center provide mechanisms to collect and contain information on cyber threats in a highly secure environment for evaluation by US-CERT analysts. NSD and US-CERT officials stated that DHS initially focused funding and development efforts on analytical functions associated with supporting the intrusion detection and prevention functions of NCPS. However, the more complex analytics development is planned for later stages of system development. Specifically, DHS has enhancements planned through fiscal year 2018. These planned enhancements are intended to better facilitate the near real-time analysis of various data streams and advanced malware behavioral analysis, and to conduct forensic analysis in more collaborative way. Information sharing is a key control recommended by NIST in support of effective information system security. Additionally, the presence of good information sharing, particularly the ability to effectively notify an affected entity of potentially malicious activity, is a key component of effective intrusion detection and prevention, and thus a key objective of NCPS. Further, NIST states that organizations should develop standard operating procedures to ensure that consistent and accurate information is available for reporting and management oversight. Also, US-CERT’s Concept of Operations for NCPS establishes monitoring the status of mitigation actions and strategies as a requirement of the program. NSD officials stated that the information-sharing capability has only recently been approved and funded for development, and thus current information-sharing efforts are manual and largely ad hoc. DHS first requested funding for the development of information-sharing capabilities in 2010, but NSD officials stated the effort was given a lower priority than the intrusion prevention capability and was not funded to begin planning activities until 2014. As a result, DHS has yet to develop a majority of planned functionality for the information-sharing capability of NCPS. Though the operational requirements for the NCPS information-sharing functionality were approved in November 2014, DHS did not formally authorize NSD to initiate development of the capability until August 2015. As a result no substantive actions have yet been taken to develop this capability. Regarding the current information-sharing efforts, officials from the five customer agencies we reviewed stated that DHS is not always effectively communicating its intrusion detection notifications to customer agencies. Specifically, DHS officials provided evidence that they sent 74 incident notifications that they believed were related to NCPS to the five agencies in our review during fiscal year 2014. However, evidence provided by the agencies showed that only 56 of these notifications had been received by the customer agencies. The five impacted agencies and DHS disagreed as to whether the other 20 incident notifications had been sent and received. Specifically, for 18 of 20 these notifications, DHS provided evidence that an e-mail may have been sent, but the agencies had no record of receiving the notifications. For the 2 additional notifications, one customer agency had a record of receiving them; however, DHS had no evidence of transmitting the e-mails. For the 56 NCPS-related notifications that the five agencies acknowledged receiving, the agencies stated that 31 incidents notifications were timely and useful, 10 incidents notifications were not timely or useful, 7 incident notifications were identified by agency officials as false 7 incident notifications were not related to an NCPS intrusion- detection. Additionally, DHS did not always solicit, and agencies did not always provide, feedback on the notifications. Specifically: Of the 56 incident notifications mentioned above, DHS requested that the impacted agency provide feedback on 36 of them. Of these 36, the agencies stated that they provided feedback on 15 notifications, but did not provide feedback on 21. For an additional 10 notifications, officials from 3 of 5 agencies stated they provided feedback even though DHS had not explicitly requested follow-up action. For an additional 10 notifications, DHS did not request feedback and the customer agencies did not provide any. As another channel for sharing information, US-CERT holds weekly calls with representatives of the security operation centers of various federal agencies. These calls provide a forum for the voluntary exchange of a variety of information security information, including NCPS-related information. Officials from the five customer agencies involved in our review expressed value in the information received from these discussions. One reason DHS and agencies do not agree about whether notifications were received may be that DHS does not always explicitly ask for feedback or confirmation of receipt of the notification. Additionally, officials from one customer agency stated that DHS has no way of determining which of its analysts were responsible for transmitting a particular notification, so it is difficult to obtain context after a notification is sent. US-CERT officials stated that standard operating procedures and a quality control procedure are being developed as part of the implementation of a new version of the incident management database. However, these procedures were not developed during the scope of our review, fiscal year 2014. In August 2015, US-CERT provided us with a draft standard operating procedure related to the incident notification process. The policy provides an overview of the types of questions a US- CERT analyst should ask a customer agency when transmitting a notification. However, the draft policy does not instruct them specifically to include a solicitation of feedback within the notification. Further, US- CERT could not provide any information regarding the timetable for when these procedures would take effect. Regarding the usefulness of the notifications, two of the agencies in our review stated that because of the placement of the intrusion detection sensors on their networks, a significant amount of effort was required to evaluate the context of the DHS notifications. Thus, both agencies stated, and DHS agreed, the value of the notifications could be enhanced by giving US-CERT analysts access to the agencies’ network diagrams, which could allow them to identify the specific location of the intrusion. Officials from customer agencies stated that they did not provide feedback for a variety of reasons. For example, one agency stated that due to its federated nature, getting a response from the impacted entity within their agency was a challenge and could only be rectified by reaching out to site owners for every incident notification they receive. Consequently, officials stated, they typically only reached out when the notification had met the threshold of a security event. Officials from this agency stated that they had instituted a new standard operating procedure that requires the analyst processing the incident notification to reach out to DHS prior to closing it out. They added that this policy went into effect after fiscal year 2014 and did not impact the data set we reviewed. An additional agency stated that a request for feedback is not always clearly stated within the notification they receive from US-CERT. Without verifying the receipt of intrusion detection notifications and soliciting feedback on their usefulness, DHS may be hindered in assessing the effectiveness of NCPS’s current information-sharing capabilities. According to NIST, a number of laws—including the Federal Information Security Management Act—cite performance measurement in general, and information security performance measurement in particular, as a requirement. Further, NIST 800-55 states that an information security measurement program provides a number of organizational and financial benefits, including increased accountability for information security performance, improved effectiveness of information security activities, demonstrated compliance with laws, and quantified inputs and allocation decisions. Further, effectiveness or efficiency measures are used to monitor if program-level processes and system-level security controls are implemented correctly, operating as intended, and meeting the desired outcome. Metrics for NCPS, as provided by DHS, do not provide information about how well the system is enhancing government information security or the quality, efficiency and accuracy of supporting actions. DHS has established three department-wide NCPS-related performance metrics, as part of its Performance and Accountability report: Percentage of traffic monitored for cyber intrusions at civilian federal executive branch agencies: According to Executive Program Management Office and NSD officials, this measure assesses NCPS’s intrusion detection capability by providing information on the scope of coverage for potentially malicious cyber-activity across participating civilian federal government agencies. During fiscal year 2014, DHS reported that approximately 88.5 percent of the total Internet traffic of 23 civilian, executive branch agencies was monitored by NCPS intrusion detection sensors. Though this metric provides insight into the amount of federal executive-branch traffic that NCPS is able to provide intrusion detection for, it does not provide insight into the quality or efficiency of the intrusion detection function for that traffic. Percentage of incidents detected by US-CERT that targeted agencies are notified of within 30 minutes: According to Executive Program Management Office and NSD officials, this is an additional measure of NCPS’s intrusion detection capability. Specifically, DHS documentation stated that there were 297 cyber incidents identified on federal networks using the NCPS’s intrusion detection capability in fiscal year 2014. The average time to notify impacted agencies was 18 minutes, with 87.2 percent (259 of 297) of notifications occurring within 30 minutes. While this metric provides insight into the speed at which DHS could share information related to detected incidents, it does not provide a measure for the accuracy or value of those notifications. Further, of 24 incident notifications for the five selected agencies that support this metric, DHS could not provide evidence that 12 of these notifications were sent. Without appropriately sharing the notifications with the affected agency, we are unsure how DHS classifies these 12 notifications as incidents. Percentage of known malicious cyber traffic prevented from causing harm at federal agencies: According to Executive Program Management Office and NSD officials, this measure assesses NCPS’s intrusion prevention capability. Specifically, DHS documents stated that each currently deployed indicator of a malicious threat is paired with a countermeasure to prevent the malicious threat from harming those networks. In fiscal year 2014, 389 indicators were deployed amongst intrusion prevention sensors. Though this metric would track whether a particular countermeasure was engaging (i.e., if prevention occurred) it does not necessarily evaluate the effectiveness or efficiency of the intrusion prevention capability. DHS officials agreed with this observation and stated that the agency was in the process of retiring this metric and developing a new one that would better measure and evaluate the effectiveness of intrusion prevention. Further, NSD has established key performance parameters that provide an indication of the system’s ability to perform functions supporting NCPS’s objectives. For example, the following measures were developed to track the performance of the intrusion detection function: Detect known cyber events through automated intrusion detection within 1 minute of event occurrence. Provide automated notification within the operations center that a cyber event took place within 1 minute of event detection. Aggregate and correlate detected cyber events for known indicators within 30 minutes of event notification. While these are valuable for determining how NCPS is operating as a system, officials from the Executive Program Management Office and NSD agreed that they did not provide a qualitative or quantitative assessment of the system’s ability to fulfill the aforementioned objectives. Further, as we reported in April 2015, a DHS acquisition official questioned whether the NCPS key performance parameters were defined properly. Regarding the system’s benefits, NSD and US-CERT officials stated that the total amount of incident notifications sent to customer agencies does indicate that NCPS is providing value. However, as our analysis of a selected group of customer notifications from fiscal year 2014 indicates, customer agencies do not perceive every notification transmitted as valuable. Without the deployment of comprehensive measures, DHS cannot appropriately articulate the value provided by NCPS. While DHS developed an executive road map for the intrusion detection, prevention, analytics, and information sharing objectives that describes future NCPS capabilities to be developed through fiscal year 2018, it has not defined requirements, as called for by OMB guidance and best practice, for two intrusion detection capabilities to be provided in fiscal year 2016. In addition, although DHS officials stated that they do consider threat information as part of the required risk-based approach for determining future capabilities to protect federal information systems, they do not consider specific vulnerabilities affecting agencies’ networks and systems, as information on these is not currently available. The lack of vulnerability information prevents DHS from taking a full risk-based approach to designing future NCPS intrusion prevention capabilities. OMB’s Capital Programming Guide states that requirements should be developed to support program budgeting activities. The guidance also states that agencies should avoid “specification creep,” where requirements become uncontrolled by defining requirements to meet future potential needs or incorporating emerging technology that would be “nice” to have. Further, a recognized best practice in requirements development from the Software Engineering Institute notes that requirements should be expressed in a way that can be used for design decisions. NSD maintains a road map that is used to track potential additional capabilities for NCPS’s intrusion detection, intrusion prevention, analytics, and information-sharing objectives to be developed in future fiscal years, up to fiscal year 2018. For each NCPS objective, the Executive Road Map identifies the current state of operations (“as-is”) and the desired state of operations (“target”). According to NSD officials, this road map facilitates discussions with senior DHS management, and is revised at several points in the fiscal year. The road map identifies technology and techniques that may increase the department’s ability to perform activities to support the four objectives. For example, DHS plans to begin work on a “web gateway proxy scan encryption” capability in fiscal year 2016. DHS also plans to seek funding for a wireless network protection capability in fiscal year 2018, which may add an additional type of intrusion detection and prevention technology described in guidance issued by NIST. Requirements have not been fully defined for all items in the road map. Specifically, two capabilities DHS stated will be provided in fiscal year 2016—expanding the intrusion detection capability to identify malware present on customer agency internal networks and identifying malicious traffic entering and exiting cloud-based service provider services—are based on requirements that have not been fully defined. NSD officials stated that these capabilities were based upon the requirement to detect intrusion attempts in near real time across the federal government. They added that identifying malware on customer agency internal networks and malicious traffic entering and exiting cloud- based service providers is a logical expansion of responding to the cyber threat, and the program office needs flexibility to adapt to the threat. However, these capabilities could represent a significant departure from the version of NCPS currently deployed and envisioned in the governance documents. Specifically, the technical nature of cloud computing—where customer agency data may be stored and accessed by multiple physical sites—and the number of cloud service providers that could be used by customer agencies may require a different infrastructure deployment methodology than the existing NCPS sensor deployments at Internet service providers and at customer agency locations. Further, while the Executive Road Map indicates that NCPS will detect malware on customer agency internal networks using log data from DHS’s Continuous Diagnostics and Mitigation program, it is unclear how DHS plans to accomplish this. Until it fully defines requirements for these two capabilities, DHS increases the risk that it will invest in functionality that does not effectively support information security efforts at the customer agencies and across the federal government. The Federal Information Security Modernization Act of 2014 and guidance issued by NIST call for a risk-based approach to protecting federal systems. According to NIST’s Guide for Conducting Risk Assessments, information security risk is assessed by considering the threats posed to the federal government, the vulnerabilities (or weaknesses) in information systems, the impact (or harm) that could occur given the potential for threats exploiting vulnerabilities, and the likelihood that threats would use the exploits to allow harm to occur. DHS has incorporated selected elements of a risk-based approach when considering the next capabilities of the NCPS intrusion prevention objective. Specifically, NSD coordinated and leveraged threat information from the National Security Agency and the National Cybersecurity and Communications Integration Center, along with information provided by the Internet service providers, to develop a list of countermeasures that DHS believed would be reasonable to implement. NSD officials stated that decisions regarding existing and upcoming countermeasures were made based on the capabilities of the Internet service providers. Specifically, the e-mail and DNS countermeasures were used as the first countermeasures for the NCPS intrusion prevention capability because they were already deployed at Internet service providers as part of the Enhanced Cybersecurity Services program. However, NSD did not consider and does not currently have access to vulnerability information for the agency information systems it is helping to protect. NSD officials stated that vulnerability data about customer agency information systems and networks are difficult to obtain. For example, agency information security reports required under the Federal Information Security Modernization Act of 2014 do not contain vulnerability information that NSD could use to inform future capabilities. Also, NSD officials stated—and the Executive Road Map confirmed—that DHS’s Continuous Diagnostics and Mitigation program may provide additional vulnerability information that could be valuable in determining future capabilities. However, at this time the program is relatively immature and NSD had not developed processes and procedures on how to use this vulnerability information to inform decisions on future capabilities at the time of our review. Further, DHS also has a separate program to collect vulnerability information on federal executive branch agency systems and networks that could be useful for determining future NCPS intrusion prevention capabilities. In an October 2014 memo, OMB directed DHS to scan Internet-accessible addresses and public-facing segments of federal civilian agency systems for vulnerabilities on an ongoing basis, and report to OMB on the identification and mitigation of vulnerabilities across federal agency information systems. However, NSD did not provide details—including processes and procedures—of how this information could be used to inform future NCPS intrusion prevention capabilities. Until the department develops processes and procedures for using such vulnerability information, DHS will not be able to adopt an effective risk- based approach for planning future NCPS intrusion prevention capabilities. OMB Memorandum M-08-05 established the requirement for almost all federal executive branch agencies to implement the intrusion detection capabilities (within Einstein 2) of NCPS. In July 2015, the White House noted that deployment of the NCPS intrusion prevention capabilities was to be accelerated, with DHS awarding a contract to provide intrusion prevention services for all federal civilian agencies by the end of 2015. Agencies have had mixed results in adopting NCPS capabilities. According to DHS program documentation, all 23 of the non-defense CFO Act agencies had routed traffic to NCPS intrusion detection sensors. However, NSD documents indicated that only 5 of the 23 agencies were receiving intrusion prevention services. Further, NSD documents showed that for 3 of these 5 agencies, adoption of intrusion prevention services for e-mail was limited—only 1 agency appeared to have fully adopted intrusion prevention for e-mail service, another agency had adopted intrusion prevention for only one part of its network e-mail, and a third agency was just beginning to adopt the e-mail service. Further, four of the five selected agencies in our review reported that not all of their traffic was being sent to NCPS intrusion detection sensors. In addition, two of the selected agencies reported that they had adopted the DNS intrusion prevention service, and only one had completed the adoption process for its e-mail service. See table 4 below for a summary of NCPS intrusion detection and prevention adoption at the selected five agencies. Officials from NSD, the selected agencies in our review, and the Internet service providers identified several policy and implementation challenges to adopt the NCPS intrusion prevention capabilities, along with efforts to address these challenges: Approval of memoranda of agreement (MOA): An MOA is required in order to establish NCPS service for an agency. Among other things, the MOA identifies responsibilities for both DHS and the customer agency (including interactions with Internet service providers), as well as identifies points of contact for the respective organizations. Sixteen of the 23 non-defense CFO Act agencies had an MOA in place with DHS to provide intrusion prevention services. Three of the five agencies in our review were in the process of approving an MOA for intrusion prevention services and cited barriers to approving the agreement. Specifically, two of these agencies did not sign an agreement because their Internet service providers had not been capable of providing NCPS intrusion prevention services. NSD officials stated that they are in the process of accelerating the availability of Internet service providers to those agencies which are not currently provided NCPS intrusion prevention capabilities. Officials from the third agency stated that there were questions whether existing law protecting sensitive information in its possession prohibited participation in NCPS intrusion prevention or not. In July 2015, officials from this agency stated that, working with DHS, it had agreed to adopt some NCPS intrusion prevention capabilities. Agency capabilities and concerns: NSD officials noted that the ability to meet DHS security requirements (e.g., encrypted tunnels) to use the intrusion prevention capabilities varies from agency to agency. NSD officials also stated that because each agency has unique network infrastructures, implementation must be specific to that agency. Further, NSD officials added that agencies are generally concerned about interfering with any mission-critical applications, such as e-mail. Also, while chief information officers usually sign the MOA, NSD officials noted that network operators within the agency can be unaware of the agreement, which can pose a potential barrier to full deployment. To address these issues, NSD staff stated that they work with agencies to tailor implementation and explain details of the prevention capabilities to reassure them that business operations will not be impeded. Additionally, officials from one agency in our review that had adopted the DNS intrusion prevention capability initially hesitated to adopt the e-mail capability due to records management concerns. Agency officials stated in July 2015 that they are in the process of working with DHS to adopt the e-mail intrusion prevention capability. Viability of solution for cloud e-mail: Officials from one agency in our review stated that they obtain e-mail services from cloud providers, and added that they hesitate to participate in NCPS intrusion prevention e-mail capability because there is currently no solution that is easily implemented. Officials from another agency in the process of signing an MOA stated that they also use cloud service providers for e-mail. This agency will also not be able to implement the e-mail intrusion prevention capability. NSD has noted the challenges associated with implementing a cloud solution, but plans to refine this capability over time. However, as we previously stated, the plans to initiate development efforts on a cloud solution during fiscal year 2016 are not based on fully developed requirements. Development and operational challenges at Internet service providers: NSD and two of the Internet service providers noted a challenge with designing, developing, and operating a classified infrastructure on unclassified network traffic. For example, the complex and changing security requirements of one of DHS’s partners who provides threat information created delays in the service providers’ ability to deliver intrusion prevention capabilities. In addition, obtaining and retaining personnel with appropriate security clearances posed a challenge for the Internet service providers. NSD has acknowledged the inherent complexity of using classified information to address cyber risks in non-classified network traffic and has ongoing efforts to work with the Internet service providers to address this. Further, NCPS faces additional implementation challenges in ensuring that agency traffic is sent to the intrusion detection sensors. Specifically, four of the five agencies in our review cited several challenges in routing all of their traffic through NCPS intrusion detection sensors, including capacity limitations of the sensors, agreements with external business partners that use direct network connections, interagency network connections that do not route through Internet gateways, use of encrypted communications mechanisms, and backup network circuits that are not used regularly. NSD officials stated that agencies are responsible for routing their traffic to the intrusion detection sensors, and DHS does not have a role in that aspect of NCPS implementation. NCPS also faces a challenge in implementing a portion of the intrusion detection capability and all of the intrusion prevention capability when routing traffic through sensors at the Internet service providers. Of the five agencies in our review, four depend on their Internet service provider to receive NCPS intrusion detection services (through the Managed Trusted Internet Protocol Service program) and/or intrusion prevention services. Two of these four agencies had taken steps to securely route traffic to the sensors, while one agency did not implement an authentication mechanism to ensure that network routes received by their router were legitimate. The other agency stated that its Internet service provider managed its routing configurations and did not provide evidence for us to verify if secure routing configurations were in place. This occurred in part because NSD did not provide guidance to customer agencies on how to securely route their information to the Internet service providers. NSD officials stated that providing network routing guidance to customer agencies is not the role of DHS. Rather, they believe that is best handled by the customer agency and their Internet service provider. However, without providing network routing guidance, NSD has no assurance the traffic they see constitutes all or only a subset of the traffic the customer agencies intend to send. Further, by not providing routing guidance, NSD has less assurance that customer agency traffic will actually be picked up at the sensors, since the routing may bypass those sensors, reducing the effectiveness of NCPS. DHS has devoted significant resources to developing and deploying NCPS, with the goal of strengthening agencies’ ability to detect and prevent intrusions on their networks, as well as the capability for analyzing network activity and sharing information between DHS and agencies. The system’s intrusion detection capabilities are the most fully developed of the four system objectives, and they provide the ability to detect known malicious patterns of activity on agency networks. However, without the ability to effectively detect intrusions across multiple types of traffic or provide other types of detection capabilities, such as anomaly- based and stateful purpose detection, NCPS is limited in its ability to identify potential threats. In addition, without making use of publicly available, open-source repositories to enhance the system’s signatures and data available from its Continuous Diagnostics and Mitigation program, DHS may not be providing the ability to detect attacks that exploit known vulnerabilities. The system’s intrusion prevention capability is less fully developed, with limited deployment across different types of network traffic, such as content from websites, limiting its ability to prevent malicious code from penetrating agencies’ networks. Further, NCPS’s support of a number of analytics capabilities, and ongoing efforts to enhance these, should provide DHS and agencies with improved ability to analyze potentially malicious traffic in a timely and efficient manner. However, DHS’s sharing of information with agencies has not always been effective, with disagreement among agencies about the number of notifications sent and received and their usefulness. Finalizing the incident notification process, to include the solicitation of feedback from customer agencies, could help ensure that DHS is effectively communicating information that helps agencies strengthen their security posture. Another step that could assist in ensuring the effectiveness of NCPS is developing metrics that measure the quality, efficiency, and accuracy of the services it provides. DHS has continued to plan for future capabilities of the system, but without clearly defined requirements, it risks investing in functionality that does not effectively support agency information security. Moreover, to ensure a risk-based approach is being pursued to select future NCPS capabilities, information about vulnerabilities on agency networks could be a valuable input. The effectiveness of NCPS further depends on its adoption by agencies. While the adoption of the intrusion detection capabilities is widespread among the 23 agencies required to use NCPS, the implementation of intrusion prevention capabilities is more limited due to policy and implementation challenges that DHS is working to overcome. However, addressing a lack of guidance for routing network traffic through NCPS sensors could help better ensure a wider and more effective use of NCPS capabilities. We recommend the Secretary of Homeland Security direct: NSD to determine the feasibility of enhancing NCPS’s current intrusion detection approach to include functionality that would detect deviations from normal network behavior baselines; NSD to determine the feasibility of developing enhancements to current intrusion detection capabilities to facilitate the scanning of traffic not currently scanned by NCPS; US-CERT to update the tool it uses to manage and deploy intrusion detection signatures to include the ability to more clearly link signatures to publicly available, open-source data repositories; US-CERT to consider the viability of using vulnerability information, such as data from the Continuous Diagnostics and Mitigation program as it becomes available, as an input into the development and management of intrusion detection signatures; US-CERT to develop a timetable for finalizing the incident notification process, to ensure that customer agencies are being sent notifications of potential incidents, which clearly solicit feedback on the usefulness and timeliness of the notification; The Office of Cybersecurity and Communications to develop metrics that clearly measure the effectiveness of NCPS’s efforts, including the quality, efficiency, and accuracy of supporting actions related to detecting and preventing intrusions, providing analytic services, and sharing cyber-related information; The Office of Cybersecurity and Communications to develop clearly defined requirements for detecting threats on agency internal networks and at cloud service providers to help better ensure effective support of information security activities; NSD to develop processes and procedures for using vulnerability information, such as data from the Continuous Diagnostics and Mitigation program as it becomes available, to help ensure DHS is using a risk-based approach for the selection/development of future NCPS intrusion prevention capabilities; and NSD to work with their customer agencies and the Internet service providers to document secure routing requirements in order to better ensure the complete, safe, and effective routing of information to NCPS sensors. We provided a draft of this report to the Departments of Homeland Security, Energy, and Veterans Affairs; the General Services Administration; the Nuclear Regulatory Commission; and the National Science Foundation for their review and comment. In written comments signed by the Director, Departmental GAO-OIG Liaison Office, DHS concurred with each of our nine recommendations. DHS also provided details about steps that it plans to take to address eight of the nine recommendations, including estimated time frames for completion. If effectively implemented, these actions should help address the weaknesses we identified in the NCPS program. Regarding our recommendation to develop clearly defined requirements for detecting threats on agency internal networks and at cloud service providers, the Director asked that we consider it resolved and closed because a formal requirements working group and requirements management process had been developed. We will review the evidence and determine if these actions address the recommendation. DHS’s written comments are reprinted in appendix III. Officials from DHS also provided technical comments via e-mail, which we incorporated as appropriate. Officials from the Departments of Energy and Veterans Affairs, General Services Administration, Nuclear Regulatory Commission, and the National Science Foundation stated that they had no comments. We are sending copies of this report to the appropriate congressional committees, the departments and agencies in our review, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Gregory Wilshusen at (202) 512-6244 or wilshuseng@gao.gov or Dr. Nabajyoti Barkakati at (202) 512-4499 or barkakatin@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Our objectives were to determine the extent to which (1) the National Cybersecurity Protection System (NCPS) meets stated objectives, (2) the Department of Homeland Security (DHS) has designed requirements for future stages of the system, and (3) federal agencies have adopted the system. To determine the extent to which NCPS meets stated objectives, we compared four of the overarching capabilities of the system (intrusion detection, intrusion prevention, analytics, and information sharing) to leading federal practices, including the National Institute of Standards and Technology’s Special Publication 800-53: Security and Privacy Controls for Federal Information Systems and Organizations; Special Publication 800-55: Performance Measurement Guide for Information Security, and Special Publication 800-94: Guide to Intrusion Detection and Prevention Systems (IDPS). We also examined program information and documents, as well as interviewed DHS officials within the Office of Cybersecurity and Communications responsible for designing, developing, maintaining, and operating NCPS. For the information-sharing objective we examined NCPS-related incident notifications DHS stated were sent by the United States Computer Emergency Readiness Team in fiscal year 2014 to five selected Chief Financial Officers Act agencies: the Departments of Energy and Veterans Affairs, the General Services Administration, the National Science Foundation, and the Nuclear Regulatory Commission. These agencies were selected based on information provided by DHS regarding the relative number of NCPS-related incident notifications sent to the agencies (one with a higher amount of notifications, two with around the median amount of notifications, and two with the fewest amount of notifications) and NCPS capabilities received. We also interviewed information security staff from each of these agencies and collected information regarding each agency’s perceived usefulness and timeliness of the incident notifications, along with any feedback provided in response to the notification. To evaluate the intrusion detection signatures deployed, we selected 10 common vulnerabilities from 2014 commonly affecting client and server applications and determined the extent to which the NCPS signatures provided reasonable coverage for the vulnerability the signature was intended to mitigate. Additionally, we conducted a similar evaluation for the signatures associated with a selection of 12 common advanced persistent threats from 2014. Further we evaluated the number of intrusion detection signatures DHS had issued for each client vulnerability during fiscal year 2014 and compared them to the number of signatures from publicly available repositories, such as common vulnerabilities and exposures (CVE) published for each corresponding category of vulnerability during the same period. We then determined the percentage of DHS coverage by comparing the number of signatures DHS had that addressed each of the vulnerabilities to the total number of CVEs released in 2014 for that category. To determine the extent to which DHS has designed requirements for future stages of the system, we reviewed NCPS program planning documentation and interviewed program officials in order to identify how future capabilities are planned. We compared this information to federal guidance for planning and requirements development found in the Office of Management and Budget’s Capital Programming Guide. In addition, for all new capabilities identified for funding in DHS’s fiscal year 2016 funding request (such as expanding information sharing, streaming and near-real- time analytics, and deploying intrusion detection sensors at Internet service providers’ traffic aggregation sites), we determined if formalized requirements supporting these capabilities had been documented and approved in program documentation. Further, we determined if plans for future capabilities to address NCPS’s intrusion prevention objective were determined using a risk-based approach, including a consideration of threat, vulnerability, impact, and likelihood. To determine the extent to which federal agencies have adopted NCPS, we reviewed policy issued by the Office of Management and Budget and DHS documentation (such as memoranda of agreement) for the 23 non- defense agencies identified in the Chief Financial Officers Act. We also discussed any challenges to adoption with DHS officials. To gain a better understanding of how federal agencies adopt the system, including the amount of traffic and any challenges or limitations associated with adoption, we interviewed officials from the five Chief Financial Officers Act agencies identified previously and reviewed agency network documentation. We also interviewed officials from the three Internet service providers currently participating in NCPS to obtain their perspective on agency adoption of the system. We conducted this performance audit from June 2014 to January 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The National Cybersecurity Protection System (NCPS), operationally known as the Einstein program, is an integrated system-of-systems that is intended to deliver a range of capabilities, including intrusion detection, intrusion prevention, analytics, and information sharing. The sensors deployed to support the 2003 version of NCPS, or Einstein 1, collect network flow records of data entering and exiting participating agencies’ networks, which are to be analyzed by U.S. Computer Emergency Readiness Team (US-CERT) analysts and tools to detect certain types of malicious activity. If the system detects malicious activity, US-CERT analysts are to coordinate with the appropriate agencies to support the mitigation of those threats and vulnerabilities. US- CERT also is to use the information from the sensors to create analyses of cross-governmental trends that offer agencies an aggregate picture of external threats against the federal government’s networks. In 2009, the Department of Homeland Security (DHS) incorporated network intrusion detection technology into the capabilities of the initial version of the system, enabling NCPS to monitor Einstein 1 network data from participating federal agencies for specific predefined patterns of known malicious activity, referred to as signatures. The NCPS intrusion detection capability, or Einstein 2, is to use signatures derived from numerous sources, such as commercial and public computer security information, incidents reported to US-CERT, information from federal partners, and independent US-CERT in-depth analysis. When NCPS’s intrusion detection function detects traffic consistent with malicious patterns denoted by a particular signature, it provides US-CERT analysts with a notification. The analyst is then to investigate the detection to determine if it was in fact an incident and provide mitigation support to the affected agency, as appropriate. In 2013, DHS’s Network Security Deployment division (NSD) began deployment of an initial operational capability of the intrusion prevention function, operationally known as Einstein 3A, which is intended to support DHS’s ability to actively defend .gov network traffic. One of the major components supporting the capability is the “Nest,” which is a classified facility located at each of the participating Internet service providers that is responsible for off-ramping (i.e., routing traffic to the Nest from the agency) and on-ramping (i.e., routing traffic from the Nest back to the Internet) .gov traffic. DHS shares specific indicators of malicious activity with Internet service providers, who then configure the indicators into signatures for testing and implementation and match patterns against established indicators based on known or suspected malicious traffic traveling to or from the participating agencies. Table 5 below highlights additional intrusion prevention functions currently available in NCPS. Once fully deployed across the government, NCPS is intended to leverage available information from commercial and government sources to apply in-line protection measures to a wide set of federal network traffic protocols. When a signature detects a known or suspected cyber threat, NCPS is supposed to act on that threat to stop malicious traffic and prevent harm to the intended targets. Figure 3 provides an overview of how NCPS intrusion prevention capability is designed to work. NCPS’s analytic capability is intended to capture, organize, and analyze data collected from NCPS sensors and other data feeds. Table 6 below highlights key analytics functions currently available in NCPS. These capabilities are expected to enable US-CERT to fuse information and correlate malicious network activities across participating federal executive branch agencies to achieve situational awareness of high- profile cyber threats. US-CERT is responsible for sharing situational awareness about current and potential cybersecurity threats and vulnerabilities with federal agencies, state and local governments, private sector partners, infrastructure owners and operators, and the public. NCPS’s information-sharing capability is intended to enable enhanced sharing of information between DHS and its partners through real-time or near-real-time response; collaboration and coordination; and analysis of network intrusion attempts, suspicious intrusion activity, and analytical best practices. When fully developed, NCPS information sharing is intended to promote the rapid exchange of appropriate cyber threat and cyber incident information among NCCIC cybersecurity analysts and their cybersecurity partners, at multiple classification levels. Further, the capabilities are intended reduce time required to respond to incidents with better coordination and collaboration, and improved efficiencies with more automated information sharing and exposure of analysis capabilities. In addition to the contacts named above, Lon C. Chin, Michael W. Gilmore, Harold Lewis, Christopher Warweg (assistant directors); Andrew Banister, Bradley Becker, Christopher Businsky, Kush K. Malhotra, Lee McCracken, and David Plocher made key contributions to this report.

Summary: Cyber-based attacks on federal systems continue to increase. GAO has designated information security as a government-wide high-risk area since 1997. This was expanded to include the protection of critical cyber infrastructure in 2003 and protecting the privacy of personally identifiable information in 2015. NCPS is intended to provide DHS with capabilities to detect malicious traffic traversing federal agencies' computer networks, prevent intrusions, and support data analytics and information sharing. Senate and House reports accompanying the 2014 Consolidated Appropriations Act included provisions for GAO to review the implementation of NCPS. GAO determined the extent to which (1) the system meets stated objectives, (2) DHS has designed requirements for future stages of the system, and (3) federal agencies have adopted the system. To do this, GAO compared NCPS capabilities to leading practices, examined documentation, and interviewed officials at DHS and five selected agencies. This is a public version of a report that GAO issued in November 2015 with limited distribution. Certain information on technical issues has been omitted from this version. The Department of Homeland Security's (DHS) National Cybersecurity Protection System (NCPS) is partially, but not fully, meeting its stated system objectives: Intrusion detection: NCPS provides DHS with a limited ability to detect potentially malicious activity entering and exiting computer networks at federal agencies. Specifically, NCPS compares network traffic to known patterns of malicious data, or “signatures,” but does not detect deviations from predefined baselines of normal network behavior. In addition, NCPS does not monitor several types of network traffic and its “signatures” do not address threats that exploit many common security vulnerabilities and thus may be less effective. Intrusion prevention: The capability of NCPS to prevent intrusions (e.g., blocking an e-mail determined to be malicious) is limited to the types of network traffic that it monitors. For example, the intrusion prevention function monitors and blocks e-mail. However, it does not address malicious content within web traffic, although DHS plans to deliver this capability in 2016. Analytics: NCPS supports a variety of data analytical tools, including a centralized platform for aggregating data and a capability for analyzing the characteristics of malicious code. In addition, DHS has further enhancements to this capability planned through 2018. Information sharing: DHS has yet to develop most of the planned functionality for NCPS's information-sharing capability, and requirements were only recently approved. Moreover, agencies and DHS did not always agree about whether notifications of potentially malicious activity had been sent or received, and agencies had mixed views about the usefulness of these notifications. Further, DHS did not always solicit—and agencies did not always provide—feedback on them. In addition, while DHS has developed metrics for measuring the performance of NCPS, they do not gauge the quality, accuracy, or effectiveness of the system's intrusion detection and prevention capabilities. As a result, DHS is unable to describe the value provided by NCPS. Regarding future stages of the system, DHS has identified needs for selected capabilities. However, it had not defined requirements for two capabilities: to detect (1) malware on customer agency internal networks or (2) threats entering and exiting cloud service providers. DHS also has not considered specific vulnerability information for agency information systems in making risk-based decisions about future intrusion prevention capabilities. Federal agencies have adopted NCPS to varying degrees. The 23 agencies required to implement the intrusion detection capabilities had routed some traffic to NCPS intrusion detection sensors. However, only 5 of the 23 agencies were receiving intrusion prevention services, but DHS was working to overcome policy and implementation challenges. Further, agencies have not taken all the technical steps needed to implement the system, such as ensuring that all network traffic is being routed through NCPS sensors. This occurred in part because DHS has not provided network routing guidance to agencies. As a result, DHS has limited assurance regarding the effectiveness of the system. GAO recommends that DHS take nine actions to enhance NCPS's capabilities for meeting its objectives, better define requirements for future capabilities, and develop network routing guidance. DHS concurred with GAO's recommendations.