TSA uses security-related technologies to help secure approximately 1.8 million passengers, 1.2 million checked bags, and 3 million carry-on bags on 25,000 flights at roughly 440 federally regulated airports every day. As of August 2015, TSA had deployed about 15,000 units of security-related technology to airports nationwide and anticipates spending a significant portion of its $3.6 billion security capability budget on technologies such as these over the next 5 years. Within TSA, the Office of Security Capabilities (OSC) provides security-related technology solutions through two major DHS acquisition programs, the Passenger Screening Program (PSP) and the Electronic Baggage Screening Program (EBSP): PSP technologies, such as advanced imaging technology and bottled liquid scanners, work in combination at airport checkpoints to screen passengers and their carry-on baggage for threats. EBSP technologies, such as explosives trace detectors and explosives detection systems, help ensure that TSA screens 100 percent of checked baggage for explosives, as mandated by law. For each technology type, vendors develop their own specific version or system. See table 1 for an overview of key PSP and EBSP technology types. According to TSA, EBSP and PSP are transitioning into a sustainment mode, meaning they are largely not procuring new types of screening technologies and are instead focused on the recapitalization of over 2,400 systems that are reaching their end-of-life over the next 5 years. In our past work, we found key challenges related to TSA’s efforts to acquire technologies. For example, in March 2014, we found that TSA’s performance assessments of advanced imaging technology systems with automatic target recognition capability did not account for all factors affecting the effectiveness of the system, such as how well the system would perform with human operators, because the assessments were conducted in a laboratory. We recommended that TSA establish protocols to facilitate capturing operational data on secondary passenger screening at the checkpoint to determine the extent to which rates of false alarms for various advanced imaging technology systems affect operational costs once the systems with automatic target recognition capability are networked. TSA concurred with this recommendation. In its comments to our report, TSA stated that it will monitor, update, and report the results of its efforts to capture operational data and evaluate its associated impacts to operational costs. In January 2012, we found that TSA did not fully follow DHS acquisition policies when acquiring a different version of advanced imaging technology, which resulted in DHS approving nationwide deployment without full knowledge of TSA’s revised specifications. Among other things, we found that TSA had changed key performance parameters during the acquisition process without formally informing DHS acquisition officials, in violation of DHS policy. We recommended that TSA develop a road map describing when vendors will meet milestones for further developing advanced imaging technology and that TSA brief Congress as part of the budget process. Findings from this report resulted in a reduction in planned advanced imaging technology purchases amounting to approximately $1.4 billion. DHS policies and processes for managing its major acquisition programs, such as TSA’s PSP and EBSP, are primarily set forth in DHS Acquisition Management Directive 102-01 and DHS Instruction Manual 102-01-001, Acquisition Management Instruction/Guidebook. Within TSA, the PSP and EBSP program management offices are responsible for planning and executing the acquisition programs within the cost, schedule, and performance parameters in their program baselines, which are approved by DHS. DHS’s Deputy Secretary or Under Secretary for Management serve as the decision authority for PSP and EBSP acquisitions since both are large acquisition programs with life cycle cost estimates of over $1 billion. The decision authority reviews programs at five predetermined acquisition decision events to assess whether the programs are ready to proceed through each of the four acquisition life cycle phases. During the first phase of the acquisition life cycle—the “need” phase— TSA develops a mission need statement for a potential solution to an identified problem, such as threats concealed on passengers and in their carry-on items. If the acquisition decision authority concludes that the need is of sufficiently high priority to DHS, the program moves to the second phase known as the “analyze/select” phase. The TSA program management offices—such as PSP and EBSP—communicate this need to potential vendors through requests for information, requests for proposals, and broad agency announcements in search of potential solutions, such as innovative screening technologies. Program managers select the best technology option and, if the acquisition decision authority approves the proposed technology, the program moves to phase three. The third phase is the “obtain” phase in which TSA develops, tests, and evaluates systems—vendors’ versions of the selected technology. With acquisition decision authority approval, programs enter the final phase, the “produce/deploy/support” phase, and TSA can proceed with procurement and full deployment of tested and qualified systems. The goal of test and evaluation is to make sure that a product works as intended before it is provided to end-users, such as TSA’s transportation security officers. According to DHS policy, the primary purpose of test and evaluation is to provide timely, accurate information to managers, decision makers, and other stakeholders to reduce programmatic, financial, schedule, and performance risk. TSA’s test and evaluation process is guided by DHS and TSA policies. DHS issued its test and evaluation policy in May 2009 and TSA issued its policy in July 2010. Within OSC, the Operations Support Division is responsible for testing and evaluating screening technologies in coordination with DHS’s Transportation Security Laboratory. Table 2 provides an overview of DHS and TSA test and evaluation roles and responsibilities. As noted in the table, DHS and TSA both have responsibilities for evaluating vendors’ qualification data packages and determining whether to accept systems for testing. Once accepted, the systems undergo TSA’s three phase pre-acquisition test and evaluation process to verify requirements. Responsibility for verifying requirements is assigned to testers for each of the three phases: (1) qualification and certification testing at the Transportation Security Laboratory, (2) qualification testing at the TSA Systems Integration Facility, and (3) operational testing at selected airports. 1. Qualification and certification testing at the Transportation Security Laboratory. Qualification testing is the formal verification and validation of a system’s performance and must provide confidence that the system will satisfy desired capabilities in an operational environment. Qualification testing is typically conducted at two facilities, each serving distinct functions. Testing conducted at the DHS Science and Technology Directorate’s Transportation Security Laboratory in Atlantic City, New Jersey is focused on qualifying or certifying that a system meets the probability of detection of all categories of explosives while meeting TSA’s designated false alarm requirement. Typically, systems complete detection testing at the Transportation Security Laboratory prior to undergoing the balance of qualification testing at the TSA Systems Integration Facility. 2. Qualification testing at the TSA Systems Integration Facility. TSA conducts additional qualification testing at the TSA Systems Integration Facility in Arlington, Virginia to verify system performance against defined functional requirements, such as system capacities, human factors, physical characteristics, user safety, and system reliability, availability, and maintainability. This testing may include transportation security officers who exercise standard operating procedures to provide feedback regarding the system’s functions in a controlled environment. Systems must pass qualification testing and undergo an operational test readiness review before they can proceed to operational testing. 3. Operational testing at select airports. TSA conducts operational testing at select airports to evaluate a system’s operational effectiveness and suitability in a realistic environment and to ensure that the airport infrastructure is ready to accept the system. According to TSA, it selects test sites, in coordination with various stakeholders, using criteria intended to facilitate an unbiased evaluation of the systems. TSA injects mock threats during operational testing to assess performance of the entire system, including users and standard operating procedures. Operational tests focus on demonstrating that operational requirements have been met and that all critical operational issues have been resolved. Following operational testing, TSA prepares a system evaluation report, which provides effectiveness and suitability determinations along with potential system improvement recommendations. The DHS Director of Operational Test and Evaluation reviews the report and prepares a letter of assessment. The letter assesses the system evaluation report and the adequacy of TSA’s operational test and concludes whether the system is operationally suitable and effective for procurement. TSA’s program management offices place approved systems on qualified products lists, which contain systems that have successfully completed the test and evaluation process and have been approved by DHS. Once a system is on a qualified products list, which TSA maintains for most technology types, such as explosives detection systems, the vendor can participate in TSA’s procurement process. However, placement on a qualified products list does not guarantee that the vendor will receive a procurement contract. TSA conducts acceptance testing on systems at vendors’ factories following production and again once installed at airports to ensure consistency of the manufacturing process, system configuration, and functionality. Post deployment, TSA’s Office of Inspection and the DHS Office of Inspector General conduct covert tests in airports to identify vulnerabilities in screening processes. TSA and DHS test and evaluate systems prior to procurement, but covert testing has demonstrated that systems, when integrated into TSA’s screening process, do not always work as intended once deployed. TSA’s test and evaluation activities are funded by the program offices, and TSA’s life cycle cost estimate for EBSP reflects average annual testing costs of approximately $24 million. PSP officials estimate that the program averages $10 million in annual testing costs. TSA’s security screening must adapt to meet evolving threats since potential terrorists use their knowledge of aviation security measures when planning aviation-related attacks. New explosives threats can result in heightened detection standards, which may require software upgrades to existing systems or the development of new technologies.