The Aviation and Transportation Security Act (ATSA) established TSA as the primary federal agency with responsibility for securing the nation’s civil aviation system. This responsibility includes the screening of all passengers and property transported from and within the United States by commercial passenger aircraft. In accordance with ATSA, all passengers, their accessible property, and their checked baggage are to be screened pursuant to TSA-established procedures at the more than 450 airports at which TSA performs, or oversees the performance of, security screening operations. These procedures generally provide, among other things, that passengers pass through security checkpoints where their person, identification documents, and accessible property are checked by screening personnel. The following are some of TSA’s transportation security technologies, processes, and programs. AIT systems: According to TSA officials, AIT systems provide enhanced security benefits compared with those of walk-through metal detectors by identifying nonmetallic objects and liquids. Following the deployment of AIT, the public and others raised privacy concerns because AIT systems produced images of passengers’ bodies that image operators analyzed to identify objects or anomalies that could pose a threat to an aircraft or to the traveling public. To mitigate those concerns, TSA began installing automated target recognition (ATR) software on deployed AIT systems in July 2011. AIT systems equipped with ATR (AIT-ATR) automatically interpret the image and display anomalies on a generic outline of a passenger instead of displaying images of actual passenger bodies. Screening officers use the generic image of a passenger to identify and resolve anomalies on-site in the presence of the passenger. Expedited Screening and TSA’s Managed Inclusion Process: TSA Pre✓TM —TSA’s expedited screening program—is intended to allow TSA to devote more time and resources at the airport to screening the passengers TSA determined to be higher or unknown risk while providing expedited screening to those passengers determined to pose a lower risk to the aviation system. To assess whether a passenger is eligible for expedited screening, TSA considers, in general, (1) inclusion on an approved TSA Pre✓TM list of known travelers; (2) results from the automated TSA Pre✓TM risk assessments of all passengers; and (3) real-time threat assessments of passengers, known as Managed Inclusion, conducted at airport checkpoints. Through its Managed Inclusion process, TSA has utilized a combination of security measures, including behavior detection officers (BDO) and passenger screening canine teams at the checkpoint to identify passengers suitable for expedited screening. Behavior Detection and Analysis program: TSA’s Behavior Detection and Analysis program, formerly known as the Screening of Passengers by Observation Techniques (SPOT) program, is intended to identify persons who may pose a risk to aviation security. Through these behavior detection activities, TSA’s BDOs are to identify passenger behaviors indicative of stress, fear, or deception and refer passengers meeting certain criteria for additional screening of their persons and carry-on baggage. During this referral screening, if passengers exhibit additional such behaviors, or if other events occur, such as the discovery of a suspected fraudulent document, BDOs are to refer these passengers to a law enforcement officer for further investigation, which could result in an arrest, among other outcomes. Secure Flight: Since TSA began implementing Secure Flight in 2009, the passenger prescreening program has changed from a program that identifies passengers as high risk solely by matching them against federal government watch lists—for example, the No Fly List, comprised of individuals who should be precluded from boarding an aircraft, and the Selectee List, comprised of individuals who should receive enhanced screening at the passenger security checkpoint—to one that uses additional lists and risk-based criteria to assign passengers to a risk category: high risk, low risk, or unknown risk. In 2010, following the December 2009 attempted attack on a U.S.-bound flight, which exposed gaps in how agencies used watch lists to screen individuals, TSA began using risk-based criteria to create additional lists for Secure Flight screening. These lists are composed of high-risk passengers who may not be in the Terrorist Screening Database (TSDB)—the U.S. government’s consolidated watch list of known or suspected terrorists— but whom TSA has determined should be subject to enhanced screening procedures. Further, in 2011, TSA began screening passengers against additional identities in the TSDB that are not included on the No Fly or Selectee Lists. In addition, as part of TSA Pre✓™, TSA began screening against several new lists of preapproved low-risk travelers. TSA also began conducting TSA Pre✓™ risk assessments—an activity distinct from matching against watch lists—that use the Secure Flight system to assign passengers scores based upon their travel-related data, for the purpose of identifying them as low risk for a specific flight. National Explosives Detection Canine Team Program: One of TSA’s security layers is its National Explosives Detection Canine Team Program (NEDCTP), composed of over 800 explosives detection canine teams—a canine paired with a handler—aimed at deterring and detecting the use of explosive devices in the U.S. transportation system. Through NEDCTP, TSA trains, deploys, and certifies explosives detection canine teams. TSA deploys the teams to screen passengers and air cargo at airports and other transportation modes, including mass transit. In our 2014 reviews of TSA’s AIT-ATR systems and Managed Inclusion process, we found that TSA had conducted some testing before adopting the new technology and process, but it had not fully demonstrated their effectiveness. We have also previously reported on challenges TSA has faced in designing studies and protocols to test the effectiveness of security systems and programs in accordance with established methodological practices, such as in the case of our 2013 review of TSA’s behavior detection activities. TSA has since taken steps to more comprehensively test the effectiveness of the next generation of AIT, known as AIT-2, and further test aspects of the Managed Inclusion process. With regard to the AIT-ATR system, in March 2014, we reported that, according to TSA officials, checkpoint security is a function of technology, people, and the processes that govern them; however, we found that TSA did not include each of those factors in determining overall AIT-ATR system performance. Specifically, we found that TSA evaluated the technology’s performance in the laboratory to determine system effectiveness. Laboratory test results provide important insights but do not accurately reflect how well the technology will perform in the field with actual human operators. Additionally, we found that TSA did not assess how alarms are resolved by considering how the technology, people, and processes function collectively as an entire system when determining AIT-ATR system performance. AIT-ATR system effectiveness relies on both the technology’s capability to identify threat items and its operators to resolve those threat items. Given that TSA was seeking to procure the second generation of AIT systems, known as AIT-2, we reported that DHS and TSA would be hampered in their ability to ensure that future AIT systems meet mission needs and perform as intended at airports unless TSA evaluated system effectiveness based on both the performance of the AIT-2 technology and screening officers who operate the technology. According to best practices related to federal acquisitions, technologies should be demonstrated to work in their intended environment. We recommended that TSA measure system effectiveness based on the performance of the AIT-2 technology and screening officers who operate the technology while taking into account current processes and deployment strategies. TSA concurred and has addressed this recommendation. Specifically, in June 2015, TSA provided documentation showing that, while conducting operational testing of the AIT-2 system, the agency considered screening officer performance and measured AIT-2 system effectiveness based on both the performance of the AIT-2 technology and the screening officers who operate it. This should help TSA assess whether this screening system will meet mission needs and perform as intended. With regard to the Managed Inclusion process, in December 2014, we reported that TSA had tested the security effectiveness of the individual components of the Managed Inclusion process, but had not tested the overall effectiveness of the Managed Inclusion process as it functions as a whole. According to TSA officials, TSA tested the security effectiveness of the individual components being used in the Managed Inclusion process at the time—such as BDOs, passenger screening canine teams, and explosives trace detection (ETD) devices—before implementing Managed Inclusion, and TSA determined that each layer alone provides an effective level of security. However, in our prior body of work, we identified challenges in several of the layers used in the Managed Inclusion process, raising questions regarding their effectiveness. Further, as of the time of our report, TSA officials stated that they had not yet tested the security effectiveness of the Managed Inclusion process as it functions as a whole. TSA officials explained that they had been planning to test the process as a whole and estimated that such testing would begin in October 2014 and would take 12 to 18 months to complete. However, TSA could not provide us with specifics or a plan or documentation showing how the testing was to be conducted, the locations where it was to occur, how those locations were to be selected, or the timeframes for conducting testing at each location. In general, evaluations are most likely to be successful when key steps are addressed during design, including defining research questions appropriate to the scope of the evaluation, and selecting appropriate measures and study approaches that will lead to valid conclusions.