DHS and TSA share responsibility for the research, development, and deployment of passenger checkpoint screening technologies. The Aviation and Transportation Security Act established TSA as the federal agency with primary responsibility for securing the nation’s civil aviation system, which includes the screening of all passengers and property transported to, from, and within the United States by commercial passenger aircraft. Additionally, the Homeland Security Act of 2002 established DHS and, within it, the Science and Technology Directorate for, among other things, conducting research, development, demonstration, and testing and evaluation activities relevant to DHS.DHS’s Science and Technology Directorate is responsible for testing and evaluating aviation security technologies, including AIT systems, at the TSL on behalf of TSA. DHS and TSA conducted five types of tests to evaluate the performance of AIT-ATR systems. Qualification testing. TSL conducted qualification tests in a laboratory setting to evaluate the technology’s capabilities against TSA’s procurement specification and detection standard that specified the required detection rate AIT systems must meet in order to qualify for procurement. Qualification tests evaluate the technology’s detection of threat items that are not artfully concealed as they are in covert tests, but do not test the entire system, including the SO’s interpretation and resolution of alarms. Qualification testing also includes testing of the system’s false alarm rate. For the purposes of this report, we refer to qualification testing as laboratory testing. Operational testing. TSA conducted operational tests that assessed the technology’s detection performance, called threat-inject tests, at airports to evaluate the AIT-ATR systems’ ability to function in an operational environment. Operational testing also assesses how well AIT systems are suited for use in a real-world, aviation checkpoint environment after systems have successfully completed qualification testing in a laboratory setting. For example, operational testing includes determining whether the system interfered with other equipment fielded at the checkpoint and whether the system met TSA’s requirements. Further, DHS’s acquisition policy requires that operational tests be conducted prior to an agency procuring a technology. According to TSA testing documentation, threat- inject tests are not intended to evaluate effectiveness of the entire AIT- ATR system, which includes the technology, the personnel who use the technology, and the processes that govern screening, in an operational setting. Covert testing. TSA’s Office of Inspection and the DHS Office of Inspector General conducted covert tests of AIT-ATR systems at the passenger checkpoint to identify vulnerabilities in TSA’s screening process. According to TSA officials, those tests were intended to identify weaknesses in the technology, the operators who used it, and TSO compliance with SOPs by artfully concealing threat objects intended to simulate a likely terrorist attack. Performance assessments. TSA conducted covert performance assessments of TSO compliance with SOPs, under the Aviation Screening Assessment Program (ASAP), which TSA uses as a standard performance measurement for the Office of Management and Budget. According to TSA officials, ASAP assessments determine SO adherence to TSA’s SOPs and are not intended to test AIT-ATR system capabilities. Checkpoint drills. In accordance with TSA’s IED checkpoint drill operational directive, TSA requires personnel at airports to conduct drills to assess TSO compliance with TSA’s screening SOPs and to train TSOs to better resolve anomalies identified by AIT-ATR systems. TSA conducts those drills at airports using test kits that contain inert bombs, bomb parts, and other threat items. According to TSA officials, IED checkpoint drills assess SO adherence to TSA’s SOPs and are not intended to test AIT-ATR system capabilities. TSA uses a multilayered security strategy aimed to enhance aviation security. Within those layers of security, TSA’s airport passenger checkpoint screening system includes, among other things, (1) screening personnel; (2) SOPs that guide screening processes conducted by TSOs; and (3) technology, such as AIT-ATR systems, used to conduct screening According to TSA, those elements collectively determine of passengers.the effectiveness and efficiency of passenger checkpoint screening. In strengthening one or more elements of its checkpoint screening system, TSA aims to balance its security goals with the need to efficiently process passengers. Passenger screening is a process by which TSOs inspect individuals and their property to deter and prevent an act of violence, such as carrying an explosive, weapon, or other prohibited item onboard an aircraft or into the airport sterile area—in general, an area of an airport for which access is controlled through screening of persons and property. individuals for prohibited items at designated screening locations, referred to as checkpoints, where TSOs use technology and follow SOPs to screen passengers. According to TSA’s SOP for AIT-ATR systems, three TSOs are required to operate lanes equipped with AIT systems: one divestiture officer (of either gender), one male SO, and one female SO. See 49 C.F.R. § 1540.5. As we reported in January 2012, TSA’s requirements for the AIT system have evolved over time. TSA continued to use those revised requirements to determine whether the AIT-ATR system met the agency’s needs. Additionally, TSA used those requirements to evaluate the next generation of AIT systems, referred to as AIT-2. Further, TSA’s requirements for AIT systems are based on tiers that correspond to the relative size of items that the AIT system must identify and requirements that the AIT system must meet, with Tier I being the level currently deployed AIT systems already meet and Tier IV being TSA’s anticipated goal for AIT systems to meet. TSA’s procurement of AIT-2 systems requires vendors to ensure AIT-2 systems meet Tier II requirements and provide faster throughput, among other things. TSA plans to seek proposals from AIT-2 vendors to provide Tier III and Tier IV capabilities by time frames specified in its AIT roadmap. TSA did not initially plan for AIT- IO systems to meet levels beyond Tier III, but included Tier IV in response to our recommendation. TSA does not collect or analyze three types of available information that could be used to enhance the effectiveness of the entire AIT-ATR system. First, TSA does not collect or analyze available airport-level IED checkpoint drill data on SO performance at resolving alarms detected by the AIT-ATR system to identify weaknesses and enhance SO performance at resolving alarms at the checkpoint. Second, TSA is not analyzing AIT-ATR systems’ false alarm rate in the field using data that could help it monitor the number of false alarms that occur on AIT-ATR systems to help monitor the potential impacts that AIT-ATR systems may have on operational costs. Third, TSA assesses the overall AIT-ATR system performance using laboratory test results that do not reflect the combined performance of the technology, the personnel that operate it, and the process that governs AIT-related security operations. TSA does not collect or analyze IED checkpoint drill data, because it does not ensure compliance with its operational directive that requires each airport to conduct IED checkpoint drills each week. Specifically, the operational directive, originally issued in February 2010 and updated in November 2012, requires TSA personnel at airports to conduct a certain number of IED drills per checkpoint lane every week at each airport. The total number of drills per pay period must be split evenly between carry- on baggage and passenger screening. Additionally, for those airports equipped with AIT systems, a certain percentage of on-person drills must be conducted on AIT systems and a certain percentage must be conducted on walk-through metal detectors. TSA is not enforcing compliance with its directive, and as a result, data on SO performance are not being consistently collected or reported by approximately half of airports with AIT-ATR systems. For example, according to TSA data, we found that TSA personnel at almost half of the airports with AIT-IO or AIT-ATR systems did not report any IED checkpoint drill results on those systems from March 2011 through February 2013. Of the airports at which TSA personnel conducted IED checkpoint drills, the number of drills conducted by TSA personnel at airports varied from 1 to 8,645. Further, roughly four-fifths of the on- person IED drills were conducted by screening passengers with metal detectors, with the rest of the IED drills conducted by screening passengers with AIT systems, which did not comply the directive’s specified requirements on the number of drills that must be conducted on each type of technology. According to TSA officials, TSA’s Office of Security Operations is responsible for overseeing compliance with the operational directive at airports, but it does not analyze the IED checkpoint drill data at the headquarters level. Further, TSA officials told us that TSA formerly tracked the number of IED checkpoint drills in a monthly management report for federal security directors, but in fiscal year 2012, that report was replaced by an executive scorecard that tracks each airport’s IED checkpoint drill pass rate but does not include the number of drills conducted. TSA officials stated that federal security directors could conduct very few drills that are easy for SOs to identify in order to achieve a high pass rate, since the details of the drills are not provided to headquarters or analyzed beyond the pass rate. According to TSA officials, the agency does not ensure compliance with the directive at every airport because it is unclear which office within the Office of Security Operations should oversee enforcing the operational directive.