Our review of the pilot test identified several challenges related to pilot planning, data collection, and reporting, which affected the completeness, accuracy, and reliability of the results. DHS did not correct planning shortfalls that we identified in our November 2009 report. We determined that these weaknesses presented a challenge in ensuring that the pilot would yield information needed to inform Congress and the card reader rule and recommended that DHS components implementing the pilot—TSA and USCG—develop an evaluation plan to guide the remainder of the pilot and identify how it would compensate for areas where the TWIC reader pilot would not provide the information needed. DHS agreed with the recommendations; however, while TSA developed a data analysis plan, TSA and USCG reported that they did not develop an evaluation plan with an evaluation methodology or performance standards, as we recommended. The data analysis plan was a positive step because it identified specific data elements to be captured from the pilot for comparison across pilot sites. If accurate data had been collected, adherence to the data analysis plan could have helped yield valid results. However, TSA and the independent did not utilize the data analysis plan. According to officials test agentfrom the independent test agent, they started to use the data analysis plan but stopped using the plan because they were experiencing difficulty in collecting the required data and TSA directed them to change the reporting approach. TSA officials stated that they directed the independent test agent to change its collection and reporting approach because of TSA’s inability to require or control data collection to the extent required to execute the plan. We identified eight areas where TWIC reader pilot data collection, supporting documentation, and recording weaknesses affected the completeness, accuracy, and reliability of the pilot data 1. Installed TWIC readers and access control systems could not collect required data on TWIC reader use, and TSA and the independent test agent did not employ effective compensating data collection measures. The TWIC reader pilot test and evaluation master plan recognizes that in some cases, readers or related access control systems at pilot sites may not collect the required test data, potentially requiring additional resources, such as on-site personnel, to monitor and log TWIC card reader use issues. Moreover, such instances were to be addressed as part of the test planning. However, the independent test agent reported challenges in sufficiently documenting reader and system errors. For example, the independent test agent reported that the logs from the TWIC readers and related access control systems were not detailed enough to determine the reason for errors, such as biometric match failure, an expired TWIC card, or that the TWIC was identified as being on the list of revoked credentials. The independent test agent further reported that the inability to determine the reason for errors limited its ability to understand why readers were failing, and thus it was unable to determine whether errors encountered were due to TWIC cards, readers, or users, or some combination thereof. 2. Reported transaction data did not match underlying documentation. A total of 34 pilot site reports were issued by the independent test agent. According to TSA, the pilot site reports were used as the basis for DHS’s report to Congress. We separately requested copies of the 34 pilot site reports from both TSA and the independent test agent. In comparing the reports provided, we found that 31 of the 34 pilot site reports provided to us by TSA did not contain the same information as those provided by the independent test agent. Differences for 27 of the 31 pilot site reports pertained to how pilot site data were characterized, such as the baseline throughput time used to compare against throughput times observed during two phases of testing. However, at two pilot sites, Brownsville and Staten Island Ferry, transaction data reported by the independent test agent did not match the data included in TSA’s reports. Moreover, data in the pilot site reports did not always match data collected by the independent test agent during the pilot. 3. Pilot documentation did not contain complete TWIC reader and access control system characteristics. Pilot documentation did not always identify which TWIC readers or which interface (e.g., contact or contactless interface) the reader used to communicate with the TWIC card during data collection.different readers were tested. However, the pilot site report did not identify which data were collected using which reader. For example, at one pilot site, two 4. TSA and the independent test agent did not record clear baseline data for comparing operational performance at access points with TWIC readers. Baseline data, which were to be collected prior to piloting the use of TWIC with readers, were to be a measure of throughput time, that is, the time required to inspect a TWIC card and complete access-related processes prior to granting entry. However, it is unclear from the documentation whether acquired data were sufficient to reliably identify throughput times at truck, other vehicle, and pedestrian access points, which may vary. 5. TSA and the independent test agent did not collect complete data on malfunctioning TWIC cards. TSA officials observed malfunctioning TWIC cards during the pilot, largely because of broken antennas. If a TWIC with a broken antenna was presented for a contactless read, the reader would not identify that a TWIC had been presented, as the broken antenna would not communicate TWIC information to a contactless reader. In such instances, the reader would not log that an access attempt had been made and failed. 6. Pilot participants did not document instances of denied access. Incomplete data resulted from challenges documenting how to manage individuals with a denied TWIC across pilot sites. Specifically, TSA and the independent test agent did not require pilot participants to document when individuals were granted access based on a visual inspection of the TWIC, or deny the individual access as may be required under future regulation. This is contrary to the TWIC reader pilot test and evaluation master plan, which calls for documenting the number of entrants “rejected” with the TWIC card reader system operational as part of assessing the economic impact. Without such documentation, the pilot sites were not completely measuring the operational impact of using TWIC with readers. 7. TSA and the independent test agent did not collect consistent data on the operational impact of using TWIC cards with readers. TWIC reader pilot testing scenarios included having each individual present his or her TWIC for verification; however, it is unclear whether this actually occurred in practice. For example, at one pilot site, officials noted that during testing, approximately 1 in 10 individuals was required to have his or her TWIC checked while entering the facility because of concerns about causing a traffic backup. Despite noted deviations in test protocols, the reports for these pilot sites do not note that these deviations occurred. Noting deviations in each pilot site report would have provided important perspective by identifying the limitations of the data collected at the pilot site and providing context when comparing the pilot site data with data from other pilot sites. 8. Pilot site records did not contain complete information about installed TWIC readers’ and access control systems’ design. TSA and the independent test agent tested the TWIC readers at each pilot site to ensure they worked before individuals began presenting their TWIC cards to the readers during the pilot. However, the data gathered during the testing were incomplete. For example, 10 of 15 sites tested readers for which no record of system design characteristics were recorded. In addition, pilot reader information was identified for 4 pilot sites but did not identify the specific readers or associated software tested. According to TSA, a variety of challenges prevented TSA and the independent test agent from collecting pilot data in a complete and consistent fashion. Among the challenges noted by TSA, (1) pilot participation was voluntary, which allowed pilot sites to stop participation at any time or not adhere to established testing and data collection protocols; (2) the independent test agent did not correctly and completely collect and record pilot data; (3) systems in place during the pilot did not record all required data, including information on failed TWIC card reads and the reasons for the failure; and (4) prior to pilot testing, officials did not expect to confront problems with nonfunctioning TWIC cards. Additionally, TSA noted that it lacked the authority to compel pilot sites to collect data in a way that would have been in compliance with federal standards. In addition to these challenges, the independent test agent identified the lack of a database to track and analyze all pilot data in a consistent manner as an additional challenge to data collection and reporting. The independent test agent, however, noted that all data collection plans and resulting data representation were ultimately approved by TSA and USCG. As required by the SAFE Port Act and the Coast Guard Authorization Act of 2010, DHS’s report to Congress on the TWIC reader pilot presented several findings with respect to technical and operational aspects of implementing TWIC technologies in the maritime environment. However, DHS’s reported findings were not always supported by the pilot data, or were based on incomplete or unreliable data, thus limiting the report’s usefulness in informing Congress about the results of the TWIC reader pilot. For example, reported entry times into facilities were not based on data collected at pilot sites as intended. Further, the report concluded that TWIC cards and readers provide a critical layer of port security, but data were not collected to support this conclusion. Because of the number of concerns that we identified with the TWIC pilot, in our March 13, 2013, draft report to DHS, we recommended that DHS not use the pilot data to inform the upcoming TWIC card reader rule. However, after receiving the draft that we sent to DHS for comment, on March 22, 2013, USCG published the TWIC card reader notice of proposed rulemaking (NPRM), which included results from the TWIC card reader pilot. We subsequently removed the recommendation from our final report, given that USCG had moved forward with issuing the NPRM and had incorporated the pilot results into the proposed rulemaking.