DHS acquisitions support a wide range of missions and investments including ships and aircraft, border surveillance and screening equipment, nuclear detection equipment, and systems to track the department’s financial and human resources. In support of these investments, DHS, in 2003, established an investment review process to help reduce risk and increase the chances for successful acquisition outcomes by providing departmental oversight of major investments throughout their life cycles and to help ensure that funds allocated for investments through the budget process are being spent wisely, efficiently, and effectively. Our work over the past several years has consistently pointed to the challenges DHS has faced in effectively managing and overseeing its acquisition of programs and technologies. In November 2008, we reported that DHS had not effectively implemented its investment review process, and as a result, the department had not provided the oversight needed to identify and address cost, schedule, and performance problems for its major acquisitions. Specifically, we reported that of the 48 major investments reviewed requiring milestone or annual reviews, 45 were not reviewed in accordance with the departments’ investment review policy, and 18 were not reviewed at all. Four of these investments had transitioned into a late acquisition phase—production and deployment—without any required reviews. We recommended and DHS concurred that DHS identify and align sufficient management resources to implement oversight reviews in a timely manner throughout the investment life cycle. In June 2010, we reported that over half of the 15 DHS programs we reviewed awarded contracts to initiate acquisition activities without component or department approval of documents essential to planning acquisitions, setting operational requirements, and establishing acquisition program baselines. Our work noted that without the development, review, and approval of these key acquisition documents, agencies are at risk of having poorly defined requirements that can negatively affect program performance and contribute to increased costs. In January 2011, DHS reported that it has begun to implement an initiative to assist programs with completing departmental approval of acquisition program baselines. In our February 2011 biennial update of the status of high-risk areas needing attention by Congress and the executive branch, we continued to designate DHS’s implementation and transformation, which includes the department’s management functions, as a high-risk area. For example, because of acquisition management weaknesses, major programs, such as SBInet, have not met capability, benefit, cost, and schedule expectations. Further, DHS had not fully planned for or acquired the workforce needed to implement its acquisition oversight policies as we previously recommended. As of January 2011, DHS reported that it had increased its acquisitions management staffing and planned to hire more staff to develop cost estimates. DHS has taken several actions to address these recommendations and implement more discipline and rigor in its acquisition processes. Specifically, DHS created the Acquisition Program Management Division in 2007 to develop and maintain acquisition policies, procedures, and guidance as a part of the system acquisition process. DHS also issued an interim acquisition directive and guidebook in November 2008 for programs to use in preparing key documentation to support component and departmental making. In January 2010, DHS finalized the acquisition directive which established acquisition life-cycle phases and senior-level approval of each major acquisition program at least three times at key acquisition decision events during a program’s acquisition life-cycle. This directive established the acquisition life-cycle framework with four phases: (1) identify a capability need (need phase); (2) analyze and select the means to provide that capability (analyze/select phase); (3) obtain the capability (obtain phase); and (4) produce, deploy, and support the capability (produce/deploy/support phase). Each acquisition phase culminates in a presentation to the Acquisition Review Board (ARB), which is to review each major acquisition (that is, those designated as level 1 or level 2 programs) at least three times at key acquisition decision events during a program’s acquisition life cycle. The acquisition decision authority—the Chief Acquisition Officer or other designated senior-level official—is to chair the ARB and decide whether the proposed acquisition meets certain requirements necessary to move on to the next phase and eventually to full production. The directive outlines the extent and scope of required program, project, and service management; level of reporting requirement; and the acquisition decision authority based on whether the acquisition is classified as level 1, 2, or 3. The acquisition decision authority for major acquisitions—level 1 and level 2—is to be at the department or component level and the acquisition decision authority for nonmajor acquisitions—level 3—is to be at the component level.An acquisition may be raised to a higher level acquisition level by the ARB. The ARB supports the acquisition decision authority in determining the appropriate direction for an acquisition at key Acquisition Decision Events. Following an ARB meeting, the Acquisition Program Management Division is to prepare an acquisition decision memorandum as the official record of the meeting to be signed by the acquisition decision authority. This memo is to describe the approval or other decisions made at the ARB and any action items to be satisfied as conditions of the decision. The ARB reviews are to provide an opportunity to determine a program’s readiness to proceed to the following life-cycle phase. However, we reported in March 2011 that the ARB had not reviewed most of DHS’s major acquisition programs by the end of fiscal year 2009 and programs that were reviewed had not consistently implemented action items identified as part of the review by established deadlines. Our prior work has shown that when these types of reviews are skipped or not fully implemented, programs move forward with little, if any, early department-level assessment of the programs’ costs and feasibility, which contributes to poor cost, schedule, and performance outcomes. As a part of its responsibilities, the Acquisition Program Management Division has identified major DHS acquisition programs, projects, or services for oversight through the ARB process. According to Acquisition Program Management Division officials, beginning in fiscal year 2009, the list was to be updated on a yearly basis through interviews with and documentation from component program offices. In May 2010, the Undersecretary for Management identified 86 programs on DHS’s major oversight list for fiscal year 2010, 62 of which TES and component officials determined required T&E oversight—that is programs that were in an acquisition phase where T&E was being planned or conducted. Several of the 62 programs consisted of multiple subprojects, such as TSA’s Passenger Screening Program. For more information on these 86 major acquisition programs, see appendix II. DHS’s 2010 acquisition directive also includes guidance for preparing documentation to support component and departmental decision making and specifies requirements for developmental and operational T&E as a part of the acquisition review process. Developmental T&E may include a variety of tests, such as system qualification testing, system acceptancetesting, and software testing. Developmental testing may be carried out by the user and may be conducted in simulated environments, such as laboratories, test facilities, or engineering centers that might or might not be representative of the complex operational environment. Operational T&E is a field test, performed under realistic conditions by actual users in order to determine the operational effectiveness and suitability of a system, and the corresponding evaluation of the data resulting from the test. To carry out its responsibilities for overseeing T&E, S&T established TES in 2006 and created the position of Director of TES in June 2007. TES’s mission is to establish and manage DHS T&E policies and procedures and to oversee and coordinate T&E resources to verify attainment of technical performance specifications and operational effectiveness and suitability. To carry out its T&E oversight, in fiscal year 2010, TES had a budget of about $23 million and as of February 2011 had a staff of 26, which includes the TES Director, 19 staff dedicated to T&E activities, and 6 dedicated to developing standards. In May 2009, DHS issued a delegation which specified the responsibilities and duties of the Director of Operational Test & Evaluation. The TES Director and Director of Operational Test and Evaluation, while distinct positions in the T&E directive, share some advisory, review, and oversight responsibilities. For example, both are responsible for advising program managers in developing T&E documentation and approving test and evaluation master plans. The TES Director is responsible for developing DHS T&E policy and the Director of Operational Test and Evaluation is to approve operational test plans and report to the ARB after assessing operational test reports. Since May 2009, the Director of Operational Test and Evaluation position has not been continuously filled according to the current TES Director. In a November 2010 memo, the Under Secretary for Science and Technology designated one person as both the director of TES and the Director of Operational Test and Evaluation until further notice. The T&E directive outlines the responsibilities of the TES Director and the Director of Operational Test and Evaluation. According to the directive, the TES Director is to establish the department’s testing and evaluation policies and processes and the Director of Operational Test and Evaluation is to administer those policies and processes. The directive also outlines TES’s responsibilities in overseeing T&E across DHS components and its role in the acquisition review process. Table 1 describes TES’s T&E responsibilities as outlined in the T&E directive for all level 1, level 2, and special oversight acquisition programs. The T&E directive requires TES to review and approve required component acquisition documentation before an ARB meets for an acquisition decision event. These documents are meant to be reviewed and, if required, approved in a sequential order associated with the acquisition phase, because these documents build upon one another. Figure 1 presents TES’s responsibilities throughout the four DHS acquisition phases as defined in the acquisition directive. To carry out these responsibilities for the 62 acquisition programs under its oversight in fiscal year 2010, TES has test area managers who assist component officials in fulfilling their T&E responsibilities and provide guidance and clarification in regard to the requirements in the T&E directive. According to TES, each major acquisition program is assigned a test area manager and as of February 2011, TES employed nine test area managers. TES met its oversight requirements when approving test plans and test reports in accordance with DHS acquisition and T&E directives for the 11 major acquisition programs we selected for review. However, TES did not consistently document its review and approval of operational test agents or its review of other required acquisition documentation, which could provide more assurance that components were meeting T&E directives when TES reviewed these documents. Further, TES does not plan an independent assessment of TSA’s Advanced Spectroscopic Portal’s operational test results, as required by the T&E directive. TES is to oversee T&E of major DHS acquisition programs by ensuring that the requirements set forth in the T&E directive are met and by working with component program officials to develop T&E documentation, such as test and evaluation master plans, as required by DHS’s acquisition directive. TES’s T&E oversight responsibilities set forth in the T&E and acquisition directives pertain to programs primarily in the analyze/select and obtain phases of the acquisition process because most testing and evaluation efforts occur in these phases. As a result, the requirements of the T&E directive and TES’s oversight vary depending on when a program progresses through certain phases of the acquisition process. For example, when a program is in the produce/deploy/support phase there is usually little to no T&E activity, so TES’s involvement is limited. We reviewed TES’s T&E oversight efforts for 11 DHS programs and found that TES had conducted oversight of components’ test plans and test reports, as set forth in the acquisition and T&E directives, as it asserted. The 11 programs, each managed by different DHS components, were in one phase of the acquisition process or had two or more subprojects simultaneously in different phases of the acquisition process. For example, Coast Guard’s H-65 helicopter program has 6 discrete subprojects, each with its own completion schedule, including 4 subprojects in the Produce/Deploy/Support phase and 2 subprojects in the Obtain phase. Acquisition Program Management Division, TES, and component officials determine if subprojects need to develop separate sets of acquisition documents as they progress through the acquisition process. Figure 2 provides an overview of these programs and their associated acquisition phases. Additional details on these programs can be found in appendix I. As shown in figure 3, for the 11 selected DHS programs, TES reviewed and approved test and evaluation master plans for 6 of the 7 programs that were required to develop such plans by the T&E and acquisition directives and had documented their approval of these plans. For the one program that was in the phase that required such a plan—ATLAS Tactical Communications—the program had not yet drafted its test and evaluation master plan. The remaining 4 programs had plans in draft form that had not yet been submitted to TES for review. As a result, TES was not yet required to review these plans. Component officials from each of these six programs stated that TES provided input to the development of the test and evaluation master plans. For example, Office of Health Affairs officials stated that TES officials suggested that the BioWatch Gen-3 program office incorporate an additional test event to ensure that the program was tested under specific environmental conditions described in the program’s operational requirements document, which resulted in more tests. In addition, U.S. Customs and Border Protection (CBP) officials stated that TES participated in a line-by-line review of the SBInet test plan and provided detailed suggestions. Further, TES suggested that the criteria used for operational testing in the test and evaluation master plan needed to be expanded, and that an update may be required for SBInet to progress to the next acquisition phase. All of the component program officials who had undergone TES review or approval told us that TES test area managers provided their input in a variety of ways, including participating in T&E working groups, in specific meetings to discuss T&E issues, or by providing written comments culminating in TES’s approval of the plan. After the test and evaluation master plan is developed, the test agent is to develop operational test plans, which detail field testing of the system under realistic conditions for determining that the system’s overall effectiveness and suitability for use before deployment of the system. As shown in figure 4, of the 11 selected acquisition programs, TES reviewed and approved operational test plans for the 4 programs that were required to develop such plans by the acquisition directive and documented their approval of these plans. Component officials from these 4 programs said that TES provided input into their test plans. For example, National Protection and Programs Directorate officials from the National Cybersecurity Protection System program stated that TES had significant comments on their operational test plan, such as including confidence levels associated with the syste key performance requirements and helping program officials select a sample size necessary to measure statistically significant results. In addition, TES officials requested that the plan include different testing scenarios in order to demonstrate a varied use of the system. In a officials from the Transportation Security Administration’s (TSA) Advanced Technology-2 program indicated that TES provided significan input to their plan through a working group. The remaining 7 of the 11 programs had not yet begun to develop their operational test plan. At the conclusion of operational testing, the test agent is to write a re on the results of the test. The T&E directive specifies that TES is to receive the operational test report, which is to address all the critic issues and provide an evaluation of the operational suitability and operational effectiveness of the system. After reviewing the operatio test report, TES then is to write a letter of assessment—which is an d independent assessment of the adequacy of the operational test an provides TES’s concurrence or nonconcurrence on the test agent evaluation of operational suitability and operational effectiveness. TES is to provide the letter of assessment to the ARB as it is determining whe a program should progress to the production and deployment phase. Of the 11 programs we selected to review, TES developed a letter of assessment for the 1 program— TSA’s Advanced Technology 2 —that had completed operational testing and had a written operational T&E report on the results. The assessment concluded that while the T&E activities were adequate to inform the ARB as to system performance, TES did not concur with TSA’s test agent’s assessment as to system effectiveness because the system did not achieve a key performance parameter during testing. The ARB considered the letter of assessment and TES’s input and granted TSA permission to procure and deploy a limited number of screening machines. TSA will have to go before the ARB again to determine if full-scale production can proceed after TSA has provided the ARB with a business case and risk mitigation plan related to testing issues. The remaining 10 selected programs had not completed operational testing and thus, were not ready for letters of assessment. In addition to letters of assessment, TES officials told us that they regularly discuss T&E issues and concerns either verbally or through e- mails with Acquisition Program Management Division officials, who are responsible for organizing ARB meetings. For example, Acquisition Program Management Division officials stated that they rely on TES to provide candid information about the suitability of various programs’ T&E and whether these issues impact their program’s readiness to go before the ARB. Further, the officials told us that TES’s input at the ARBs, if any, is to be documented in acquisition decision memorandums. Acquisition Program Management Division officials also noted that TES’s input may be used in making the decision about when to hold an ARB for a particular program. T&E input from TES is one of many factors the ARB uses in overseeing acquisitions. For example, according to S&T officials, the ARB considers the current threat assessments and the extent to which the program, if implemented sooner, would help to address that threat. The ARB also considers factors such as the cost of the program and potential costs of conducting more testing and whether the results of operational testing were sufficient to achieve the intended benefits of the program. As a result, the ARB may accept a higher level of risk and allow a program to proceed even if testing concerns have been raised, if it determines that other reasons for quicker implementation outweigh these concerns. TES officials also stated that they work extensively with components prior to ARB meetings to ensure that T&E issues are addressed, with the goal to address these issues before going before the ARB. TES meets with component officials during regular acquisition review team meetings to resolve various issues before ARB meetings are convened. For example, due to concerns about the results of system qualification tests, TES recommended to SBInet program and ARB officials that the program should not proceed to the next milestone—site preparation, tower construction, and sensor and communication equipment installation at the Ajo, Arizona test site—until after operational testing was completed at the Tucson, Arizona test site. In May 2009, the ARB authorized SBInet to proceed with plans for the Ajo, Arizona site despite TES’s advice to the contrary, and directed TES to work with component officials to revise test plans, among other things. While TES’s oversight of the test plans and reports for major acquisition programs selected for review is in accordance with provisions in the T&E directive, it did not consistently document its review and approval of certain acquisition documentation or document the extent to which certain requirements in the T&E directive were met. The T&E directive requires that an operational test agent—a government agency or independent contractor carrying out independent operational testing for major acquisition programs—is to meet certain requirements to be qualified and approved by TES, but does not specify how TES’s approval is to be documented. According to the T&E directive, the test agent may be within the same component, another government agency, or a contractor, but is to be independent of the developer and the development contractor. Because the responsibilities of a test agent are significant throughout the T&E process, this independence is to allow the agent to present objective and unbiased conclusions regarding the system’s operational effectiveness and suitability to DHS decision makers, such as the ARB. For example, some the test agent’s responsibilities in the T&E directive include: Being involved early in the acquisition cycle by reviewing draft requirements documents to help ensure that requirements are testable and measurable. Assisting the component program manager in the preparation of the test and evaluation master plan. Planning, coordinating, and conducting operational tests, and preparing the operational T&E report. Reporting operational test results to the program manager and TES. According to TES officials, the test agent is also to meet other requirements in order to be approved by TES, such as having the expertise or knowledge about the product being tested and having the capacity and resources to execute the operational tests. To ensure that criteria for test agents are met, the T&E directive requires TES to approve all agents for major acquisition programs. As shown in figure 5, of the 11 programs we reviewed, 8 programs had selected a test agent and the others were in the process of selecting a test agent. TES provided documentation, such as memoranda, of its approval for 3 of these 8 programs. For the remaining 5 programs, there was no documentation of the extent to which these test agents had met the criteria and that TES had approved them. According to TES officials, they did not have a mechanism in place requiring a consistent method for documenting their review and approval of component agents or the extent to which criteria used in reviewing these agents were met. In the absence of such a mechanism in fiscal year 2010, TES’s approval of test agents was not consistently documented. TES and component officials stated that the approval for the five programs was implicit or provided verbally without documentation regarding whether the test agent met the T&E directive requirements. The T&E directive states that the test agent is to be identified and approved as early as possible in the acquisition process to, among other things, assist the component program officials in developing the test and evaluation master plan and review draft requirements documents to provide feedback regarding the testability of proposed requirements. TES and component officials stated that they assumed that test agents were approved using various approaches. Specifically, of the five programs that had test agents sign the test and evaluation master plan, one program had documented approval from TES. For example, Coast Guard and Office of Health Affairs officials stated that they did not have explicit documentation of TES’s approval of their agents; however, they believed that TES’s approval was implicit when TES approved their test and evaluation master plan since the test agent and TES are both signatories on the plan. CBP and National Protection and Programs Directorate officials told us that TES provided verbal approval for their test agents. Since there is no mechanism requiring TES to document its approval of the agent, and approval was granted verbally, there is no institutional record for DHS or an independent third party to validate whether TES followed its criteria when approving these test agents and whether the test agent was identified and approved before the test and evaluation master plan and requirements documents were finalized, as outlined in the T&E directive. With regard to the three programs in which TES had documented its approval in memoranda, these memoranda detailed TES’s agreement or nonagreement with a particular agent and highlighted whether the agent met the criteria outlined in the T&E directive. For example, TES provided interim approval to all three of the programs with the conditions that the programs prove at a later date that the test agents met all the requirements. For example: In April 2010, TES wrote a memo and granted interim approval with “serious reservations” for 1 year to TSA’s test agent for the Passenger Screening program. In the memo, TES cited concerns about the organizational structure and the lack of independence of the test agent since the test agent was part of the same TSA office responsible for managing the program. The memo outlined several steps that TSA should take, including the implementation of interim measures, such as new procedures, to ensure the necessary independence critical to testing and evaluation efforts as required by DHS directives. TES officials told us that by documenting TES’s interim approval in a memo, they were able to communicate their concerns about the test agent’s independence to TSA and DHS decision makers and set forth interim measures that TSA needed to address regarding their concerns. In July 2010, TES granted conditional approval to the test agent for the U.S. Citizenship and Immigration Services’ (USCIS) Transformation program’s test agent. TES made its approval contingent on the program developing a plan to ensure that the test agent was familiar with the component’s business practices. According to TES officials, after component officials gave a briefing to TES, they determined that the test agent met the requirements and it was approved. In January 2011, TES granted conditional approval for the U.S. Secret Service’s Information Integration and Transformation program to bring its selected test agent on board. TES’s final approval will be given after program officials brief TES on the test agent’s operational testing approach, which is to demonstrate that the test agent has knowledge of the product and has the capacity to execute the tests. TES officials told us that they do not have approval memos for all of the test agents that have been hired by program offices since the T&E directive was implemented in May 2009. Because TES did not consistently document their approvals of test agents, it is unclear whether TES has ever disapproved a test agent. TES officials acknowledged that they did not consistently document that the test agents met T&E requirements and did not document their approval of test agents. TES officials said that it would be beneficial to do so to ensure that agents met the criteria required in the T&E directive. In addition, Standards for Internal Control in the Federal Government and associated guidance state that agencies should document key decisions in a way that is complete and accurate, and that allows decisions to be traced from initiation, through processing, to after completion. These standards further state that documentation of key decisions should be readily available for review. Without a mechanism for documenting its review and approval of test agents for major acquisition programs, it will be difficult for DHS or an independent third party to validate TES’s decision-making process to ensure that it is effectively overseeing component testing. Moreover, it will be difficult for TES to provide reasonable assurance that these agents met the criteria outlined in the T&E directive, such as the requirement that they be independent of the program being tested. In addition to reviewing and approving test plans, under the T&E directive, TES is required to review certain component acquisition documents, including the mission need statements, operational requirements document, concept of operations, and developmental test reports, amongst others. These documents, which are required at the need, Analyze/Select, and Obtain phases of the acquisition process, are to be reviewed by TES to assist component program managers in identifying and resolving technical, logistical, and operational issues early in the acquisition process and to ensure that these documents meet relevant criteria. Specifically, as outlined in the T&E directive, TES is to review the mission need statement to establish awareness of the program and help ensure that the required standards are developed and that the component has identified the appropriate resources and support needed to conduct testing. TES is also to review the operational requirements document, including the key performance parameters and critical operational issues that specify the operational effectiveness and operational suitability issues that the test agent is to examine in order to assess the system’s capability to perform the mission. Further, TES is to review the concept of operations, since this document describes how the technology or equipment will be used in an operating environment. TES is to review the developmental test reports to maintain knowledge of contractor testing and to assist in its determination of the program’s readiness to progress to operational testing. We have previously reported that inadequate attention to developing requirements results in requirements instability, which can ultimately cause cost escalation, schedule delays and fewer end items. Further, we reported that without the required development and review of key acquisition data, DHS cannot provide reasonable assurance that programs have mitigated risks to better ensure program outcomes. TES officials stated that they do not have a mechanism to document or track those that they did review, what criteria they used when reviewing these documents, and the extent to which the documents reviewed met those criteria. For the 11 DHS programs that we reviewed, 8 programs had component-approved mission need statements; 2 programs, Atlas Tactical Communications and Transformation, had not yet completed such statements; and 1 program, the initial SBInet program, had completed a mission need statement in October 2006 before the T&E directive was issued and did not develop a separate mission need statement for the Block 1 increment of the program. Of the 8 programs that had mission need statements, 6 components told us that they did not have evidence that TES reviewed the mission need statement in accordance with the T&E directive. Further, TES could not demonstrate that it had received or reviewed these documents. Since TES did not have documentation of its review, it is difficult to determine the extent to which the documents were reviewed and the extent to which these documents met the review criteria. TES officials told us that they do not usually provide substantial input into the mission need statements and that they receive these documents to establish awareness of a new program. Further, while one TES test area manager told us that he reviews all developmental test reports, another test area manger told us that some programs do not routinely send him developmental test reports. Also, for example, Secret Service officials said that for the Information Integration and Transformation program they provided the operational requirements document, concept of operations, and integrated logistic support plan to TES. Specifically, the officials said that TES officials were very helpful in providing input on draft documents and made improvements to the documents by suggesting, for example, that the tests be more realistic by including personnel from field offices, headquarters, and external agencies in the live/production test environment. In contrast, officials from TSA stated that while they provided their mission need statement, concept of operations, integrated logistics support plan, and acquisition program baseline documents for the Advanced Technology 2 (AT-2) program to TES, TES officials did not provide input or comments on any of those documents. TES officials told us that the AT-2 program was initiated and developed some acquisition documentation prior to May 2009 when the T&E directive was issued. Specifically the operational requirements document was approved and finalized by TSA in June 2008 prior to the T&E directive and provided later to TES in February 2010 when the program was being reviewed. When TES reviewed the operational requirements document along with other documents such as the test and evaluation master plan, TES wrote a memo to TSA in March 2010 requesting that detection performance requirements be clarified and that users concur with the requirements. After several months of discussion, TSA and TES agreed on an approach which was used as the basis for initial operational T&E. Standards for Internal Controls in the Federal Government, as outlined earlier, state that agencies should document key decisions, and further that documentation of key decisions should be readily available for review. TES officials stated that they do not have a mechanism requiring that they document their review of certain acquisition documentation or the extent to which the document met the criteria used in reviewing these documents, and recognized that doing so would be beneficial. Developing a mechanism for TES to document its review of key acquisition documents could better position TES to provide reasonable assurance that it is reviewing key documentation and providing input that is important for determining the outcome of future testing and evaluation efforts, as required by the T&E directive. Moreover, such a policy could help to ensure that an institutional record exists for DHS or an independent third party to use in determining whether TES is effectively overseeing component T&E efforts and assisting in managing DHS major acquisition programs. According to the T&E directive, TES is to conduct an independent assessment of the adequacy of an operational test, provide a concurrence or nonconcurrence on the test agent’s evaluation of operational suitability and operational effectiveness, and provide any further independent analysis it deems necessary for all major DHS acquisition programs. TES is to document this independent assessment by writing a letter of assessment within 30 days of receiving the operational test report from the components’ test agent and provide the letter of assessment to the ARB, who then uses the assessment in making its determination of whether the program can proceed to purchase and implementation. While TES has developed a letter of assessment for the two other programs undergoing an ARB decision to enter into the production and deployment phase since the T&E directive was issued in May 2009, TES officials told us that they do not plan to write such an assessment for the Advanced Spectroscopic Portal (ASP) program because they are the test agent for ASP and thus, are not in a position to independently assess the results of testing that they conducted. In April 2008, over a year before the T&E directive was issued, senior level executives from DHS, S&T, CBP, and the Domestic Nuclear Detection Office (DNDO) signed a memorandum of understanding regarding arrangements for ASP operational testing. The memo designated Pacific Northwest National Lab, a U.S. Department of Energy laboratory, as the test agent. However, the memo also outlined the roles and responsibilities of TES, many of which reflected the duties of a test agent, such as developing and approving all operational test plans, responsibility for the management of testing and field validation, and developing and approving operational test reports. TES officials told us that they were using Pacific Northwest National Lab staff to carry out the operational tests, but are acting, for all intents and purposes, as the test agent for ASP. TES and DNDO officials told us that this arrangement was made after repeated testing issues arose with the ASP program. In September 2008, we reported that ASP Phase 3 testing by DNDO provided little information about the actual performance capabilities of ASP and that the resulting test report should not be used in determining whether ASP was a significant improvement over currently deployed equipment. Specifically, we found that the ASP Phase 3 test results did not help determine an ASP’s “true” level of performance because DNDO did not design the tests to assess ASP performance with a high degree of statistical confidence. In response to our report, DHS convened an independent review team to assist the Secretary in determining whether he should certify that there will be a significant increase in operational effectiveness with the procurement of the ASP system. The independent review team found that the test results and measures of effectiveness were not properly linked to operational outcomes. In May 2009, we reported that DHS had increased the rigor of ASP testing in comparison with previous tests. For example, DNDO mitigated the potential for bias in performance testing (a concern we raised about prior testing) by stipulating that there would be no ASP contractor involvement in test execution. However, the testing still had limitations, such as a limited set of scenarios used in performance testing to conceal test objects from detection. Moreover, we also reported that TES was to have the lead role in the final phase of ASP testing. As of February 2011, TES officials told us that the final phase of testing, consisting of 21 days of continuous operation, had not yet been scheduled. With TES acting as the test agent, it is not in a position to exercise its responsibilities during the operational testing phase, such as approving the operational test plan or writing a letter of assessment of the final results of operational testing. As it has done for two other recent DHS acquisition programs, TES was able to confirm through its independent assessment whether the test agent conducted operational testing as described in the test and evaluation master plan and operational test plan. For example, TES outlined concerns in its letter of assessment to the ARB that the AT-2 system did not meet a stated operational requirement key performance parameter—a throughput measure of bags per hour—for the majority of the time under test which resulted in a “not effective” determination by TES. TES officials recognized that, as the test agent, they are not in a position to conduct an independent assessment of operational test results and write a letter of assessment for ASP and that they are the highest level organization within DHS for both T&E oversight and operational test expertise. They further stated that the decision to have TES serve as the test agent was made prior to the issuance of the T&E directive and that it was too late in the program’s development to go back and select another agent. Nevertheless, TES officials recognized that this one-time situation would result in the lack of an independent assessment of ASP test results and there were no plans to conduct or contract for such an independent assessment. While we acknowledge that this decision was made prior to the T&E directive and the requirement that TES write a letter of assessment of all major acquisition programs, it is nonetheless important that ASP undergo an independent assessment of its test results since its operational test plan, which was developed by TES, was not subject to oversight. Because ASP has faced testing issues, many of which we have reported on in past years, it is important that this program undergo oversight to help avoid similar problems from reoccurring. Without an independent assessment of ASP’s operational test results, it will be difficult to ensure that operational testing was properly planned, conducted, and that the performance results are useful. In addition, arranging for an independent assessment of operational tests results could provide the ARB with critical information on testing and evaluation efforts to help it determine whether ASP should be approved for purchase and implementation. TES and component officials reported challenges faced in coordinating and overseeing T&E across DHS components that fell into four primary categories: (1) ensuring that a program’s operational requirements—the key requirements that must be met for a program to achieve its intended goals—can be effectively tested; (2) working with DHS component program staff that have limited T&E expertise and experience; (3) using existing T&E directives and guidance to oversee complex information technology acquisitions; and (4) ensuring that components allow sufficient time and resources for T&E while remaining within program cost and schedule estimates. Both TES and DHS, more broadly, have begun initiatives to address some of these challenges, but it is too early to determine their effectiveness. Both TES and component officials stated that one of their challenges is developing requirements that are testable, consistent, accurate, and complete. Specifically, six of the nine TES test area managers told us that working with DHS components to ensure that operational requirements can be tested and are suitable to meet mission needs is important because requirements development is one of the biggest challenges facing DHS. For example, one TES test area manager described the difficulty in drafting a test and evaluation master plan if operational requirements are not testable and measurable. Another TES test area manager indicated that programs’ operational requirements documents often do not contain user needs or operational requirements for system performance. This leads to difficulties in testing those requirements later. Further, six of the nine TES test area managers cited that some components’ operational requirements are difficult to test as written, which results in delays in drafting T&E documents as well as impacting the program cost and schedule parameters. Our prior work has found that program performance cannot be accurately assessed without valid baseline requirements established at the program start. According to DHS guidance, the baseline requirements must include a threshold value that is the minimum acceptable value which, in the user’s judgment, is necessary to satisfy the need. In June 2010, we reported that if threshold values are not achieved, program performance is seriously degraded, the program may be too costly, or the program may no longer be timely. In addition, we reported that inadequate knowledge of program requirements is a key cause of poor acquisition outcomes, and as programs move into the produce and deploy phase of the acquisition process, problems become much more costly to fix. To help remedy these issues, we have made a number of recommendations to address them. DHS has generally agreed with these recommendations and, to varying degrees, has taken actions to address them. For example: In May 2010, we reported that not all of the SBInet operational requirements that pertain to Block 1—a surveillance, command, control, communications, and intelligence system being fielded in two portions of the international border in Arizona—were achievable, verifiable, unambiguous, and complete. For example, a November 2007 DHS assessment determined that 19 operational requirements, which form the basis for the lower-level requirements used to design and build the system, were not complete, achievable, verifiable, or affordable. Further, the DHS assessment noted that a requirement that the system should provide for complete coverage of the border was determined to be unverifiable and unaffordable because defining what complete coverage meant was too difficult and ensuring complete coverage, given the varied and difficult terrain along the border, was cost prohibitive. To address these issues, we recommended that the currently defined Block 1 requirements, including key performance parameters, are independently validated as complete, verifiable, and affordable and any limitations found in the requirements are addressed. Furthermore, CBP program officials told us that they recognized the difficulties they experienced with requirements development practices with the SBInet program. Within CBP, the Office of Technology, Innovation, and Acquisition has responsibility for managing the SBInet program. Office of Technology, Innovation, and Acquisition officials told us that their office was created to strengthen expertise in acquisition and program management of SBInet. In May 2009, we reported that ASP testing uncovered multiple problems in meeting the requirements for successful integration into operations at ports of entry. As a result, we recommended that DHS assess ASPs against the full potential of current equipment and revise the program schedule to allow time to conduct computer simulations of ASP’s capabilities and to uncover and resolve problems with ASPs before full-scale deployment. We also reported that other TSA technology projects were delayed because TSA had not consistently communicated clear requirements in order to test the technologies. We recommended that TSA evaluate whether current passenger screening procedures should be revised to require the use of appropriate screening procedures until it is determined that existing emerging technologies meet their functional requirements in an operational environment. In March 2011 testimony, the Under Secretary for S&T stated that S&T had begun working with the DHS Under Secretary for Management to use their collective expertise and resources to better address the “front end” of the acquisition cycle, namely, the translation of mission needs into testable requirements. Further, in response to this challenge, S&T has reorganized and established an Acquisition Support and Operations Analysis Group, which is to provide a full range of coordinated operations analysis, systems engineering, T&E, and standards development support for DHS components. In addition, TES’s T&E Council is currently focusing on the challenges related to requirements development. Specifically, TES test area managers have presented specific briefings to component officials at council meetings which provide information on how to better generate requirements. Further, in response to our previously mentioned report designating DHS on the high-risk list, DHS developed a strategy to, among other things, strengthen its requirements development process. DHS’s January 2011 strategy describes the establishment of a capabilities and requirements council to evaluate and approve operational requirements early in the acquisition process. Specifically, the capabilities and requirements council is to, among other things, reconcile disagreements across program offices and approve analyses of alternatives and operational requirement documents. We stated in a March 2011 response to DHS on its strategy that it was unclear how the introduction of new governance groups will streamline the process and address previously identified issues because it appeared that the governance groups are chaired by the Deputy Secretary and have many of the same participants. Since the S&T reorganization has only recently taken place and the T&E Council and the department’s strategy have only recently begun to address the challenge of requirements generation, it is too soon to determine the effectiveness of these actions in addressing this challenge. TES officials told us that T&E experience and expertise within DHS components varies, with some components possessing staff with extensive T&E experience and expertise and others having relatively little. For example, TES officials noted that the Coast Guard and TSA have T&E policies and procedures in place, as well as staff with extensive T&E experience, which limited their dependence on TES for T&E expertise. Other components in DHS told us they rely more on TES or contractors for T&E expertise. For the 11 DHS programs we reviewed, officials from components which do not have many acquisition programs, such as the Office of Intelligence and Analysis, reported needing more assistance from TES in identifying and selecting appropriate and qualified test agents, for example. Conversely, components with more acquisition programs, such as the Coast Guard, told us that they have well-established test agents and procedures in place, and require little guidance from TES. For example, we reported in April 2011 that most Coast Guard major acquisition programs leverage Navy expertise, in some way, to support a range of testing, engineering, and other program activities. Furthermore, CBP recently established a new office whose goal is to strengthen expertise in acquisition and program management, including T&E, and ensure that CBP’s technology efforts are focused on its mission and integrated across the agency. In response to this challenge, TES has worked with DHS’s Acquisition Workforce Office to develop T&E certification requirements and training for components. TES officials told us that they have worked with the Acquisition Workforce Branch and developed pilot courses on T&E for component T&E staff, including Fundamentals of Test and Evaluation, Intermediate Test and Evaluation, and Advanced Test and Evaluation. In April 2010, DHS issued an acquisition workforce policy which establishes the requirements and procedures for certification of DHS T&E managers. The policy allows T&E managers to be certified at a level that is commensurate with their education, training, and experience. Component staff from 6 of the 11 programs we reviewed said they participated in TES’s certification training program and believed that the training would assist them in carrying out their T&E responsibilities. In addition, TES is in the process of hiring four additional staff to assist the test area managers in their T&E oversight responsibilities and hoped to have the additional staff hired by the end of fiscal year 2011. Lack of DHS staff to conduct acquisition oversight, including T&E, is a departmentwide challenge. In our previous reports, DHS acquisition oversight officials said that funding and staffing levels have limited the number of programs they can review. We recommended that DHS identify and align sufficient management resources to implement oversight reviews in a timely manner. DHS generally concurred with the recommendation and, as of January 2011, has reported taking action to address it by identifying needed capabilities and hiring staff to fill identified gaps. Further, to address this challenge, in 2009 and 2010, T&E Council representatives from the Acquisition Workforce Branch made presentations at council meetings to update members on the status of various acquisition workforce issues, including T&E certification. For example, presenters asked T&E Council members to inform their respective components about new T&E certification courses and to provide information on how to sign up for the courses. In 2010, the Acquisition Workforce Policy was implemented by DHS, which allowed the department to begin to certify T&E acquisition personnel. While DHS has undertaken efforts to help address these challenges, it is too soon to evaluate the impact that these efforts will have in addressing them. Effectively managing IT acquisitions is a governmentwide challenge. TES and component officials we interviewed told us that T&E guidance, such as specific guidance for integrating developmental testing and operational testing, may not be sufficient for the acquisition of complex IT systems. Specifically, component officials stated that the assessment of risks and environmental factors are different for IT programs than other acquisitions and that conducting testing in an operational environment may not be necessary for IT programs because the operational environment is no different than the test environment. In addition, four of the nine test area managers told us that aspects of the existing T&E guidance may not directly apply to IT acquisitions. The department is in the process of making modifications to its acquisitions process to better accommodate information technology acquisitions. According to the previously mentioned January 2011 strategy submitted to GAO, DHS is piloting a new model for IT acquisitions. This model, which is to be consistent with the department’s overall acquisition governance process, is to have many of the steps in the modified process that are similar or the same as what currently exists but time frames for different types of acquisitions would be instituted. For example, acquisition programs designated as IT programs may go through a more streamlined acquisition process that may better fit the rapidly changing IT environment, and the ARB would have the option to delegate oversight responsibilities to an executive steering committee. In other cases, TES and component officials are investigating the possibility of conducting integrated testing—the combination of developmental and operational testing—for some programs although this process may take longer to plan and pose greater risks because testing is being done simultaneously. Further, the T&E Best Practices Integrated Working Group, a subgroup of the T&E Council, including TES, Acquisition Program Management Division, and Office of Chief Information Officer officials, was working to identify and promote T&E best practices for IT system acquisition. This group drafted an operational test agent risk assessment process to validate the streamlining process approach while adhering to acquisition and T&E policy and directives, and as of March 2011, one component, USCIS, has made use of this process. Additionally, three other programs are investigating the possible use of this process and the possibility of tailoring or eliminating T&E deliverables or operational T&E requirements for IT programs, with the approval of TES. The group has identified three IT acquisition programs to serve as a pilot for this effort. As DHS considers modifications to its T&E process for IT programs, it also must consider the effect such a change could have on determining a system's technical performance and evaluating the system's operational effectiveness and suitability. For example, we have previously reported on testing problems with SBInet, a CBP program designated as an IT program. We found that SBInet testing was not performed in a manner that would adequately ensure that the system would perform as intended. Among the factors contributing to these problems was insufficient time for reviewing and approving test documentation, which in part, led to test plans and test cases not being well-defined. As a result, we recommended that test schedules, plans, cases, and procedures are adequately reviewed and approved consistent with the revised test and evaluation master plan. Since the efforts DHS is taking to address this challenge have only recently been initiated, it is too early to tell what impact they will have on the overall challenges of T&E for IT programs. Both TES and component officials stated that balancing the need to conduct adequate T&E within the confines of a program’s costs and schedule is a recurring challenge, and a challenge that is difficult to solve. We have previously reported on the challenges associated with balancing the need to conduct testing within program cost and schedules. Our past review of the Department of Defense’s (DOD) Director of Operational Test and Evaluation found that while the acquisition community has three central objectives—performance, cost, and schedule—the Director of Operational Test and Evaluation has but one--operational testing of performance. We reported that these distinct priorities can lead to testing disputes. We reported that these disputes encompassed issues such as (1) how many and what types of test to conduct; (2) when testing should occur; (3) what data to collect, how to collect them, and how b to analyze them; and (4) what conclusions were supportable, given th e analysis and limitations of the test program. The foundation of most of these disputes laid in different notions of the costs and benefits of testing and the levels of risk that were acceptable when making full-rate production decisions. The DOD Director of Operational Test and Evaluation consistently urged more testing (and consequently more time, resources, and cost) to reduce the level of risk and number of unknowns before the decision to proceed to full-rate production, while the services consistently sought less testing and accepted more risk when making production decisions. These divergent dispositions frequently led to est healthy debates about the optimal test program, and in a small number of cases, the differences led to contentious working relations. TES and DHS component officials expressed views similar to those expressed in our past work at DOD. Of the nine TES test area managers we talked with, four told us that allowing appropriate time and resources for T&E within program cost and schedule is a challenge. According to the test area manager’s, component program management officials often do not incorporate sufficient time within their schedule for T&E or reduce the time allowed for T&E to save time and money. In one test area managers’ view, doing so can reduce the effectiveness of testing or negatively impact the results of the tests. However, TSA officials told us that TES wanted to insert new test requirements for the AT-2 program—including the involvement of more TSA staff in the tests—after the program schedule was established and it was difficult to accommodate the changes and resulted in some delays. TES officials told us that these test requirements were in lieu of other planned field testing, which were not consistent with the program’s concept of operations and that TSA officials agreed with the new test requirements. According to TES and component officials we spoke with, both the program officials and TES understand the views and perspectives of one another and recognize that a balance must be struck between effective T&E and managing programs within cost and schedule. As a result, TES is working with program officials through the T&E Council or T&E working groups to discuss these issues early in the acquisition cycle (before it is too late), particularly while developing the test and evaluation master plan, which outlines the time allowed for testing and evaluation. Timely and accurate information resulting from T&E of major acquisitions early in the acquisition process can provide valuable information to DHS’s senior level managers to make informed decisions about the development, procurement, deployment, and operation of DHS’s multibillion dollar portfolio of systems and services. Improving the oversight of component T&E activities is but one part of the significant challenges DHS faces in managing its acquisitions. Components themselves are ultimately responsible for the management and implementation of their programs and DHS senior level officials are responsible for making key acquisitions decisions which lead to production and deployment. TES helps support acquisition decisions by providing oversight over major acquisitions’ T&E, which can help reduce, but not eliminate, the risk that new systems will not be operationally effective and suitable. Since the Homeland Security Act creating DHS was enacted in 2002, S&T has had the responsibility for overseeing T&E activities across the department. However, S&T did not have staff or the acquisition and T&E directives in place to conduct such oversight across DHS components until May 2009 when DHS issued its T&E directive. Since then, TES has implemented some of the requirements and overseen T&E of major acquisitions we reviewed, as well as provided independent assessments of operational test results to the ARB. However, TES has not consistently documented its compliance with the directives. Documenting that TES is fulfilling the requirements within DHS acquisition and T&E directives and the extent to which the criteria it is using to review and approve these documents are met, including approving operational test agents and reviewing key acquisition documentation, would assist TES in demonstrating that it is conducting T&E oversight and meeting requirements in these directives. Furthermore, without an independent assessment of operational test results for the Advance Spectroscopic Portal program, a key T&E oversight requirement in the T&E directive, the ARB will lack T&E oversight and input it needs to determine whether ASP is ready to progress toward production and deployment. This is especially important, given that program’s troubled history, which we have highlighted in a series of prior reports. To better ensure that testing and evaluation requirements are met, we recommend that the Secretary of Homeland Security direct the Under Secretary for Science & Technology to take the following two actions: Develop a mechanism to ensure that TES documents its approval of operational test agents and the extent that the test agents meet the requirements in the T&E directive, and criteria that TES use in reviewing these test agents for major acquisition programs. Develop a mechanism to ensure that TES documents its required review of component acquisition documents, including the mission need statements, concept of operations, operational requirements documents, developmental test reports, test plans, and other documentation required by the T&E directive, the extent that these documents meet the requirements in the T&E directive, and criteria that TES uses in reviewing these documents. To ensure that the ARB is provided with an independent assessment of the operational test results of the Advanced Spectroscopic Portal program to help determine whether the program should be approved for purchase and implementation, we recommend that the Secretary of Homeland Security take the following action: Arrange for an independent assessment, as required by the T&E directive, of ASP’s operational test results, to include an assessment of the adequacy of the operational test and a concurrence or nonconcurrence on the operational test agent’s evaluation of operational suitability and operational effectiveness. We received written comments on a draft of this report from DHS on June 10, 2011, which are reproduced in full in appendix III. DHS concurred with all three of our recommendations. DHS concurred with our first recommendation (1) that S&T develop a mechanism to ensure that TES documents its approval of operational test agents, (2) the extent that the test agents meet the requirements in the T&E directive, and (3) the criteria that TES uses in reviewing these test agents for major acquisition programs. Specifically, DHS stated that the Director of TES issued a memorandum to test area managers and TES staff regarding the operational test agent approval process which describes the responsibilities, considerations for selection, and the process necessary to select an operational test agent. In addition, DHS stated that TES is drafting memos approving operational test agents using the new test agent approval process. DHS also concurred with our second recommendation that S&T develop a mechanism to ensure that TES documents (1) its required review of component acquisition documents required by the T&E directive, (2) the extent that these documents meet the requirements in the T&E directive, and (3) the criteria that TES uses in reviewing these documents. DHS stated that the Director of TES issued a memorandum to test area managers and TES staff detailing the role of TES in the document review process and the process that TES staff should follow for submitting their comments to these documents. Finally, DHS concurred with our third recommendation that S&T arrange for an independent assessment of ASP’s operational test results. DHS stated that the ASP program is under review and does not have an operational test scheduled. However, TES is investigating the option of using a separate test agent to conduct operational testing of ASP, which would allow TES to perform the independent assessment and fulfill its independent oversight role as outlined in DHS policy. Such actions, if taken, will fulfill the intent of this recommendation. DHS also provided technical comments on the report, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to interested congressional committees and the Secretary of Homeland Security. The report will be available at no charge on GAO’s Web site at http://www.gao.gov. If you or your staff have questions regarding this report, please contact me at (202) 512-9627 or at maurerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV. An effort to develop nd deploy technologie to llow Custom nd Border Protection to detect ncler or rdiologicl mteri from conveynce, such as trck, entering the United Ste t lnd nd port of entry. U.S. Immigrtion nd Custom Enforcement (ICE) An effort to modernize ICE’scticl commniction tem nd equipment, tht Ice gent nd officer use to support mission-criticl commniction from otdted log tem to modern nd ndrdized digittem. Project 25 pgrde will modernize tcticl commniction nd deploy ite infrastrctre nd end-user subscrier rdio. Interoperable Rpid Deployment Stem (IRDS) will otfit ICE with trportable commniction tem to support rpid deployment requirement fro rotine, emergency nd disaster repone, nd pecil opertion. The progrm i divided into ix egment, inclding: (1) P25 pgrde for the Atlnt Region, (2) P25 pgrde for the Boton Region, () P25 pgrde for the Denver Region, (4) P25 pgrde for the centrl hub infrastrctre, (5) IRDS moile rdio commniction kit thsupport disaster nd emergency repone opertion, nd (6) IRDS Moile Commniction Stem (MCS) moile commniction vehicle thsupport disaster nd emergency repone opertion. In Mrch 2011, the Component Acquition Exective determined tht the TACCOM progrm wold e conolidted with other ICE infrastrctre progr nd tht the progrm wold e required to submit quition docmenttion to the ARB prior to Aust 2011. Segment 1: Prodce/ Deploy/ Support Segment 2-6: In the process of eing pdted. A ntionwide, interoperting network of detector/identifier tht i to provide autonomous ir-sampling ly of the environment for iologicgent of concern. The tem i to enable detection, identifiction, nd reporting of recognized orgni within 6-hor period. (4) Obsolete Component Moderniztion (OCM) – Replce obsolete component nd substem; (5) Ship Helicopter Secre Trvere & Stem (SHSTS) – Provide the ability to automticlly ecre the ircrft to the flight deck nd trvere it into the hnger; nd, (6) Atomtic Flight Control Stem (AFCS/Avionic) Modernize digitl Common Avionic Architectre Stem (CAAS) common with the H-60T pgrde nd digitl Atomtic Flight Control Stem. An effort to modernize Secret Service’s IT infrastrctre, commniction tem, ppliction, nd process. The progrm i divided into for dicrete egment: (1) Enabling Cabilitie: IT Infrastrctre Moderniztion/Cyer Secrity/ Dabase Architectre; (2) Commniction Cabilitie; () Control Cabilitie; nd, (4) Mission Support Cabilitie. In Feuary 2011, the ARB grnted quition deciion event 2A nd quition deciion event 2B deciion for the Enabling Cabilitie egment. The remining three egment remined in the Anlyze/Select phase. An integrted tem of intrusion detection, lyticl, intrusion prevention, nd informtion-ring cabilitie thre to used to defend the federl civilin government’s informtion technology infrastrctre from cyer thre. Inclde the hrdwre, oftwre, supporting process, trining, nd ervice thre to e developed nd quired to support the mission. The inititem, known as Eintein, was renmed as Block 1.0 nd incldeabilitie such as centrlized d torge. (1) Block 2.0 i to dd n Intrusion Detection Stem (IDS) which i to assss network trffic for the preence of mlicious ctivity; (2) Block 2.1 i to provide Secrity Incident nd Event Mgement which i to enable d ggregtion, correltion, nd visualiztion. () Block .0 i to provide n intrusion prevention cability. A joint inititive etween Intelligence nd Anly (I&A) nd the Office of the Chief Informtion Officer which i to ring nified, enterpripproch to the mgement of ll classified informtion technology infrastrctre inclding: (1) Homelnd Secre D Network (HSDN) for ecret level commniction infrastrctre; (2) Homelnd Top Secret Network (HTSN) for top ecret commniction infrastrctre; nd () Homelnd Secre Commniction (HSC) for classified voice nd video teleconference cabilitie. A next genertion of x-ry technology tht i to complement the trditionl x-ry technology nd provide new technicl cabilitie, such as automted detection lgorithm, thret imge projection, lternte viewing tion, bulk exploive lgorithm, nd expnded thret lit tht incorporte emerging thre to vition ecrity. A tem which i to provide Trporttion Secrity Officerability to creen passenger' crry-on baggge irporttionwide. A progrm which i to deliver surveillnce nd deciion-support technologie tht crete virtual fence nd ituationreness long the U.S. order with Mexico nd C. The firSBInet deployment of the Block I tem took plce in the Ton, Arizon tion. The econd deployment of the Block I tem took plce in the Ajo, Arizon tion. In Juary 2011, the SBInet progrm ended as originlly conceived; however, limited deployment of technology, inclding 15 enor tower nd 10 commniction tower, remined deployed nd opertionl in Arizon. The T&E result on thee tower were to e reported ometime in April 2011. TASC i to develop nd field n integrted finncil mgement, asset mgement, nd procrement mgement tem oltion. The progrm i to usndrd business process nd ingle line of cconting complint with the common governmentwide cconting classifiction trctre. The TASC Exective Steering Committee determined tht the Federl Emergency Mgement Agency will e the firt DHS component to migrte to TASC. U.S. Citizenhip nd Immigrtion Service (USCIS) An effort to move immigrtion ervice from per-based model to n electronic environment. The progrm i to deliver implified, We-based tem for enefit eeker to submit nd trck their ppliction. The new, ccont-based tem i to provide customer with improved ervice. In fiscal year 2010, there were 86 acquisition programs on the Acquisition Program Management Division’s oversight list, which included the acquisition level and designation as an information technology acquisition. Table 2 lists information on these 86 acquisition programs, and in addition, includes information on the acquisition phase for each program as of April 2011 and whether the program was subject to the test and evaluation (T&E) directive. For example, some programs, such as Customs and Border Protection’s acquisition of Border Patrol Facilities would not involve any T&E activities and therefore would not be subject to the requirements in the T&E directive or DHS Science and Technology Directorate’s Test and Evaluation and Standards office (TES) oversight. In addition to the contact named above, Christopher Currie (Assistant Director), Nancy Kawahara, Bintou Njie, Melissa Bogar, Jessica Drucker, Caitlin White, Richard Hung, Michele Fejfar, Labony Chakraborty, Tracey King, Paula Moore, Dan Gordon, Michele Mackin, Molly Traci, and Sean Seales made significant contributions to this report.

Summary: In recent years, GAO has reported on challenges the Department of Homeland Security (DHS) has faced in effectively managing major acquisitions, including programs which were deployed before appropriate testing and evaluation (T&E) was completed. In 2009 and 2010 respectively, DHS issued new T&E and acquisition directives to address these challenges. Under these directives, DHS Science and Technology Directorate's (S&T) Test & Evaluation and Standards Office (TES) is responsible for overseeing T&E of DHS major acquisition programs--that is, those with over $300 million in life-cycle costs--to ensure that T&E and certain acquisitions requirements are met. GAO was asked to identify (1) the extent to which TES oversees T&E of major acquisitions; and (2) what challenges, if any, TES officials report facing in overseeing T&E across DHS components. GAO reviewed DHS directives and test plans, interviewed DHS officials, and reviewed T&E documentation from a sample of 11 major acquisition programs from each of 11 different DHS components. The results of the sample cannot be generalized to all DHS programs, but provided insights. TES met some of its oversight requirements for T&E of acquisition programs GAO reviewed, but additional steps are needed to ensure that all requirements are met. Specifically, since DHS issued the T&E directive in May 2009, TES has reviewed or approved T&E documents and plans for programs undergoing testing, and conducted independent assessments for the programs that completed operational testing during this time period. TES officials told GAO that they also provided input and reviewed other T&E documentation, such as components' documents describing the programs' performance requirements, as required by the T&E directive. DHS senior level officials considered TES's T&E assessments and input in deciding whether programs were ready to proceed to the next acquisition phase. However, TES did not consistently document its review and approval of components' test agents--a government entity or independent contractor carrying out independent operational testing for a major acquisition--or document its review of other component acquisition documents, such as those establishing programs' operational requirements, as required by the T&E directive. For example, 8 of the 11 acquisition programs GAO reviewed had hired test agents, but documentation of TES approval of these agents existed for only 3 of these 8 programs. Approving test agents is important to ensure that they are independent of the program and that they meet requirements of the T&E directive. TES officials agreed that they did not have a mechanism in place requiring a consistent method for documenting their review or approval and the extent to which the review or approval criteria were met. Without mechanisms in place for documenting its review or approval of acquisition documents and T&E requirements, such as approving test agents, it is difficult for DHS or a third party to review and validate TES's decision-making process and ensure that it is overseeing components' T&E efforts in accordance with acquisition and T&E directives and internal control standards for government entities. TES and DHS component officials stated that they face challenges in overseeing T&E across DHS components which fell into 4 categories: (1) ensuring that a program's operational requirements--the key performance requirements that must be met for a program to achieve its intended goals-- can be effectively tested; (2) working with DHS component program staff who have limited T&E expertise and experience; (3) using existing T&E directives and guidance to oversee complex information technology acquisitions; and (4) ensuring that components allow sufficient time for T&E while remaining within program cost and schedule estimates. Both TES and DHS, more broadly, have begun initiatives to address some of these challenges, such as establishing a T&E council to disseminate best practices to component program managers, and developing specific guidance for testing and evaluating information technology acquisitions. In addition, S&T has reorganized to assist components in developing requirements that can be tested, among other things. However, since these efforts have only recently been initiated to address these DHS-wide challenges, it is too soon to determine their effectiveness. GAO recommends, among other things, that S&T develop mechanisms for TES to document its review or approval of component acquisition documentation and T&E requirements, such as approving operational test agents. DHS agreed with GAO's recommendations.