The fiscal year 2005 expenditure plan satisfied or partially satisfied the conditions specified in DHS’s appropriations act. Specifically, the plan, including related program documentation and program officials’ statements, satisfied or provided for satisfying all key aspects of (1) meeting the capital planning and investment control review requirements of the Office of Management and Budget (OMB) and (2) review and approval by DHS and OMB. The plan partially satisfied the conditions that specify (1) compliance with the DHS enterprise architecture and (2) compliance with the acquisition rules, requirements, guidelines, and systems acquisition management practices of the federal government. CBP is working toward addressing our open recommendations. Each recommendation, along with the status of actions to address it, is summarized below. Develop and implement a rigorous and analytically verifiable cost- estimating program that embodies the tenets of effective estimating as defined in the Software Engineering Institute’s (SEI) institutional and project-specific estimating models. The CBP Modernization Office’s (CBPMO) implementation of this recommendation is in progress. CBPMO has (1) defined and documented processes for estimating expenditure plan costs (including management reserve costs); (2) hired a contractor to develop cost estimates, including contract task orders, that are independent of the ACE development contractor’s estimates; and (3) tasked a support contractor with evaluating the independent estimates and the development contractor’s estimates against SEI criteria. According to the summary-level results of this evaluation, the independent estimates either satisfied or partially satisfied the SEI criteria, and the development contractor’s estimates satisfied or partially satisfied all but two of the seven SEI criteria. Ensure that future expenditure plans are based on cost estimates that are reconciled with independent cost estimates. CBPMO’s implementation of this recommendation is complete with respect to the fiscal year 2005 expenditure plan. In August 2004, CBP’s support contractor completed an analysis comparing the cost estimates in the fiscal year 2005 expenditure plan (which are based on the ACE development contractor’s cost estimates) with the estimate prepared by CBPMO’s independent cost estimating contractor; this analysis concluded that the two estimates are consistent. Immediately develop and implement a human capital management strategy that provides both near- and long-term solutions to the program office’s human capital capacity limitations, and report quarterly to the appropriations committees on the progress of efforts to do so. CBPMO’s implementation of this recommendation is in progress, and it has reported on its actions to the Congress. Following our recommendation, CBPMO provided reports dated March 31, 2004, and June 30, 2004, to the appropriations committees on its human capital activities, including development of a staffing plan that identifies the positions it needs to manage ACE. However, in December 2004, CBPMO implemented a reorganization of the modernization office, which makes the staffing plan out of date. As part of this reorganization, CBP transferred government and contractor personnel who have responsibility for the Automated Commercial System, the Automated Targeting System, and ACE training from non-CBPMO organizational units to CBPMO. According to CBPMO, this change is expected to eliminate redundant ACE-related program management efforts. Have future ACE expenditure plans specifically address any proposals or plans, whether tentative or approved, for extending and using ACE infrastructure to support other homeland security applications, including any impact on ACE of such proposals and plans. CBP’s implementation of this recommendation is in progress. In our fiscal year 2004 expenditure plan review, we reported that CBPMO had discussed collaboration opportunities with DHS’s United States Visitor and Immigrant Status Indicator Technology (US-VISIT) program to address the potential for ACE infrastructure, data, and applications to support US- VISIT. Since then, ACE and US-VISIT managers have again met to identify potential areas for collaboration between the two programs and to clarify how the programs can best support the DHS mission. The US-VISIT and ACE programs have formed collaboration teams that have drafted team charters, identified specific collaboration opportunities, developed timelines and next steps, and briefed ACE and US-VISIT program officials on the teams’ progress and activities. Establish an independent verification and validation (IV&V) function to assist CBP in overseeing contractor efforts, such as testing, and ensure the independence of the IV&V agent. CBP has completed its implementation of this recommendation. To ensure independence, CBPMO has selected an IV&V contractor that, according to CBP officials, has had no prior involvement in the modernization program. The IV&V contractor is to be responsible for reviewing ACE products and management processes and is to report directly to the CBP chief information officer. Define metrics, and collect and use associated measurements, for determining whether prior and future program management improvements are successful. CBPMO’s implementation of this recommendation is in progress. CBPMO has implemented a program that generally focuses on measuring the ACE development contractor’s performance through the use of earned value management, metrics for the timeliness and quality of deliverables, and risk and issue disposition reporting. Additionally, it is planning to broaden its program to encompass metrics and measures for determining progress toward achieving desired business results and acquisition process maturity. The plan for expanding the metrics program is scheduled for approval in early 2005. Reconsider the ACE acquisition schedule and cost estimates in light of early release problems, including these early releases’ cascading effects on future releases and their relatively small size compared to later releases, and in light of the need to avoid the past levels of concurrency among activities within and between releases. CBP has completed its implementation of this recommendation. In response to the cost overrun on Releases 3 and 4, CBPMO and the ACE development contractor established a new cost baseline of $196 million for these releases, extended the associated baseline schedule, and began reporting schedule and cost performance relative to the new baselines. Additionally, in July 2004, a new version of the ACE Program Plan was developed that rebaselined the ACE program, extending delivery of the last ACE release from fiscal year 2007 to fiscal year 2010, adding a new screening and targeting release, and increasing the ACE life-cycle cost estimate by about $1 billion to $3.1 billion. Last, the new program schedule reflects less concurrency between future releases. Report quarterly to the House and Senate Appropriations Committees on efforts to address open GAO recommendations. CBP’s implementation of this recommendation is in progress. CBP has submitted reports to the committees on its efforts to address open GAO recommendations for the quarters ending March 31, 2004, and June 30, 2004. CBPMO plans to submit a report for the quarter ending September 30, 2004, after it is approved by DHS and OMB. We made observations related to ACE performance, use, testing, development, cost and schedule performance, and expenditure planning. An overview of the observations follows: Initial ACE releases have largely met a key service level agreement. According to a service level agreement between the ACE development contractor and CBPMO, 99.9 percent of all ACE transactions are to be executed successfully each day. The development contractor reports that ACE has met this requirement on all but 11 days since February 1, 2004, and attributed one problem that accounted for 5 successive days during which the service level agreement was not met to CBPMO’s focus on meeting schedule commitments. Progress toward establishing ACE user accounts has not met expectations. CBPMO established a goal of activating 1,100 ACE importer accounts by February 25, 2005, when Release 4 is to become operational. Weekly targets were established to help measure CBPMO’s progress toward reaching the overall goal. However, CBPMO has not reached any of its weekly targets, and the gap between the actual and targeted number of activated accounts has continued to grow. To illustrate, as of November 26, 2004, the goal was 600 activated accounts and the actual number was 311. Release 3 testing and pilot activities were delayed and have produced system defect trends that raise questions about decisions to pass key milestones and about the state of system maturity. Release 3 test phases and pilot activities were delayed and revealed system defects, some of which remained open at the time decisions were made to pass key life- cycle milestones. In particular, we observed the following: Release 3 integration testing started later than planned, took longer than expected, and was declared successful despite open defects that prevented the system from performing as intended. For example, the test readiness milestone was passed despite the presence of 90 severe defects. Release 3 acceptance testing started later than planned, concluded later than planned, and was declared successful despite having a material inventory of open defects. For example, the production readiness milestone was passed despite the presence of 18 severe defects. Release 3 pilot activities, including user acceptance testing, were declared successful, despite the presence of severe defects. For example, the operational readiness milestone was passed despite the presence of 6 severe defects. The current state of Release 3 maturity is unclear because defect data reported since user acceptance testing are not reliable. Release 4 test phases were delayed and overlapped, and revealed a higher than expected volume and significance of defects, raising questions about decisions to pass key milestones and about the state of system maturity. In particular, we observed the following: Release 4 testing revealed a considerably higher than expected number of material defects. Specifically, 3,059 material defects were reported, compared with the 1,453 estimated, as of the November 23, 2004, production readiness milestone. Changes in the Release 4 integration and acceptance testing schedule resulted in tests being conducted concurrently. As we previously reported, concurrent test activities increase risk and have contributed to past ACE cost and schedule problems. The defect profile for Release 4 shows improvements in resolving defects, but critical and severe defects remain in the operational system. Specifically, as of November 30, 2004, which was about 1.5 weeks from deployment of the Release 4 pilot period, 33 material defects were present.