The fiscal year 2005 expenditure plan satisfied or partially satisfied the conditions specified in DHS’s appropriations act. Specifically, the plan, including related program documentation and program officials’ statements, satisfied or provided for satisfying all key aspects of (1) meeting the capital planning and investment control review requirements of the Office of Management and Budget (OMB) and (2) review and approval by DHS and OMB. The plan partially satisfied the conditions that specify (1) compliance with the DHS enterprise architecture and (2) compliance with the acquisition rules, requirements, guidelines, and systems acquisition management practices of the federal government. CBP is working toward addressing our open recommendations. Each recommendation, along with the status of actions to address it, is summarized below. Develop and implement a rigorous and analytically verifiable cost- estimating program that embodies the tenets of effective estimating as defined in the Software Engineering Institute’s (SEI) institutional and project-specific estimating models. The CBP Modernization Office’s (CBPMO) implementation of this recommendation is in progress. CBPMO has (1) defined and documented processes for estimating expenditure plan costs (including management reserve costs); (2) hired a contractor to develop cost estimates, including contract task orders, that are independent of the ACE development contractor’s estimates; and (3) tasked a support contractor with evaluating the independent estimates and the development contractor’s estimates against SEI criteria. According to the summary-level results of this evaluation, the independent estimates either satisfied or partially satisfied the SEI criteria, and the development contractor’s estimates satisfied or partially satisfied all but two of the seven SEI criteria. Ensure that future expenditure plans are based on cost estimates that are reconciled with independent cost estimates. CBPMO’s implementation of this recommendation is complete with respect to the fiscal year 2005 expenditure plan. In August 2004, CBP’s support contractor completed an analysis comparing the cost estimates in the fiscal year 2005 expenditure plan (which are based on the ACE development contractor’s cost estimates) with the estimate prepared by CBPMO’s independent cost estimating contractor; this analysis concluded that the two estimates are consistent. Immediately develop and implement a human capital management strategy that provides both near- and long-term solutions to the program office’s human capital capacity limitations, and report quarterly to the appropriations committees on the progress of efforts to do so. CBPMO’s implementation of this recommendation is in progress, and it has reported on its actions to the Congress. Following our recommendation, CBPMO provided reports dated March 31, 2004, and June 30, 2004, to the appropriations committees on its human capital activities, including development of a staffing plan that identifies the positions it needs to manage ACE. However, in December 2004, CBPMO implemented a reorganization of the modernization office, which makes the staffing plan out of date. As part of this reorganization, CBP transferred government and contractor personnel who have responsibility for the Automated Commercial System, the Automated Targeting System, and ACE training from non-CBPMO organizational units to CBPMO. According to CBPMO, this change is expected to eliminate redundant ACE-related program management efforts. Have future ACE expenditure plans specifically address any proposals or plans, whether tentative or approved, for extending and using ACE infrastructure to support other homeland security applications, including any impact on ACE of such proposals and plans. CBP’s implementation of this recommendation is in progress. In our fiscal year 2004 expenditure plan review, we reported that CBPMO had discussed collaboration opportunities with DHS’s United States Visitor and Immigrant Status Indicator Technology (US-VISIT) program to address the potential for ACE infrastructure, data, and applications to support US- VISIT. Since then, ACE and US-VISIT managers have again met to identify potential areas for collaboration between the two programs and to clarify how the programs can best support the DHS mission. The US-VISIT and ACE programs have formed collaboration teams that have drafted team charters, identified specific collaboration opportunities, developed timelines and next steps, and briefed ACE and US-VISIT program officials on the teams’ progress and activities. Establish an independent verification and validation (IV&V) function to assist CBP in overseeing contractor efforts, such as testing, and ensure the independence of the IV&V agent. CBP has completed its implementation of this recommendation. To ensure independence, CBPMO has selected an IV&V contractor that, according to CBP officials, has had no prior involvement in the modernization program. The IV&V contractor is to be responsible for reviewing ACE products and management processes and is to report directly to the CBP chief information officer. Define metrics, and collect and use associated measurements, for determining whether prior and future program management improvements are successful. CBPMO’s implementation of this recommendation is in progress. CBPMO has implemented a program that generally focuses on measuring the ACE development contractor’s performance through the use of earned value management, metrics for the timeliness and quality of deliverables, and risk and issue disposition reporting. Additionally, it is planning to broaden its program to encompass metrics and measures for determining progress toward achieving desired business results and acquisition process maturity. The plan for expanding the metrics program is scheduled for approval in early 2005. Reconsider the ACE acquisition schedule and cost estimates in light of early release problems, including these early releases’ cascading effects on future releases and their relatively small size compared to later releases, and in light of the need to avoid the past levels of concurrency among activities within and between releases. CBP has completed its implementation of this recommendation. In response to the cost overrun on Releases 3 and 4, CBPMO and the ACE development contractor established a new cost baseline of $196 million for these releases, extended the associated baseline schedule, and began reporting schedule and cost performance relative to the new baselines. Additionally, in July 2004, a new version of the ACE Program Plan was developed that rebaselined the ACE program, extending delivery of the last ACE release from fiscal year 2007 to fiscal year 2010, adding a new screening and targeting release, and increasing the ACE life-cycle cost estimate by about $1 billion to $3.1 billion. Last, the new program schedule reflects less concurrency between future releases. Report quarterly to the House and Senate Appropriations Committees on efforts to address open GAO recommendations. CBP’s implementation of this recommendation is in progress. CBP has submitted reports to the committees on its efforts to address open GAO recommendations for the quarters ending March 31, 2004, and June 30, 2004. CBPMO plans to submit a report for the quarter ending September 30, 2004, after it is approved by DHS and OMB. We made observations related to ACE performance, use, testing, development, cost and schedule performance, and expenditure planning. An overview of the observations follows: Initial ACE releases have largely met a key service level agreement. According to a service level agreement between the ACE development contractor and CBPMO, 99.9 percent of all ACE transactions are to be executed successfully each day. The development contractor reports that ACE has met this requirement on all but 11 days since February 1, 2004, and attributed one problem that accounted for 5 successive days during which the service level agreement was not met to CBPMO’s focus on meeting schedule commitments. Progress toward establishing ACE user accounts has not met expectations. CBPMO established a goal of activating 1,100 ACE importer accounts by February 25, 2005, when Release 4 is to become operational. Weekly targets were established to help measure CBPMO’s progress toward reaching the overall goal. However, CBPMO has not reached any of its weekly targets, and the gap between the actual and targeted number of activated accounts has continued to grow. To illustrate, as of November 26, 2004, the goal was 600 activated accounts and the actual number was 311. Release 3 testing and pilot activities were delayed and have produced system defect trends that raise questions about decisions to pass key milestones and about the state of system maturity. Release 3 test phases and pilot activities were delayed and revealed system defects, some of which remained open at the time decisions were made to pass key life- cycle milestones. In particular, we observed the following: Release 3 integration testing started later than planned, took longer than expected, and was declared successful despite open defects that prevented the system from performing as intended. For example, the test readiness milestone was passed despite the presence of 90 severe defects. Release 3 acceptance testing started later than planned, concluded later than planned, and was declared successful despite having a material inventory of open defects. For example, the production readiness milestone was passed despite the presence of 18 severe defects. Release 3 pilot activities, including user acceptance testing, were declared successful, despite the presence of severe defects. For example, the operational readiness milestone was passed despite the presence of 6 severe defects. The current state of Release 3 maturity is unclear because defect data reported since user acceptance testing are not reliable. Release 4 test phases were delayed and overlapped, and revealed a higher than expected volume and significance of defects, raising questions about decisions to pass key milestones and about the state of system maturity. In particular, we observed the following: Release 4 testing revealed a considerably higher than expected number of material defects. Specifically, 3,059 material defects were reported, compared with the 1,453 estimated, as of the November 23, 2004, production readiness milestone. Changes in the Release 4 integration and acceptance testing schedule resulted in tests being conducted concurrently. As we previously reported, concurrent test activities increase risk and have contributed to past ACE cost and schedule problems. The defect profile for Release 4 shows improvements in resolving defects, but critical and severe defects remain in the operational system. Specifically, as of November 30, 2004, which was about 1.5 weeks from deployment of the Release 4 pilot period, 33 material defects were present. Performance against the revised cost and schedule estimates for Releases 3 and 4 has been mixed. Since the cost and schedule for Releases 3 and 4 were revised in April 2004, work has been completed under the budgeted cost, but it is being completed behind schedule. In order to improve the schedule performance, resources targeted for later releases have been retained on Release 4 longer than planned. While this has resulted in improved performance against the schedule, it has adversely affected cost performance. The fiscal year 2005 expenditure plan does not adequately describe progress against commitments (e.g., ACE capabilities, schedule, cost, and benefits) made in previous plans. In the fiscal year 2004 expenditure plan, CBPMO committed to, for example, acquiring infrastructure for ACE releases and to defining and designing an ACE release that was intended to provide additional account management functionality. However, the current plan described neither the status of infrastructure acquisition nor progress toward defining and designing the planned account management functionality. Also, the current plan included a schedule for developing ACE releases, but neither reported progress relative to the schedule presented in the fiscal year 2004 plan nor explained how the individual releases and their respective schedules were affected by the rebaselining that occurred after the fiscal year 2004 plan was submitted. Some key bases for the commitments made in the fiscal year 2005 expenditure plan have changed, raising questions as to the plan’s currency and relevance. Neither the expenditure plan nor the program plan reflected several program developments, including the following: A key Release 5 assumption made in the program and expenditure plans regarding development, and thus cost and delivery, of the multimodal manifest functionality is no longer valid. Additional releases, and thus cost and effort, are now planned that were not reflected in the program and expenditure plans. The current organizational change management approach is not fully reflected in program and expenditure plans, and key change management actions are not to be implemented. Significant changes to the respective roles and responsibilities of the ACE development contractor and CBPMO are not reflected in the program and expenditure plans. DHS and OMB have largely satisfied four of the five conditions associated with the fiscal year 2005 ACE expenditure plan that were legislated by the Congress, and we have satisfied the fifth condition. Further, CBPMO has continued to work toward implementing our prior recommendations aimed at improving management of the ACE program and thus the program’s chances of success. Nevertheless, progress has been slow in addressing some of our recommendations, such as the one encouraging proactive management of the relationships between ACE and other DHS border security programs, like US-VISIT. Given that these programs have made and will continue to make decisions that determine how they will operate, delays in managing their relationships will increase the chances that later system rework will eventually be required to allow the programs to interoperate. Additionally, while DHS has taken important actions to help address ACE release-by-release cost and schedule overruns that we previously identified, it is unlikely that the effect of these actions will prevent the past pattern of overruns from recurring. This is because DHS has met its recently revised cost and schedule commitments in part by relaxing system quality standards, so that milestones are being passed despite material system defects, and because correcting such defects will ultimately require the program to expend resources, such as people and test environments, at the expense of later system releases (some of which are now under way). In the near term, cost and schedule overruns on recent releases are being somewhat masked by the use of less stringent quality standards; ultimately, efforts to fix these defects will likely affect the delivery of later releases. Until accountability for ACE is redefined and measured in terms of all types of program commitments—system capabilities, benefits, costs, and schedules—the program will likely experience more cost and schedule overruns. During the last year, DHS’s accountability for ACE has been largely focused on meeting its cost and schedule baselines. This focus is revealed by the absence of information in the latest expenditure plan on progress against all commitments made in prior plans, particularly with regard to measurement and reporting on such things as system capabilities, use, and benefits. It is also shown by the program’s insufficient focus on system quality, as demonstrated by its willingness to pass milestones despite material defects, and by the absence of attention to the current defect profile for Release 3 (which is already deployed). Moreover, the commitments that DHS made in the fiscal year 2005 expenditure plan have been overcome by events, which limits the currency and relevance of this plan and its utility to the Congress as an accountability mechanism. As a result, the prospects of greater accountability in delivering against its capability, benefit, cost, and schedule commitments are limited. Therefore, it is critically important that DHS define for itself and the Congress an accountability framework for ACE, and that it manage and report in accordance with this framework. If it does not, the effects of the recent rebaselining of the program will be short lived, and the past pattern of ACE costing more and taking longer than planned will continue. To strengthen accountability for the ACE program and better ensure that future ACE releases deliver promised capabilities and benefits within budget and on time, we recommend that the DHS Secretary, through the Under Secretary for Border and Transportation Security, direct the Commissioner, Customs and Border Protection, to define and implement an ACE accountability framework that ensures coverage of all program commitment areas, including key expected or estimated system (1) capabilities, use, and quality; (2) benefits and mission value; (3) costs; and (4) milestones and schedules; currency, relevance, and completeness of all such commitments made to the Congress in expenditure plans; reliability of data relevant to measuring progress against commitments; reporting in future expenditure plans of progress against commitments contained in prior expenditure plans; use of criteria for exiting key readiness milestones that adequately consider indicators of system maturity, such as severity of open defects; and clear and unambiguous delineation of the respective roles and responsibilities of the government and the prime contractor. In written comments on a draft of this report signed by the Acting Director, Departmental GAO/OIG Liaison, DHS agreed with our findings concerning progress in addressing our prior recommendations. In addition, the department agreed with the new recommendations we are making in this report and described actions that it plans to take to enhance accountability for the program. These planned actions are consistent with our recommendations. DHS’s comments are reprinted in appendix II. We are sending copies of this report to the Chairmen and Ranking Minority Members of other Senate and House committees and subcommittees that have authorization and oversight responsibilities for homeland security. We are also sending copies to the Secretary of Homeland Security, the Under Secretary for Border and Transportation Security, the CBP Commissioner, and the Director of OMB. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov. Should you or your offices have any questions on matters discussed in this report, please contact me at (202) 512-3459 or at hiter@gao.gov. Other contacts and key contributors to this report are listed in appendix III. facilitate the movement of legitimate trade through more effective trade account management; strengthen border security by identifying import/export transactions that have an elevated risk of posing a threat to the United States or of violating a trade law or regulation; and provide a single system interface between the trade community2 and the federal government,3 known as the International Trade Data System (ITDS), and thereby reduce the data reporting burden placed on the trade community while also providing federal agencies with the data and various capabilities to support their respective international trade and transportation missions. CBP was formed from the former U.S. Customs Service and other entities with border protection responsibility. Members of the trade community include importers and exporters, brokers and trade advisors, and carriers. Includes federal agencies responsible for managing international trade and transportation processes. 1. meets the capital planning and investment control review requirements established by the Office of Management and Budget (OMB), including Circular A-11, part 7,2 2. complies with DHS’s enterprise architecture; 3. complies with the acquisition rules, requirements, guidelines, and systems acquisition management practices of the federal government; 4. is reviewed and approved by the DHS Investment Review Board (IRB),3 Secretary of Homeland Security, and OMB; and 5. is reviewed by GAO. OMB Circular A-11 establishes policy for planning, budgeting, acquisition, and management of federal capital assets. The purpose of the Investment Review Board is to integrate capital planning and investment control, budgeting, acquisition, and management of investments. It is also to ensure that spending on investments directly supports and furthers the mission and that this spending provides optimal benefits and capabilities to stakeholders and customers. determine whether the ACE fiscal year 2005 expenditure plan satisfies the determine the status of our open recommendations on ACE, and provide any other observations about the expenditure plan and DHS’s management of the ACE program. We conducted our work at CBP headquarters and contractor facilities in the Washington, D.C., metropolitan area from April 2004 through December 2004, in accordance with generally accepted government auditing standards. Details of our scope and methodology are provided in attachment 1. established by OMB, including OMB Circular A-11, part 7. 2. Complies with DHS’s enterprise architecture. 3. Complies with the acquisition rules, requirements, guidelines, and systems acquisition management practices of the federal government. 4. Is reviewed and approved by the DHS Investment Review Board, Secretary of Homeland Security, and OMB. 5. Is reviewed by GAO. With respect to the fiscal year 2005 expenditure plan. Initial ACE releases have largely met a key service level agreement. Progress toward establishing ACE user accounts has not met expectations. Release 3 testing and pilot activities were delayed and have produced system defect trends that raise questions about decisions to pass key milestones and about the state of system maturity. Release 3 integration testing started later than planned, took longer than expected, and was declared successful despite open defects that prevented system from performing as intended. Release 3 acceptance testing started later than planned, concluded later than planned, and was declared successful despite material inventory of open defects. Release 3 pilot activities, including user acceptance testing, were declared successful despite severe defects remaining open. Current state of Release 3 maturity is unclear because defect data since user acceptance testing are not reliable. Release 4 test phases were delayed and overlapped, and revealed a higher than expected volume and significance of defects, raising questions about decisions to pass key milestones and about the state of system maturity. Release 4 testing revealed a considerably higher than expected number of material defects. Release 4 integration and acceptance testing schedule changes resulted in tests being conducted concurrently. Release 4 defect profile shows improvements in resolving defects, but critical and severe defects remain in operational system. Performance against the revised cost and schedule estimates for Releases 3 and 4 has been mixed. The fiscal year 2005 expenditure plan does not adequately describe progress against commitments (e.g., ACE capabilities, schedule, cost, and benefits) made in previous plans. Some key bases for the commitments made in the fiscal year 2005 expenditure plan have changed, raising questions as to the plan’s currency and relevance. A key Release 5 assumption underpinning program and expenditure plans is no longer valid. Additional release(s) are now planned that were not reflected in the program and expenditure plans. The current organizational change management approach is not fully reflected in program and expenditure plans, and key change management actions are not to be implemented. Recent changes to the respective roles and responsibilities of the ACE development contractor and CBP’s Modernization Office are not reflected in the program and expenditure plans. ACE is to support eight major CBP business areas. 1. Release Processing: Processing of cargo for import or export; tracking of conveyances, cargo and crew; and processing of in-bond, warehouse, Foreign Trade Zone, and special import and export entries. 2. Entry Processing: Liquidation and closeout of entries and entry summaries related to imports, and processing of protests and decisions. 3. Finance: Recording of revenue, performance of fund accounting, and maintenance of the general ledger. 4. Account Relationships: Maintenance of trade accounts, their bonds and CBP-issued licenses, and their activity. 5. Legal and Policy: Management of import and export legal, regulatory, policies and procedures, and rulings issues. 6. Enforcement: Enforcement of laws, regulations, policies and procedures, and rulings governing the import and export of cargo, conveyances, and crew. import and export transactions, for use in making admissibility and release decisions. 8. Risk: Decisionmaking about admissibility and compliance of cargo using risk- based mitigation, selectivity, and targeting. The ACE technical architecture is to consist of layers or tiers of computer technology: The Client Tier includes user workstations and external system interfaces. The Presentation Tier provides the mechanisms for the user workstations and external systems to access ACE. The Integration Services Tier provides the middleware for integrating and routing information between ACE software applications and legacy systems. The Applications Tier includes software applications comprising commercial products (e.g., SAP1) and custom-developed software that provide the functionality supporting CBP business processes. The Data Tier provides the data management and warehousing services for ACE, including database backup, restore, recovery, and space management. Security and data privacy are to be embedded in all five layers. SAP is a commercial enterprise resource planning software product that has multiple modules, each performing separate but integrated business functions. ACE will use SAP as the primary commercial, off-the-shelf product supporting its business processes and functions. CBP’s Modernization Office is also using SAP as part of a joint project with its Office of Finance to support financial management, procurement, property management, cost accounting, and general ledger processes. Background Summary of ACE Releases The functionality associated with, status of, and plans for the 11 ACE releases are as follows. Release 1 (ACE Foundation): Provide IT infrastructure—computer hardware and system software—to support subsequent system releases. This release was deployed in October 2003 and is operating. Release 2 (Account Creation): Give initial group of CBP national account managers1 and importers access to account information, such as trade activity. This release was deployed in October 2003 and is operating. Release 3 (Periodic Payment): Provide additional account managers and importers, as well as brokers and carriers,2 access to account information; provide initial financial transaction processing and CBP revenue collection capability, allowing importers and their brokers to make monthly payments of duties and fees. CBP national account managers work with the largest importers. Brokers obtain licenses from CBP to conduct business on behalf of the importers by filling out paperwork and obtaining a bond; carriers are individuals or organizations engaged in transporting goods for hire. Background Summary of ACE Releases This release was deployed in July 2004 and is operating. As a result, CBP reports that importers can now obtain a national view of their transactions on a monthly statement and can pay duties and fees on a monthly basis for the first time since CBP and its predecessor organizations were established in 1789. Additionally, according to CBP, Release 3 provides a national view of trade activity, thus greatly enhancing its ability to accomplish its mission of providing border security while facilitating legitimate trade and travel. CBP also reports that as of December 6, 2004, it had processed 27,777 entries and collected over $126.5 million using Release 3. Release 4 (e-Manifest: Trucks): Provide truck manifest1 processing and interfacing to legacy enforcement systems and databases. This release is under development and scheduled for deployment beginning in February 2005. Screening S1 (Screening Foundation): Establish the foundation for screening and targeting cargo and conveyances by centralizing criteria and results into a single standard database; allow users to define and maintain data sources and business rules. This release is scheduled for deployment beginning in September 2005. Manifests are lists of passengers or invoices of cargo for a vehicle, such as a truck, ship, or plane. Background Summary of ACE Releases Screening S2 (Targeting Foundation): Establish the foundation for advanced targeting capabilities by enabling CBP’s National Targeting Center to search multiple databases for relevant facts and actionable intelligence. This release is scheduled for deployment beginning in February 2006. Release 5 (Account Revenue and Secure Trade Data): Leverage SAP technologies to enhance and expand accounts management, financial management, and postrelease functionality, as well as provide the initial multimodal manifest1 capability. This release is scheduled for deployment beginning in November 2006. Screening S3 (Advanced Targeting): Provide enhanced screening for reconciliation, intermodal manifest, Food and Drug Administration data, and in- bond, warehouse, and Foreign Trade Zone authorized movements; integrate additional data sources into targeting capability; provide additional analytical tools for screening and targeting data. This release is scheduled for deployment beginning in February 2007. The multimodal manifest involves the processing and tracking of cargo as it transfers between different modes of transportation, such as cargo that arrives by ship, is transferred to a truck, and then is loaded onto an airplane. Background Summary of ACE Releases Screening S4 (Full Screening and Targeting): Provide screening and targeting functionality supporting all modes of transportation and all transactions within the cargo management lifecycle, including enhanced screening and targeting capability with additional technologies. This release is scheduled for deployment beginning in February 2009. Release 6 (e-Manifest: All Modes and Cargo Security): Provide enhanced postrelease functionality by adding full entry processing; enable full tracking of cargo, conveyance, and equipment; enhance the multimodal manifest to include shipments transferring between transportation modes. This release is scheduled for deployment beginning in February 2009. Release 7 (Exports and Cargo Control): Implement the remaining ACE functionality, including Foreign Trade Zone warehouse; export, seized asset and case tracking system; import activity summary statement; and mail, pipeline, hand carry, drawback, protest, and document management. This release is scheduled for deployment beginning in May 2010. The graphic on the following slide illustrates the planned schedule for ACE. ACE Satisfaction of Modernization Act Requirements ACE is intended to support CBP satisfaction of the provisions of Title VI of the North American Free Trade Agreement, commonly known as the Modernization Act. Subtitle B of the Modernization Act contains the various automation provisions that were intended to enable the government to modernize international trade processes and permit CBP to adopt an informed compliance approach with industry. The following table illustrates how each ACE release is to fulfill the requirements of Subtitle B. Initial program and project management; continued by task 009. Initial enterprise architecture and system engineering; continued by task 010. Initial requirements development and program planning effort; continued by tasks for specific increments/releases. Design, development, testing, and deployment of Releases 1 and 2 (initially intended to build Increment 1, which was subsequently divided into four releases) Development of Release 5 project plan, documentation of ACE business processes, and development of an ACE implementation strategy. Enterprise process improvement integration. Assistance for participating government agencies to define requirements for an integrated ACE/ITDS system. Design, development, testing, and deployment of Releases 3 and 4. Follow-on to task 001 to continue program and project management activities. Follow-on to task 002 to continue enterprise architecture and system engineering activities; continued by task 017. Acquisition and setup of the necessary infrastructure and facilities for the contractor to design, develop, and test releases. Establishment of the infrastructure to operate and maintain releases. Conversion of scripts for interfacing desktop applications (MS Word and Excel) and mainframe computer applications. Development, demonstration, and delivery of a prototype to provide CBP insight into whether knowledge-based risk management should be used in ACE. Development and demonstration of technology prototypes to provide CBP insight into whether the technologies should be used in ACE. Program management and support to organizational change management through activities such as impact assessments, end user training, communication, and outreach. Coordination of program activities and alignment of enterprise objectives and technical plans through architecture and engineering activities. Application of the CBP Enterprise Life Cycle Methodology to integrate multiple projects and other ongoing Customs operations into CBPMO. Follow-on to task 012 includes establishment, integration, configuration, and maintenance of the infrastructure to support Releases 2, 3, and 4. Design, develop, test, and deploy the Screening Foundation (S1) release. Definition of requirements for the Targeting Foundation (S2) release, and initial project authorization and definition for Release 5. Background Chronology of Six ACE Expenditure Plans Since March 2001, six ACE expenditure plans have been submitted.1 Collectively, the six plans have identified a total of $1,401.5 million in funding. On March 26, 2001, CBP submitted to its appropriations committees the first expenditure plan seeking $45 million for the modernization contract to sustain CBPMO operations, including contractor support. The appropriations committees subsequently approved the use of $45 million, bringing the total ACE funding to $50 million. On February 1, 2002, the second expenditure plan sought $206.9 million to sustain CBPMO operations; define, design, develop, and deploy Increment 1, Release 1 (now Releases 1 and 2); and identify requirements for Increment 2 (now part of Releases 5, 6, and 7 and Screenings 1 and 2). The appropriations committees subsequently approved the use of $188.6 million, bringing total ACE funding to $238.6 million. In March 2001, appropriations committees approved the use of $5 million in stopgap funding to fund program management office operations. Background Chronology of Six ACE Expenditure Plans On May 24, 2002, the third expenditure plan sought $190.2 million to define, design, develop, and implement Increment 1, Release 2 (now Releases 3 and 4). The appropriations committees subsequently approved the use of $190.2 million, bringing the total ACE funding to $428.8 million. On November 22, 2002, the fourth expenditure plan sought $314 million to operate and maintain Increment 1 (now Releases 1, 2, 3, and 4); to design and develop Increment 2, Release 1 (now part of Releases 5, 6, and 7 and Screening 1); and to define requirements and plan Increment 3 (now part of Releases 5, 6, and 7 and Screenings 2, 3, and 4). The appropriations committees subsequently approved the use of $314 million, bringing total ACE funding to $742.8 million. Background Chronology of Six ACE Expenditure Plans On January 21, 2004, the fifth expenditure plan sought $318.7 million to implement ACE infrastructure; to support, operate, and maintain ACE; and to define and design Release 6 (now part of Releases 5, 6, and 7) and Selectivity 2 (now Screenings 2 and 3). The appropriations committees subsequently approved the use of $316.8 million, bringing total ACE funding to $1,059.6 million. On November 8, 2004, CBP submitted its sixth expenditure plan, seeking $321.7 million for detailed design and development of Release 5 and Screening 2, definition of Screening 3, Foundation Program Management, Foundation Architecture and Engineering, and ACE Operations and Maintenance. Objective 1 Results Legislative Conditions DHS and OMB satisfied or partially satisfied each of its legislative conditions; GAO satisfied its legislative condition. Condition 1. The plan, in conjunction with related program documentation and program officials’ statements, satisfied the capital planning and investment control review requirements established by OMB, including Circular A-11, part 7, which establishes policy for planning, budgeting, acquisition, and management of federal capital assets. The table that follows provides examples of the results of our analysis. Provide justification and describe acquisition strategy. The plan provides a high-level justification for ACE. Supporting documentation describes the acquisition strategy for ACE releases, including Release 5 and Screening 2 activities that are identified in the fiscal year 2005 expenditure plan. Summarize life cycle costs and cost/benefit analysis, including the return on investment. CBPMO issued a cost/benefit analysis for ACE on September 16, 2004. This analysis includes a life cycle cost estimate of $3.1 billion and a benefit cost ratio of 2.7. Provide performance goals and measures. The plan and supporting documentation describe some goals and measures. For example, CBPMO has established goals for time and labor savings expected to result from using the early ACE releases, and it has begun or plans to measure results relative to these goals and measures. It has defined measures and is collecting data for other goals, such as measures for determining its progress toward defining the complete set of ACE functional requirements. Examples of A-11 conditions Results of our analysis Address security and privacy. The security of Release 3 was certified on May 28, 2004, and accredited on June 9, 2004. Release 4 was certified on November 23, 2004, and accredited on December 2, 2004. CBP plans to certify and accredit future releases. CBPMO reports that it is currently preparing a privacy impact assessment for ACE. Address Section 508 compliance. CBPMO deployed Release 3 and plans to deploy Release 4 without Section 508 compliance because the requirement was overlooked and not built into either release. CBPMO has finalized and begun implementing a strategy that is expected to result in full Section 508 compliance. For example, CBPMO has defined a set of Section 508 requirements to be used in developing later ACE releases. Condition 2. The plan, including related program documentation and program officials’ statements, partially satisfied this condition by providing for future compliance with DHS’s enterprise architecture (EA). DHS released version 1.0 of the architecture in September 2003.1 We reviewed the initial version of the architecture and found that it was missing, either partially or completely, all the key elements expected in a well-defined architecture, such as a description of business processes, information flows among these processes, and security rules associated with these information flows.2 Since we reviewed version 1.0, DHS has drafted version 2.0 of its EA. We have not reviewed this draft. Department of Homeland Security Enterprise Architecture Compendium Version 1.0 and Transitional Strategy. GAO, Homeland Security: Efforts Under Way to Develop Enterprise Architecture, but Much Work Remains, GAO-04-777 (Washington, D.C.: Aug. 6, 2004). The Center of Excellence supports the Enterprise Architecture Board in reviewing component documentation. The purpose of the Board is to ensure that investments are aligned with the DHS EA. Objective 1 Results Legislative Conditions In August 2004, the Center of Excellence approved CBPMO’s analysis intended to demonstrate ACE’s architectural alignment, and the Enterprise Architecture Board subsequently concurred with the center’s approval. However, DHS has not yet provided us with sufficient documentation to allow us to understand DHS’s architecture compliance methodology and criteria (e.g., definition of alignment and compliance) or with verifiable analysis justifying the approval. Objective 1 Results Legislative Conditions Condition 3. The plan, in conjunction with related program documentation, partially satisfied the condition of compliance with the acquisition rules, requirements, guidelines, and systems acquisition management practices of the federal government. The Software Acquisition Capability Maturity Model (SA-CMM®), developed by Carnegie Mellon University’s Software Engineering Institute (SEI), is consistent with the acquisition guidelines and systems acquisition management practices of the federal government, and it provides a management framework that defines processes for acquisition planning, solicitation, requirements development and management, project management, contract tracking and oversight, and evaluation. In November 2003, SEI assessed ACE acquisition management against the SA- CMM and assigned a level 2 rating, indicating that CBPMO has instituted basic acquisition management processes and controls in the following areas: acquisition planning, solicitation, requirements development and management, project management, contract tracking and oversight, and evaluation. Objective 1 Results Legislative Conditions In June 2003, the Department of the Treasury’s Office of Inspector General (OIG) issued a report on the ACE program’s contract, concluding that the former Customs Service, now CBP, did not fully comply with Federal Acquisition Regulation requirements in the solicitation and award of its contract because the ACE contract is a multiyear contract and not an indefinite-delivery/indefinite-quantity (IDIQ) contract. Further, the Treasury OIG found that the ACE contract type, which it determined to be a multiyear contract, is not compatible with the program’s stated needs for a contract that can be extended to a total of 15 years, because multiyear contracts are limited to 5 years. Additionally, the Treasury OIG found that Customs combined multiyear contracting with IDIQ contracting practices. For example, it plans to use contract options to extend the initial 5-year performance period. CBP disagrees with the Treasury OIG conclusion. To resolve the disagreement, DHS asked GAO to render a formal decision. We are currently reviewing the matter. Objective 1 Results Legislative Conditions Condition 4. DHS and OMB satisfied the condition that the plan be reviewed and approved by the DHS IRB, the Secretary of Homeland Security, and OMB. On August 18, 2004, the DHS IRB reviewed the ACE program, including ACE fiscal year 2005 cost, schedule, and performance plans. The DHS Deputy Secretary, who chairs the IRB, delegated further review of the fiscal year 2005 efforts, including review and approval of the fiscal year 2005 ACE expenditure plan, to the Under Secretary for Management, with support from the Chief Financial Officer, Chief Information Officer, and Chief Procurement Officer, all of whom are IRB members. The Under Secretary for Management approved the expenditure plan on behalf of the Secretary of Homeland Security on November 8, 2004. OMB approved the plan on October 15, 2004. Condition 5. GAO satisfied the condition that it review the plan. Our review was completed on December 17, 2004. For these models, see SEI’s Checklists and Criteria for Evaluating the Cost and Schedule Estimating Capabilities of Software Organizations and A Manager’s Checklist for Validating Software Cost and Schedule Estimates. With respect to the fiscal year 2005 expenditure plan. Objective 2 Results Open Recommendations Open recommendation 3: Immediately develop and implement a human capital management strategy that provides both near- and long-term solutions to program office human capital capacity limitations, and report quarterly to the appropriations committees on the progress of efforts to do so. According to the expenditure plan, CBPMO has since developed a modernization staffing plan that identifies the positions and staff it needs to effectively manage ACE. However, CBPMO did not provide this plan to us because it was not yet approved. Moreover, program officials told us that the staffing plan is no longer operative because it was developed before December 2004, when a modernization office reorganization was implemented. As part of this reorganization, CBP transferred government and contractor personnel who have responsibility for the Automated Commercial System,1 the Automated Targeting System,2 and ACE training from non-CBPMO organizational units. This change is expected to eliminate redundant ACE-related program management efforts. The Automated Commercial System is CBP’s system for tracking, controlling, and processing imports to the United States. The Automated Targeting System is CBP’s system for identifying import shipments that warrant further attention. Objective 2 Results Open Recommendations Following our recommendation, CBPMO provided reports dated March 31, 2004, and June 30, 2004, to the appropriations committees on its human capital activities, including development of the previously mentioned staffing plan and related analysis to fully define CBPMO positions. Additionally, it has reported on efforts to ensure that all modernization office staff members complete a program management training program. Objective 2 Results Open Recommendations Open Recommendation 4: Have future ACE expenditure plans specifically address any proposals or plans, whether tentative or approved, for extending and using ACE infrastructure to support other homeland security applications, including any impact on ACE of such proposals and plans. The ACE Program Plan states that ACE provides functions that are directly related to the “passenger business process” underlying the U.S. Visitor and Immigrant Status Indicator Technology (US-VISIT) program,1 and integration of certain ACE and US-VISIT components is anticipated. In recognition of this relationship, the expenditure plan states that CBPMO and US-VISIT are working together to identify lessons learned, best practices, and opportunities for collaboration. US-VISIT is a governmentwide program to collect, maintain, and share information on foreign nationals for enhancing national security and facilitating legitimate trade and travel, while adhering to U.S. privacy laws and policies. Objective 2 Results Open Recommendations Inventory, which includes identifying connections between legacy systems and establishing a technical requirements and architecture team to review, among other things, system interfaces, data formats, and system architectures; and People, Processes, and Technology, which includes establishing teams to review deployment schedules and establishing a team and process to review and normalize business requirements. Objective 2 Results Open Recommendations In September 2004, the teams met to develop team charters, identify specific collaboration opportunities, and develop timelines and next steps. In October 2004, CBPMO and US-VISIT program officials were briefed on the progress and activities of the collaboration teams. Objective 2 Results Open Recommendations Open recommendation 5: Establish an IV&V function to assist CBP in overseeing contractor efforts, such as testing, and ensure the independence of the IV&V agent. According to ACE officials, they have selected an IV&V contractor that has had no prior involvement in the modernization program to ensure independence. These officials stated that the IV&V contractor will be responsible for reviewing ACE products and management processes, and will report directly to the CBP CIO. Award of this contract is to occur on December 30, 2004. Open recommendation 6: Define metrics, and collect and use associated measurements, for determining whether prior and future program management improvements are successful. CBPMO has implemented a metrics program that generally focuses on measuring eCP’s performance through the use of earned value management (EVM), deliverable timeliness and quality metrics, and risk and issue disposition reporting. Additionally, CBPMO is planning to broaden its program to encompass metrics and measures for determining progress toward achieving desired business results and acquisition process maturity. The plan for expanding the metrics program is scheduled for approval in early 2005. One part of CBPMO’s metrics program that it has implemented relates to EVM for its contract with eCP. EVM is a widely accepted best practice for measuring contractor progress toward meeting deliverables by comparing the value of work accomplished during a given period with that of the work expected in that period. Differences from expectations are measured in the form of both cost and schedule variances. Cost variances compare the earned value of the completed work with the actual cost of the work performed. For example, if a contractor completed $5 million worth of work and the work actually cost $6.7 million, there would be a –$1.7 million cost variance. Positive cost variances indicate that activities are costing less, while negative variances indicate activities are costing more. Schedule variances, like cost variances, are measured in dollars, but they compare the earned value of the work completed to the value of work that was expected to be completed. For example, if a contractor completed $5 million worth of work at the end of the month, but was budgeted to complete $10 million worth of work, there would be a –$5 million schedule variance. Positive schedule variances show that activities are being completed sooner than planned. Negative variances show activities are taking longer than planned. In accordance with EVM principles, eCP reports on its financial performance monthly. These reports provide detailed information on cost and schedule performance on work segments in each task order. Cost and schedule variances that exceed a certain threshold are further examined to determine the root cause of the variance, the impact on the program, and mitigation strategies. Objective 2 Results Open Recommendations Open recommendation 7: Reconsider the ACE acquisition schedule and cost estimates in light of early release problems, including these early releases’ cascading effects on future releases and their relatively small size compared to later releases, and in light of the need to avoid the past levels of concurrency among activities within and between releases. As we previously reported, the cost estimate for Releases 3 and 4 had grown to $185.7 million, which was about $36.2 million over the contract baseline, and the chances of further overruns were likely.1 Subsequently, the Release 3 and 4 cost overrun grew to an estimated $46 million, resulting in CBPMO and eCP establishing a new cost baseline for Releases 3 and 4 of $196 million. eCP began reporting performance against this new baseline in April 2004. Further, in July 2004, CBPMO and eCP changed the associated contract task order baseline completion date from September 15, 2004, to May 30, 2005, revised the associated interim task order milestones, and began reporting schedule performance relative to the new baselines. GAO, Information Technology: Early Releases of Customs Trade System Operating, but Pattern of Cost and Schedule Problems Needs to Be Addressed, GAO-04-719 (Washington, D.C.: May 14, 2004). Objective 2 Results Open Recommendations In July 2004, eCP also rebaselined the ACE program, producing a new version of the ACE Program Plan. The new baseline extends delivery of the last ACE release from fiscal year 2007 to fiscal year 2010 and adds a new screening and targeting release. The new program plan also provides a new ACE life-cycle cost estimate of $3.1 billion,1 which is a $1 billion increase over the previous life-cycle cost estimate. According to the expenditure plan, the new schedule reflects less concurrency between releases. The following figure compares previous and current schedules for ACE releases and shows a reduction in the level of concurrency between releases. CBP’s ACE life-cycle cost estimate adjusted for risk is about $3.3 billion. Open recommendation 8: Report quarterly to the House and Senate Appropriations Committees on efforts to address open GAO recommendations. CBPMO submitted reports to the Committees on its efforts to address open GAO recommendations for the quarters ending March 31, 2004, and June 30, 2004. CBPMO plans to submit a report for the quarter ending September 30, 2004, after it is approved by DHS and OMB. JavaTM Archive (JAR) files bundle multiple class files and auxiliary resources associated with applets and applications into a single archive file. Verify that related system, subsystem, or module components are capable of integrating and interfacing with each other. Test Readiness Review (TRR) System acceptance test (SAT) Verify that the developed system, subsystem, or module operates in accordance with requirements. Production Readiness Review (PRR) User acceptance test (UAT) Verify that the functional scope of the release meets the business functions for the users. Operational Readiness Review (ORR) Defect prevents or precludes the performance of an operational or mission- essential capability, jeopardizes safety or security, or causes the system, application, process, or function to fail to respond or to end abnormally. Severe (Severity 2) Defect prevents or precludes system from working as specified and/or produces an error that degrades or impacts the system or user functionality. Moderate (Severity 3) Defect prevents or precludes system from working as specified and/or produces an error that degrades or impacts the system or user functionality. An acceptable (reasonable and effective) work-around is in place that rectifies the defect until a permanent fix can be made. Minor (Severity 4) Defect is inconsequential, cosmetic, or inconvenient but does not prevent users from using the system to accomplish their tasks. GAO, Information Technology: Homeland Security Needs to Improve Entry Exit System Expenditure Planning, GAO-03-563 (Washington, D.C.: June 9, 2003). that is intended to provide additional account management functionality. The fiscal year 2005 plan, however, did not address progress against these commitments. For example, the plan did not describe the status of infrastructure acquisition, nor did it discuss the expenditure of the $106.6 million requested for this purpose. While the plan did discuss the status of the initial ACE releases, it did not describe progress toward defining and designing the functionality that was to be in the former Release 6. Also, the fiscal year 2005 expenditure plan included a schedule for developing ACE releases, but neither reported progress relative to the schedule presented in the fiscal year 2004 plan nor explained how the individual releases and their respective schedules were affected by the rebaselining that occurred after the fiscal year 2004 plan was submitted. Further, while the fiscal year 2005 expenditure plan contained high-level descriptions of the functionality provided by Releases 1 and 2, it did not describe progress toward achieving the benefits they are expected to provide. Without such information, meaningful congressional oversight of CBP progress and accountability is impaired. GAO, Information Technology: DOD’s Acquisition Policies and Guidance Need to Incorporate Additional Best Practices and Controls, GAO-04-722 (Washington, D.C.: July 2004). Establish and communicate targets for ACE usage to encourage users to use ACE rather than ACS. If ACS remains available to ACE users, they may continue to use the legacy system, and as a result the full benefits of ACE will not be realized. Before training, make users aware of the major differences between ACS and ACE. If ACE users do not understand the differences between the legacy systems and ACE, then the users will not understand how best to use ACE, which may result in resistance to the new system and processes. Discuss the future needs of CBP to establish new roles and responsibilities within the Office of Information and Technology (OIT). If future roles of the OIT are not established, then OIT may not be prepared to provide technical support when ACE is transferred from eCP to OIT. Send staff to visit ports to build critical knowledge regarding organizational change objectives. If staff do not have adequate access to representatives of occupational groups at each port, then communications, training, and deployment efforts cannot be customized to each group's needs. This may delay or disrupt ACE adoption. GAO, Tax Systems Modernization: Results of Review of IRS’ Initial Expenditure Plan, GAO/AIMD/GGD-99-206 (Washington, D.C.: June 1999). coverage of all program commitment areas, including key expected or estimated system (1) capabilities, use, and quality; (2) benefits and mission value; (3) costs; and (4) milestones and schedules; currency, relevance, and completeness of all such commitments made to the Congress in expenditure plans; reliability of data relevant to measuring progress against commitments; reporting in future expenditure plans of progress against commitments contained in prior expenditure plans; use of criteria for exiting key readiness milestones that adequately consider indicators of system maturity, such as severity of open defects; and clear and unambiguous delineation of the respective roles and responsibilities of the government and the prime contractor. SEI’s institutional estimating guidelines are defined in Checklists and Criteria for Evaluating the Cost and Schedule Estimating Capabilities of Software Organizations, and SEI’s project-specific estimating guidelines are defined in A Manager’s Checklist for Validating Software Cost and Schedule Estimates. Institute of Electrical and Electronics Engineers (IEEE) Standard for Software Verification and Validation, IEEE Std 1012- 1998 (New York: Mar. 9, 1998). Attachment 1 Scope and Methodology CBP’s progress toward increasing the number of ACE user accounts, against ACE’s quality, using eCP defect data and testing results for Releases 3 and 4; cost and schedule data and program commitments from program management documentation. For DHS-, CBP-, and contractor-provided data that our reporting commitments did not permit us to substantiate, we have made appropriate attribution indicating the data’s source. We conducted our work at CBP headquarters and contractor facilities in the Washington, D.C., metropolitan area from April 2004 through December 2004, in accordance with generally accepted government auditing standards.

Summary: The Department of Homeland Security (DHS) is conducting a multiyear, multibillion-dollar acquisition of a new trade processing system, planned to support the movement of legitimate imports and exports and strengthen border security. By congressional mandate, plans for expenditure of appropriated funds on this system, the Automated Commercial Environment (ACE), must meet certain conditions, including GAO review. This study addresses whether the fiscal year 2005 plan satisfies these conditions, describes the status of DHS's efforts to implement prior GAO recommendations for improving ACE management, and provides observations about the plan and DHS's management of the program. The fiscal year 2005 ACE expenditure plan, including related program documentation and program officials' statements, largely satisfies the legislative conditions imposed by the Congress. In addition, some of the recommendations that GAO has previously made to strengthen ACE management have been addressed, and DHS has committed to addressing those that remain. However, much remains to be done before these recommendations are fully implemented. For example, progress has been slow on implementing the recommendation that the department proactively manage the dependencies between ACE and related DHS border security programs. Delays in managing the relationships among such programs will increase the chances that later system rework will be needed to allow the programs to interoperate. Among GAO's observations about the ACE program and its management are several regarding DHS's approach to addressing previously identified cost and schedule overruns. DHS has taken actions intended to address these overruns (such as revising its baselines for cost and schedule, as GAO previously recommended); however, it is unlikely that these actions will prevent future overruns, because DHS has relaxed system quality standards, meaning that milestones are being passed despite material system defects. Correcting such defects will require the program to use resources (e.g., people and test environments) at the expense of later system releases. Until the ACE program is held accountable not only for cost and schedule but also for system capabilities and benefits, the program is likely to continue to fall short of expectations. Finally, the usefulness of the fiscal year 2005 expenditure plan for congressional oversight is limited. For example, it does not adequately describe progress against commitments (e.g., ACE capabilities, schedule, cost, and benefits) made in previous plans, which makes it difficult to make well-informed judgments on the program's overall progress. Also, in light of recent program changes, GAO questions the expenditure plan's usefulness to the Congress as an accountability mechanism. The expenditure plan is based largely on the ACE program plan of July 8, 2004. However, recent program developments have altered some key bases of the ACE program plan and thus the current expenditure plan. In particular, the expenditure plan does not reflect additional program releases that are now planned or recent changes to the roles and responsibilities of the ACE development contractor and the program office. Without complete information and an up-to-date plan, meaningful congressional oversight of program progress and accountability is impaired.