Customs began ACE in 1994, and its early estimate of the cost and time to develop the system was $150 million over 10 years. At this time, Customs also decided to first develop a prototype of ACE, referred to as NCAP (National Customs Automation Program prototype), and then to complete the system. In May 1997, we testified that Customs’ original schedule for completing the prototype was January 1997, and that Customs did not have a schedule for completing ACE. At that time, Customs agreed to develop a comprehensive project plan for ACE. In November 1997, Customs estimated that the system would cost $1.05 billion to develop, operate, and maintain throughout its life cycle. Customs plans to develop and deploy the system in 21 increments from 1998 through 2005, the first four of which would constitute NCAP. Currently, Customs is well over 2 years behind its original NCAP schedule. Because Customs experienced problems in developing NCAP software in- house, the first NCAP release was not deployed until May 1998—16 months late. In view of the problems it experienced with the first release, Customs contracted out for the second NCAP release and deployed this release in October 1998—21 months later than originally planned. Customs’ most recent dates for deploying the final two NCAP releases (0.3 and 0.4) are March 1999 and September 1999, which are 26 and 32 months later than the original deployment estimates, respectively. According to Customs, these dates will slip farther because of funding delays. Additionally, Customs officials told us that a new ACE life cycle cost estimate is being developed, but that it was not ready to be shared with us. At the time of our review, Customs’ $1.05 billion estimate developed in 1997 was the official ACE life cycle cost estimate. However, a January 1998 ACE business plan specifies a $1.48 billion life cycle cost estimate. Customs is not building ACE within the context of an enterprise systems architecture, or “blueprint” of its agencywide future systems environment. Such an architecture is a fundamental component of any rationale and logical strategic plan for modernizing an organization’s systems environment. As such, the Clinger-Cohen Act requires agency chief information officers (CIO) to develop, maintain, and implement an information technology (IT) architecture. Also, the Office of Management and Budget (OMB) issued guidance in 1996 that requires agency IT investments to be architecturally compliant. These requirements are consistent with, and in fact based on, IT management practices of leading private and public sector organizations. Simply stated, an enterprise systems architecture specifies the system (e.g., software, hardware, communications, security, and data) characteristics that the organization’s target systems environment is to possess. Its purpose is to define, through careful analysis of the organization’s strategic business needs and operations, the future systems configuration that supports not only the strategic business vision and concept of operations, but also defines the optimal set of technical standards that should be met to produce homogeneous systems that can interoperate effectively and be maintained efficiently. Our work has shown that in the absence of an enterprise systems architecture, incompatible systems are produced that require additional time and resources to interconnect and to maintain and that suboptimize the organization’s ability to perform its mission. We first reported on Customs’ need for a systems architecture in May 1996 and testified on this subject in May 1997. In response, Customs developed and published an architecture in July and August 1997. We reviewed this architecture and reported in May 1998 that it was not effective because it was neither complete nor enforced. For example, the architecture did not 1. fully describe Customs’ business functions and their relationships, 2. define the information needs and flows among these functions, and 3. establish the technical standards, products, and services that would be characteristic of its target systems environment on the basis of these business specifications. Accordingly, we recommended that Customs complete its enterprise information systems architecture and establish compliance with the architecture as a requirement of Customs’ information technology investment management process. In response, Customs agreed to develop a complete architecture and establish a process to ensure compliance. Customs is in the process of developing the architecture and reports that it will be completed in May 1999. Also, in January 1999, Customs reported that it changed its internal procedures to provide for effective enforcement of its architecture, once it is completed. Until the architecture is completed and enforced, Customs risks spending millions of dollars to develop, acquire, and maintain information systems, including ACE, that do not effectively and efficiently support the agency’s mission needs. Effective IT investment management is predicated on answering one basic question: Is the organization doing the “right thing” by investing specified time and resources in a given project or system? The Clinger-Cohen Act and OMB guidance together provide an effective IT investment management framework for answering this question. Among other things, they set requirements for 1. identifying and analyzing alternative system solutions, 2. developing reliable estimates of the alternatives’ respective costs and benefits and investing in the most cost beneficial alternative, and 3. to the maximum extent practical, structuring major projects into a series of increments to ensure that each increment constitutes a wise investment. Customs did not satisfy any of these requirements for ACE. First, Customs did not identify and evaluate a full range of alternatives to its defined ACE solution before commencing development activities. For example, Customs did not consider how ACE would relate to another Treasury- proposed system for processing import trade data, known as the International Trade Data System (ITDS), including considering the extent to which ITDS should be used to satisfy needed import processing functionality. Initiated in 1995 as a project to develop a coordinated, governmentwide system for the collection, use, and dissemination of trade data, the ITDS project is headed by the Treasury Deputy Assistant Secretary for Regulatory, Tariff and Trade Enforcement. The system is expected to reduce the burden federal agencies place on organizations by requiring that they respond to duplicative data requests. Treasury intends for the system to serve as the single point for collecting, editing, and validating trade data as well as collecting and accounting for trade revenue. At the time of our review of ACE, these functions were also planned for ACE. Similarly, Customs did not evaluate different ACE architectural designs, such as the use of a mainframe-based versus client/server-based hardware architecture. Also, Customs did not evaluate alternative development approaches, such as acquisition versus in-house development. In short, Customs committed to and began building ACE without knowing whether it had chosen the most cost-effective alternative and approach. Second, Customs did not develop a reliable life cycle cost estimate for the approach it selected. SEI has developed a method for project managers to use to determine the reliability of project cost estimates. Using SEI’s method, we found that Customs’ $1.05 billion ACE life cycle cost estimate was not reliable, and that it did not provide a sound basis for Customs’ decision to invest in ACE. For example, in developing the cost estimate, Customs (1) did not use a cost model, (2) did not account for changes in its approach to building different ACE increments, (3) did not account for changes to ACE software and hardware architecture, and (4) did not have historical project cost data upon which to compare its ACE estimate. Moreover, the $1.05 billion cost estimate used to economically justify ACE omitted relevant costs. For instance, the costs of technology refreshment and system requirements definition were not included (see table 1). Exacerbating this problem, Customs represented its ACE cost estimate as a precise point estimate rather than explicitly disclosing to investment decisionmakers in Treasury, OMB, and Congress the estimate’s inherent uncertainty. Customs’ projections of ACE benefits were also unreliable because they were either overstated or unsupported. For example, the analysis includes $203.5 million in savings attributable to 10 years of avoided maintenance and support costs on the Automated Commercial System (ACS)—the system ACE is to replace. However, Customs would not have avoided maintenance and support costs for 10 years. At the time of Customs’ analysis, it planned to run both systems in parallel for 4 years, and thus planned to spend about $53 million on ACS maintenance and support during this period. As another example, $650 million in savings was not supported by verifiable data or analysis, and $644 million was based on assumptions that were analytically sensitive to slight changes, making this $644 million a “best case” scenario. Third, Customs is not making its investment decisions incrementally as required by the Clinger-Cohen Act and OMB. Although Customs has decided to implement ACE as a series of 21 increments, it is not justifying investing in each increment on the basis of defined costs and benefits and a positive return on investment for each increment. Further, once it has deployed an increment at a pilot site for evaluation, it is not validating the benefits that the increment actually provides, and it is not accounting for costs on each increment so that it can demonstrate that a positive return on investment was actually achieved. Instead, Customs estimated the costs and benefits for the entire system--all 21 increments, and used this as economic justification for ACE. Mr. Chairman, our work has shown that such estimates of many system increments to be delivered over many years are impossible to make accurately because later increments are not well understood or defined. Also, these estimates are subject to change in light of experiences on nearer term increments and changing business needs. By using an inaccurate, aggregated estimate that is not refined as increments are developed, Customs is committing enormous resources with no assurance that it will achieve a reasonable return on its investment. This “grand design” approach to managing large system modernization projects has repeatedly proven to be ineffective across the federal government, resulting in huge sums invested in systems that do not provide expected benefits. Failure of the grand design approach was a major impetus for the IT management reforms contained in the Clinger-Cohen Act.