In November 1994, the Office of the Director of Defense Procurement initiated the SPS program to acquire and deploy a single automated system to perform all contract-management-related functions for all DOD organizations. At that time, life-cycle costs were estimated to be about $3 billion over a 10-year period. From 1994 to 1996, the department defined SPS requirements and solicited commercially available vendor products for satisfying these requirements. Subsequently, in April 1997, the department awarded a contract to American Management Systems (AMS), Incorporated, to (1) use AMS’s commercially available contract management system as the foundation for SPS, (2) modify this commercial product as necessary to meet DOD requirements, and (3) perform related services. The department also directed the contractor to deliver functionality for the system in four incremental releases. The department later increased the number of releases across which this functionality would be delivered to seven, reduced the size of the increments, and allowed certain more critical functionality to be delivered sooner (see table 1 for proposed SPS functionality by increment). Since our report of July 2001, DOD has revised its plans. According to the SPS program manager, current plans no longer include increments 6 and 7 or releases 5.0 and 5.1. Instead, release 4.2 (increment 5) will include at least three, but not more than seven, subreleases. At this time, only the first of the potentially seven 4.2 subreleases is under contract. This subrelease is scheduled for delivery in April 2002, with deployment to the Army and the Defense Logistics Agency scheduled for June 2002. Based on the original delivery date, release 4.2 is about one year overdue. The department reports that it has yet to define the requirements to be included within the remaining 4.2 subreleases, and has not executed any contract task orders for these subreleases. According to SPS officials, they will decide later this year whether to invest in these additional releases. As of December 2001, the department reported that it had deployed four SPS releases to over 777 locations. The Director of Defense Procurement (DDP) has responsibility for the SPS program, and the CIO is the milestone decision authority for SPS because the program is classified as a major Defense acquisition. Our July 2001 report detailed program problems and investment management weaknesses. To address these weaknesses, we recommended, among other things, that the department report on the lessons to be learned from its SPS experience for the benefit of future system acquisitions. Similarly, other reviews of the program commissioned by the department in the wake of our review raised similar concerns and identified other problems and management weaknesses. The findings from our report are summarized below in two major categories: lack of economic justification for the program and inability to meet program commitments. We also summarize the findings of the other studies. The Clinger-Cohen Act of 1996, OMB guidance, DOD policy, and practices of leading organizations provide an effective framework for managing information technology investments, not just when a program is initiated, but continuously throughout the life of the program. Together, they provide for (1) economically justifying proposed projects on the basis of reliable analyses of expected life-cycle costs, benefits, and risks; and (2) using these analyses throughout a project’s life-cycle as the basis for investment selection, control, and evaluation decisionmaking, and doing so for large projects (to the maximum extent practical) by dividing them into a series of smaller, incremental subprojects or releases and individually justifying investment in each separate increment on the basis of costs, benefits, and risks. The department had not met these investment management tenets for SPS. First, the latest economic analysis for the program—dated January 2000— was not based on reliable estimates because most of the cost estimates in the 2000 economic analysis were estimates carried forward from the April 1997 analysis (adjusted for inflation). Only the cost estimates being funded and managed by the SPS program office, which were 13 percent of the total estimated life-cycle cost in the analysis, were updated in 2000 to reflect more current contract estimates and actual expenditures/ obligations for fiscal years 1995 through 1999. Moreover, the military services, which share funding responsibility with the SPS program office for implementing the program, questioned the reliability of these cost estimates. However, this uncertainty was not reflected in the economic analysis using any type of sensitivity analysis. A sensitivity analysis would have disclosed for decisionmakers the investment risk being assumed by relying on the estimates presented in the economic analysis. Moreover, the latest economic analysis (January 2000) was outdated because it did not reflect the program’s current status and known problems and risks. For instance, this analysis was based on a program scope and associated costs and benefits that anticipated four software releases. However, as mentioned previously, the program now consists of five releases, and subreleases within releases, in order to accommodate changes in SPS requirements. Estimates of the full costs, benefits, and risks relating to this additional release and its subreleases were not part of the 2000 economic analysis. Also, this analysis did not fully recognize actual and expected delays in meeting SPS’s full operational capability milestone, which had been slipped by 3½ years and DOD officials say that further delays are currently expected. Such delays not only increase the system acquisition costs but also postpone, and thus reduce, accrual of system benefits. Further, several DOD components are now questioning whether they will even deploy the software, which would further reduce SPS’s cost effectiveness calculations in the 2000 economic analysis. Second, the department had not used these analyses as the basis for deciding whether to continue to invest in the program. The latest economic analysis showed that SPS was not a cost-beneficial investment because the estimated benefits to be realized did not exceed estimated program costs. In fact, the 2000 analysis showed estimated costs of $3.7 billion and estimated benefits of $1.4 billion, which was a recovery of only 37 percent of costs. According to the former SPS program manager, this analysis was not used to manage the program and there was no DOD requirement for updating an economic analysis when changes to the program occurred. Third, DOD had not made its investment decisions incrementally as required by the Clinger-Cohen Act and OMB guidance. That is, although the department is planning to acquire and implement SPS as a series of five increments, it has not made decisions about whether to invest in each release on the basis of the release’s expected return on investment, as well as whether prior releases were actually achieving return-on-investment expectations. In fact, for the four increments that have been deployed, the department had not validated whether the increments were providing promised benefits and was not accounting for the costs associated with each increment so that it could even determine actual return on investment. Instead, the department had treated investment in this program as one, monolithic investment decision, justified by a single, “all-or-nothing” economic analysis. Our work has shown that it is difficult to estimate, with any degree of accuracy, cost and schedule estimates for many increments to be delivered over many years because later increments are not well understood or defined. Also, these estimates are subject to change based on actual program experiences and changing requirements. This “all-or- nothing” approach to investing in large system acquisitions, like SPS, has repeatedly proven to be ineffective across the federal government, resulting in huge sums being invested in systems that do not provide commensurate benefits. Measuring progress against program commitments is closely aligned with economically justifying information-technology investments, and is equally important to ensuring effective investment management. The Clinger- Cohen Act, OMB guidance, DOD policy, and practices of leading organizations provide for making and using such measurements as part of informed investment decisionmaking. DOD had not met key commitments and was uncertain whether it was meeting other commitments because it was not measuring them. (See table 2 for a summary of the department’s progress against commitments.) two analyses, such as the number and dollar value of estimated benefits, and the information gathered did not map to the 22 benefit types listed in the 1997 economic analysis. Instead, the study collected subjective judgments (perceptions) that were not based on predefined performance metrics for SPS capabilities and impacts. Thus, the department was not measuring SPS against its promised benefits. The former program manager told us that knowing whether SPS was producing value and meeting commitments was not the program office’s objective because there was no departmental requirement to do so. Rather, the objective was simply to acquire and deploy the system. Similarly, CIO officials told us that the department was not validating whether deployed releases of SPS were producing benefits because there was no DOD requirement to do so and no metrics had been defined for such validation. However, the Clinger-Cohen Act of 1996 and OMB guidance emphasize the need to have investment management processes and information to help ensure that information-technology projects are being implemented at acceptable costs and within reasonable and expected time frames and that they are contributing to tangible, observable improvements in mission performance (i.e., that projects are meeting the cost, schedule, and performance commitments upon which their approval was justified). For programs such as SPS, DOD required this cost, schedule, and performance information to be reported quarterly to ensure that programs did not deviate significantly from expectations. In effect, these requirements and guidance recognize that one cannot manage what one cannot measure. Shortly after receiving our draft report for comment, the department initiated several studies to determine the program’s current status, assess program risks, and identify actions to improve the program. These studies focused on such areas as program costs and benefits, planned commitments, requirements management, program office structure, and systems acceptance testing.