Historically, DOD’s programs for acquiring major weapon systems have taken longer, cost more, and often delivered fewer quantities and other capabilities than planned. GAO has documented these problems for decades. In 1970, GAO reported that considerable cost growth had been and was continuing to occur on many current development programs. Since that report was issued, numerous changes have been made to DOD’s acquisition process and environment to try to improve acquisition outcomes. Those changes include numerous executive branch initiatives and legislative actions as well as roughly 11 revisions to DOD’s acquisition policy between 1971 and 2005. Despite these efforts, defense acquisition programs in the past 3 decades continued to routinely experience cost overruns, schedule slips, and performance shortfalls. Figure 1 illustrates the continued problem of development cost overruns. The figure depicts the combined cost overruns for large development programs (programs totaling more than $1 billion for research, development, testing and evaluation in fiscal year 2005 dollars) in each of the past 3 decades. The figure also identifies some of the major studies and improvement efforts initiated during this time frame. As the figure illustrates, efforts to improve acquisition outcomes have not been successful in curbing acquisition cost problems. Programs initiated in the 1970s exceeded DOD’s initial investment estimate by 30 percent, or $13 billion (in fiscal year 2005 dollars), and similar outcomes continued during the subsequent decades despite numerous reform efforts and policy revisions. Since the mid-1990s, we have studied the best practices of leading commercial companies. Taking into account the differences between commercial product development and weapons acquisitions, we articulated a best practices product development model that relies on increasing knowledge when developing new products, separating technology development from product development, and following an evolutionary or incremental product development approach. This knowledge-based approach requires developers to make investment decisions on the basis of specific, measurable levels of knowledge at critical junctures before investing more money and before advancing to the next phase of acquisition. An evolutionary product development process defines the individual increments on the basis of mature technologies and a feasible design that are matched with firm requirements. Each increment should be managed as a separate and distinct acquisition effort with its own cost, schedule and performance baseline. An increment that excludes one of these key elements puts an extra burden on decision makers and provides a weak foundation for making development cost and schedule estimates. The knowledge-based, evolutionary approach in our model is intended to help reduce development risks and to achieve better program outcomes on a more consistent basis. Hoping to improve acquisition outcomes, DOD leaders initiated significant revisions to the department’s acquisition policy again in October 2000, by adopting the knowledge-based, evolutionary system development approach. We reported in November 2003, that much of the revised policy agrees with GAO’s extensive body of work and that of successful commercial firms. DOD’s revised policy emphasizes the importance of and provides a good framework for capturing knowledge about critical technologies, product design, and manufacturing processes. If properly implemented and enforced this approach could help DOD’s decision makers gain the confidence they need to make significant and sound investment decisions for major weapon systems. Furthermore, the policy’s emphasis on evolutionary system development sets up a more manageable environment for achieving knowledge. We also noted that DOD’s policy strongly suggests the separation of technology development from system development, a best practice that helps reduce technological risk at the start of a program and makes cost and delivery estimates much more predictable. Figure 2 depicts in general how DOD’s revised policy adopts key aspects of the best practices model. Although DOD took significant steps in the right direction, its policy does not include controls that require program officials to meet the key criteria that we believe are necessary for ensuring that acceptable levels of knowledge are actually captured before making additional significant investments. We previously recommended that DOD design and implement necessary controls to ensure that appropriate knowledge is captured and used to make decisions about moving a program forward and investing more money at critical junctures. DOD officials acknowledged the advantages of using knowledge-based controls, but stated that they believed the policy already included enough controls to achieve effective program results. The officials agreed to monitor the acquisition process to assess the effectiveness of those controls and to determine whether additional ones are necessary. The cost and schedule outcomes being achieved by development programs initiated since DOD first issued its revised policy have not improved over those achieved by programs managed under prior versions of the policy. Of the 23 major programs we assessed, 10 have already reported estimated development cost growth greater than 30 percent or expected delays of at least 1 year in delivery of an initial operational capability to the warfighter. These programs combined represent a cost increase of $23 billion (in fiscal year 2005 dollars) and an average delay in delivery of initial capability of around 2 years. Most of the other programs were still in the early stages as of December 2005 with over half of system development remaining and had not yet reported an adequate amount of cost or schedule data to effectively analyze their progress. Table 1 contains the cost and schedule increases for the 23 programs we assessed, expressed as a percentage of each program’s development estimate. The Army’s Future Combat System is a case in point. Less than 3 years after program initiation and with $4.6 billion invested, the Army has already increased its development cost estimate $8.9 billion or 48 percent and delayed delivery of initial capability by 4 years over the original business case. Similarly, just over 1 year after initiating development of the Aerial Common Sensor aircraft, the Army has reported that severe weight and design problems discovered during development have stopped work on the program. As a result, program officials are anticipating at least a 45 percent cost increase and a delay of 2 years in delivering an initial capability to the warfighter. These two Army programs are not the only ones experiencing problems. Table 2 contains cost and schedule data for 6 of the 10 largest development programs initiated under the revised policy, including the Future Combat System and Aerial Common Sensor. As the table illustrates there are several programs experiencing large cost increases and schedule delays. A good measure of acquisition performance is return on investment as expressed in acquisition program unit cost because unit cost represents the value DOD is getting for its acquisition dollars invested in a certain program. The programs listed in table 2 will not achieve the return on investment that DOD anticipated when they began development. In the case of Joint Strike Fighter, for example, DOD initially intended to purchase 2,866 aircraft at an acquisition program unit cost of about $66 million. The Navy has reduced the number of Joint Strike Fighter aircraft it plans to buy; technology and design problems encountered during development have led to the significant cost growth. As a result, the acquisition program unit cost is now about $84 million, an increase of 27 percent. We recently reported that the risk of even greater increases is likely because flight testing has not yet started and the acquisition strategy involves substantial overlap of development and production. Similar problems have led to increases in the Future Combat System program. At program initiation, the Army anticipated that each of 15 units would cost about $5.5 billion to develop and deliver. Since that time, instability in the program’s technologies and requirements have led to significant cost increases, leading to a 54 percent increase in acquisition program unit cost, now estimated to be $8.5 billion. Regarding all 23 development programs, DOD leaders originally planned to invest a total of about $83 billion (fiscal year 2005 dollars) for system development and anticipated delivering an initial operational capability to the warfighter in 77 months on average. However, development costs have grown and delivery schedules have been delayed significantly. DOD now expects to invest over $106 billion in those same programs, an increase of over $23 billion or 28 percent. The delivery of initial capability to the warfighter is expected to take an average of 88 months or nearly 1 year longer than originally planned. Figure 3 shows changes in these business case elements for these programs in the short time since their initiation. DOD is not effectively implementing the knowledge-based process and evolutionary approach emphasized in its acquisition policy. While the policy outlines a specific knowledge-based process of concept refinement and technology development to help ensure a sound business case is developed before committing to a new development program, almost 80 percent of the programs we reviewed were permitted to bypass this process. Furthermore, the policy emphasizes the need to mature all critical technologies before starting system development and to demonstrate that the product’s design is mature before beginning system demonstration. However, nearly three-fourths of the programs reported having immature critical technologies when they received approval to start development, and at least half of the programs had not achieved design maturity before holding their design review and gaining approval to enter the system demonstration phase of development. The policy also emphasizes the use of an evolutionary product development approach, yet program officials continue to structure major acquisition programs to achieve large advances in capability within a single step development program. This strategy has historically resulted in poor cost and schedule outcomes. DOD decision makers continue to approve programs for system development that have not followed key elements of the policy’s suggested knowledge-based process. The policy requires program managers to provide senior decision makers with knowledge about key aspects of a system at critical investment points in the acquisition process. Our prior reviews have identified those critical points as the start of system development or program start (referred to as Milestone B in the DOD acquisition policy), design readiness review separating system integration and system demonstration, and production commitment (Milestone C in the DOD acquisition policy). The most important point occurs at program start, when system development begins. DOD acquisition guidance emphasizes the importance of the acquisition phases preceding program start, noting that the decisions made during those phases—concept refinement and technology development—generally define the nature of an entire acquisition program. Acquisition officials continue to begin system development without following early processes for developing executable business cases. A business case should provide demonstrated evidence that (1) the warfighter’s needs are real and necessary and that they can best be met with the chosen concept and (2) the chosen concept can be developed and produced within existing resources—including technologies, design knowledge, funding, and the time to deliver the product when it is needed. Establishing a business case calls for a realistic assessment of risks and costs; doing otherwise undermines the intent of the business case and invites failure. This process requires the user and developer to negotiate whatever trade-offs are needed to achieve a match between the user’s requirements and the developer’s resources before system development begins. The revised policy and associated guidance emphasize the importance of following a sound process of systems engineering and decision making prior to initiating a system development program. The process established in the policy consists of two phases, concept refinement and technology development, and a major decision review called Milestone A, which if rigorously followed, would provide acquisition officials with an opportunity to assess whether program officials had the knowledge needed to develop an executable business case. However, almost 80 percent of the programs we reviewed began system development without holding any prior decision review. Senior officials with the Office of the Secretary of Defense confirmed that this is a common practice among defense acquisition programs. This practice eliminates a key opportunity for decision makers to assess early product knowledge needed to establish a business case that is based on realistic cost, schedule, and performance expectations. Although program officials conduct analysis before starting a development program, they do not consistently follow a process to capture the critical knowledge needed to produce executable business cases, as evidenced by the poor outcomes current programs are experiencing. Officials with the Office of the Secretary of Defense recognized this lack of rigor and discipline in acquisition process, and in February 2004, the Under Secretary of Defense (Acquisition, Technology and Logistics) issued a department-wide policy memorandum directing acquisition officials to place greater emphasis on systems engineering when planning and managing acquisition programs. The policy requires programs to develop a systems engineering plan that describes the programs’ overall technical approach, including processes, resources, metrics, and applicable performance incentives. Although DOD’s systems engineering initiative has the potential to improve program performance, officials have found that the preliminary results are mixed. Early analysis shows that implementation is inconsistent while program officials learn to develop and implement systems-engineering plans. DOD decision makers continue to permit programs to enter system development before critical technologies are mature. Our review of technology readiness assessments and acquisition decision memorandums for our nine case study programs found that seven of the nine programs were approved to begin development even though program officials reported levels of knowledge below the criteria suggested in the policy and associated guidance, specifically in the area of technology maturity. Those seven programs are not isolated cases. As illustrated in Figure 4, 13 of the programs (nearly three-fourths) that received approval to enter system development under the revised policy did so with less than 100 percent of their critical technologies mature to the level specified by DOD. Only 2 of those programs had more than 75 percent of their technologies mature when they began (see appendix III for technology maturity data for each program). Even though acquisition policy states that technologies shall be mature before beginning system development, the practice of accepting high levels of technology risk at program start continues to be the norm and not the exception. An official with Office of the Secretary of Defense responsible for reviewing and validating program assessments of technology maturity informed us that the office generally views immature critical technologies at the beginning of development as an acceptable risk as long as program officials can show that they have a plan to mature the technologies by the time the program reaches its design readiness review, which requires additional investments to move a program from system integration into system demonstration. Therefore, risk management plans are consistently viewed as acceptable substitutes for demonstrated knowledge. In addition to emphasizing the importance of capturing technology knowledge before starting system development, DOD’s policy also highlights the importance of demonstrating design maturity before moving from the integration phase of system development into system demonstration and initial manufacturing. The policy establishes a design readiness review between the two phases to determine whether a product’s design is mature and stable and whether the product is ready to move ahead. While DOD’s policy does not require programs to demonstrate any specific level of design maturity, our past work has found that a key indicator of design maturity is the completion of 90 percent of the system’s engineering drawings. We found that defense programs that moved forward with lower levels of design maturity, as indicated by drawing completion, encountered costly design changes and parts shortages that, in turn, caused labor inefficiencies, schedule delays, and quality problems. Consequently, those programs required significant increases in resources—time and money—over what was estimated at the point each program entered the system demonstration phase. We analyzed engineering drawing completion data for 8 programs initiated under the revised policy that have held a design review, and found that more than half of those programs had not completed 90 percent of their design drawings before they received approval to enter the system demonstration phase of development. We also analyzed drawing-release data for three programs that have not yet held their design review but have projected the number of drawings officials anticipate will be completed when their reviews are held. Based on projections provided by program officials, 2 of those 3 programs are expected to have less than 55 percent of their drawings complete before they seek approval to begin system demonstration and initial manufacturing. Despite the revised policy’s guidance that capabilities should be developed and delivered in individually defined and separately managed increments, a majority of major weapon acquisition programs we assessed continue to be structured to achieve revolutionary increases in capability within one development program. According to the policy, the objective of an evolutionary approach is to balance needs and available capability with resources and put capability into the hands of the user quickly. The policy states that the success of the strategy depends on consistent and continuous definition of requirements and the maturation of technologies that lead to disciplined development and production of systems that provide increasing capability. In this approach, requirements that cannot be satisfied within these limits as well as available financial resources must wait for future generations of the product and be managed as separate system development programs with separate milestones, costs, and schedules. In our case studies of nine acquisition programs initiated under the revised policy, we found only one program—the Small Diameter Bomb—that satisfied all of the criteria of an evolutionary approach. In five case studies, we found that program officials had claimed that their programs were evolutionary, yet our evidence shows they were not evolutionary in practice; and in three cases, program officials chose not to use evolutionary acquisition from the outset. Table 3 summarizes our assessment of the nine case studies. The revised acquisition policy does not contain effective controls that require the demonstration of product knowledge measured against specific criteria to ensure that acquisition officials make disciplined, transparent, and knowledge-based investment decisions. The lack of specific required criteria creates an environment in which unknowns about technology, design, and manufacturing processes are acceptable. Decision makers and program officials are left with no objective measures against which to gauge a program’s level of knowledge, making accountability difficult. In the absence of criteria, transparency in acquisition decisions is essential to ensuring accountability, but key decision documents do not provide sufficient information about major decisions. DOD believes that acquisition decision memorandums, used to document program decisions, provide adequate transparency. However, the decision memorandums we reviewed did not contain an explanation of the decision maker’s rationale and rarely identify remaining risks, especially as they relate to the key knowledge standards emphasized in the policy. Further, the timeliness, accessibility, and depth, of the data contained in the Selected Acquisition Reports, DOD’s primary means of providing Congress with a status report of program performance, inhibits the reports’ usefulness as a management and oversight tool. In November 2003, we reported that the revised acquisition policy lacked many of the controls that leading commercial companies rely on to attain an acceptable level of knowledge before making additional significant investments. Controls are considered effective if they are backed by specific criteria and if decision makers are required to consider the resulting data before deciding to advance a program to the next level. Controls used by leading companies help decision makers gauge progress in meeting cost, schedule, and performance goals and hold program managers accountable for capturing relevant product knowledge to inform key investment decisions. The controls we have articulated as best practices used by successful commercial product developers are listed below in table 5. Some senior officials with the Office of the Secretary of Defense believe that the effective use of controls in DOD’s policy and the establishment of more specific criteria for decision making would improve program outcomes. They note that specific criteria need to be established and that programs need to be held accountable to those criteria before being permitted to proceed into the next phase. They also note that the criteria for moving an acquisition effort from one phase of the process to the next, primarily documented in acquisition decision memorandums as exit criteria, are not typically specific and often do not relate to the key knowledge-based criteria suggested in the policy. We found this to be true for our nine case study programs. We reviewed acquisition decision memorandums in our case studies and determined that they were not useful in explaining the decision maker’s rationale and in almost all of the cases they did not address the key knowledge criteria suggested in the acquisition policy. In most instances, the decision maker simply noted that the program being assessed was ready to proceed into system development, but did not provide an explanation of the rationale for the decision. Senior officials with the Office of the Secretary of Defense told us that they agree that a better explanation of the decision maker’s rationale, specifically in instances where the knowledge criteria are not fully met, would provide transparency and ultimately allow for a more accountable decision-making process. The following two examples illustrate how decision documentation is lacking: The Future Combat System program received approval to enter system development and demonstration in 2003, with 19 percent of its critical technologies mature, well below the policy’s standard. The acquisition decision memorandum supporting this decision did not provide the rationale for approving the system with such a large number of immature critical technologies. The memo did direct an updated review of the decision 18 months later and that the program “remain flexible and open to accommodate trades in the system architecture and in the individual systems’ designs.” The Joint Strike Fighter program was approved to enter system development in 2001. The acquisition decision memorandum did not address the fact that 75 percent of the program’s critical technologies were not mature to the policy’s standard. The memorandum did acknowledge that the program’s requirements could be changed or modified, noting that further refinements in the requirements should be explored as a potential way to reduce program costs. However, the memorandum did not explain why the decision maker determined that the program should enter development without achieving the technology and requirements knowledge emphasized in the policy. The acquisition decision memorandums for most of the other programs we reviewed did not specifically address critical gaps in knowledge, nor did they effectively explain the decision makers’ rationale for deeming those programs ready to begin system development. In memos where we found a reference to key knowledge principles, such as technology maturity, the decision makers acknowledged that more effort was needed to meet the policy’s suggested criteria but considered the risk acceptable to begin development. These memos did not explain why risks were considered acceptable. For example, the Navy’s Multi-Mission Maritime Aircraft program had none of its critical technologies mature at program initiation. The decision maker acknowledged the need to further mature the critical technologies but approved the program to enter development. Instead of holding the program to the policy’s criteria for entering development, the decision maker simply directed the Navy to work with the Office of the Secretary of Defense to implement risk mitigation and technology maturation plans during the integration phase of system development. In addition to the lack of transparency provided through acquisition decision memoranda, we also found that the data presented to Congress in DOD’s Selected Acquisition Reports (SARs) provided only limited usefulness as an oversight tool. Since 1969, SARs have been the primary means by which DOD reports the status of major weapon system acquisitions to Congress. SARs are reports that are expected to contain information on the cost, schedule, and performance of major weapon systems in comparison with baseline values established at program start, full-scale development, and production decision points. Our analysis, as well as a previous GAO review, of current and historical SAR data found that the timeliness, accessibility, and depth of the data contained in the reports limits their usefulness as an oversight tool. Our prior review noted that a number of opportunities exist for DOD to give Congress more complete information on the performance of major defense acquisition programs. DOD agreed that SAR data could be improved to make it more useful to Congress. Failing to consistently implement the knowledge-based process and evolutionary principles emphasized in the revised acquisition policy— coupled with a lack of specific criteria for making key investment decisions—are keeping DOD on its historical path of poor cost and schedule outcomes. Most programs are incurring the same scope of cost overruns and schedule delays as programs managed under prior DOD policies. More consistent use of the early acquisition processes would improve the quality and viability of program business cases by ensuring they are founded on knowledge obtained from rigorous and disciplined analysis. The initiative by Office of the Secretary of Defense to reinstitute the use of systems engineering is a step in the right direction. However, in order for this initiative to be effective DOD must establish and enforce specific criteria at key decision points. Our past work has identified and recommended criteria and controls that should be consistently applied at major decision points. The enforcement of these criteria is critical to ensuring that programs have the knowledge necessary to successfully move forward through the acquisition process. DOD officials have acknowledged the advantages of using knowledge-based criteria and controls, but believe the policy already includes enough controls to achieve effective program results. However, without enforceable criteria, defense officials are challenged to determine whether adequate knowledge has been obtained for investing taxpayer dollars. The lack of enforceable criteria also makes it difficult to hold defense officials accountable for their decisions. DOD must ensure that appropriate knowledge is captured and used at critical junctures to make decisions about moving a program forward and investing more money. We recommend that the Secretary of Defense require program officials to demonstrate that they have captured appropriate knowledge at three key points—program start, design review for transitioning from system integration to system demonstration, and production commitment—as a condition for investing resources. At a minimum those controls should require program officials to demonstrate that they have achieved a level of knowledge that meets or exceeds the following criteria at each respective decision point: Program start (Milestone B): Start of product development Demonstrate technologies to high readiness levels Ensure that requirements for the product are informed by the systems- Establish cost and schedule estimates for product on the basis of knowledge from preliminary design using system engineering tools Conduct decision review for program start Design readiness review: Beginning of system demonstration Complete 90 percent of design drawings Complete subsystem and system design reviews Demonstrate with prototype that design meets requirements Obtain stakeholders’ concurrence that drawings are complete and Complete the failure modes and effects analysis Identify key system characteristics Identify critical manufacturing processes Establish reliability targets and growth plan on the basis of demonstrated reliability rates of components and subsystems Conduct decision review to enter system demonstration Production commitment (Milestone C): Initiation of low-rate production Demonstrate manufacturing processes Build production-representative prototypes Test production-representative prototypes to achieve reliability goal Test production-representative prototypes to demonstrate product in Collect statistical process control data Demonstrate that critical processes are capable and in statistical Conduct decision review to begin production Furthermore, to ensure that major decisions are transparent and that program officials and decision makers are held accountable, we recommend that the Secretary of Defense require decision makers to include written rationale for each major decision in acquisition decision documentation. The rationale should address the key knowledge-based criteria appropriate for milestone decisions, explain why a program’s level of knowledge in each area was deemed acceptable if criteria have not been met and provide a plan for achieving the knowledge necessary to meet criteria within a given time frame. DOD provided written comments on a draft of this report. The comments appear in appendix II. DOD partially concurred with our recommendation that the Secretary of Defense should establish specific controls to insure that program officials demonstrate that they have captured a level of knowledge that meets or exceeds specific criteria at three key points in the acquisition process: program start, design readiness review, and production commitment. DOD agreed that knowledge-based decision making is consistent with sound business practice and stated that it would continue to develop policy that reflects a knowledge-based approach and improves acquisition outcomes. DOD noted that it would consider our recommendations as it reassesses the DOD acquisition business model and the knowledge required at each decision point. We believe that DOD’s plan to reassess its business model provides a good opportunity to establish the controls and specific criteria recommended in this report. Therefore, we are retaining our recommendation that the Secretary of Defense should establish controls to insure that program officials demonstrate that they have captured a level of knowledge that meets or exceeds specific criteria at three key points in the acquisition process. DOD also partially concurred with our recommendation that the Secretary of Defense require decision makers to provide written rationale in acquisition decision documentation for each major decision. DOD agreed that acquisition decisions should be documented, decision makers should be held accountable, and that they should provide the rationale for their decisions. DOD believes that the implementation of Section 801 of the National Defense Authorization Act for FY 2006 reinforces these processes. The act calls for the decision maker to certify that the program meets certain requirements, such as technology maturity, prior to starting a new development program at Milestone B. However, the act is focused on the decision to start a development program and does not identify specific criteria for programs to be measured against at design readiness review or production commitment. We believe our recommendation adds transparency and accountability to the process because it requires the decision maker to provide the rationale for a decision to allow a program to move forward, not only at Milestone B but at other key decision points as well. Therefore, we are retaining our recommendation that the Secretary of Defense require decision makers to provide written rationale for each major decision in acquisition decision documentation. We are sending copies of this report to the Secretary of Defense; the Secretaries of the Air Force, Army, and Navy; and the Director of the Office of Management and Budget. We will provide copies to others on request. This report will also be available at no charge on GAO’s Web site at http://www.gao.gov. If you have any questions about this report or need additional information, please call me at (202) 512-4841 (sullivanm@gao.gov). Contact points for the offices of Congressional Relations and Public Affairs are located on the last page of this report. Key contributors to this report were Michael Hazard, Assistant Director; Lily Chin; Ryan Consaul; Christopher DePerro; Travis Masters; and Adam Vodraska. To assess the impact of DOD’s revised acquisition policy, we analyzed cost and schedule data for 23 major defense acquisition programs that were approved to begin system development under the revised policy. We did not assess space, missile defense, or ship programs. We collected our data from Selected Acquisition Reports, presidential budget documents, ongoing GAO work, and pertinent program officials. We utilized previous GAO reports related to defense acquisition policies and worked with knowledgeable GAO staff to ensure the use of current, accurate data. We also analyzed more than 150 annual Selected Acquisition Reports covering a 36-year period from 1969 to 2005, to determine historical trends related to outcomes of acquisition policy implementation. We assessed whether the revised policy’s knowledge-based, evolutionary acquisition principles were being effectively implemented by conducting 9 case study reviews and analyzing design maturity data for 11 programs that have made engineering-drawing data available to GAO. Our case study programs were the Aerial Common Sensor, Multi-Platform Radar Technology Insertion Program, Global Hawk Unmanned Aerial Vehicle, Small Diameter Bomb, Future Combat System, Joint Strike Fighter, Expeditionary Fighting Vehicle, Multi-Mission Maritime Aircraft, and the E-2 Advanced Hawkeye. We interacted directly with numerous program officials to seek input on current developments with their programs. We studied program documents to assess how well programs understand and are implementing the revised acquisition policy. We also analyzed drawing release data for those programs that have either passed their design review or have provided GAO with estimated drawing release data for a future design review to assess design maturity. In several cases, we asked that program offices verify information in these various documents. We also reviewed Department of Defense (DOD) Directive 5000.1, DOD Instruction 5000.2, and the Defense Acquisition Guidebook. In addition we examined each of the military services’ policy directives and guidance, DOD memorandums to include policy intent and DOD expectations regarding policy implementation as well as Joint Capabilities Integration and Development System documents. We interviewed relevant officials in Washington, D.C., from the Office of the Director, Defense Research and Engineering, the Joint Staff, the Office of the Secretary of Defense, and Army, Navy, and Air Force acquisition policy staff in order to better understand the content of these documents and the intent of DOD’s policy. We conducted our review from May 2005 to February 2006 in accordance with generally accepted government auditing standards. Appendix III: Program Data for 23 Programs Initiated under DOD’s Revised Acquisition Policy (as of December 2005) Formal Milestone I or Milestone A decision review? Program office projections. DOD Acquisition Outcomes: A Case for Change. GAO-06-257T. Washington, D.C.: November 15, 2005. Defense Acquisitions: Stronger Management Practices Are Needed to Improve DOD’s Software-Intensive Weapon Acquisitions. GAO-04-393. Washington, D.C.: March 1, 2004. Best Practices: Setting Requirements Differently Could Reduce Weapon Systems’ Total Ownership Costs. GAO-03-57. Washington, D.C.: February 11, 2003 Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 2002. Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices. GAO-02-469T. Washington, D.C.: February 27, 2002. Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 2001. Best Practices: A More Constructive Test Approach Is Key to Better Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 2000. Defense Acquisition: Employing Best Practices Can Shape Better Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000. Best Practices: DOD Training Can Do More to Help Weapon System Programs Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August16, 1999. Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 1999. Defense Acquisitions: Best Commercial Practices Can Improve Program Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999. Defense Acquisition: Improved Program Outcomes Are Possible. GAO/T- NSIAD-98-123. Washington, D.C.: March 17, 1998. Best Practices: DOD Can Help Suppliers Contribute More to Weapon System Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998. Best Practices: Successful Application to Weapon Acquisition Requires Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.: February 24, 1998. Best Practices: Commercial Quality Assurance Practices Offer Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 1996.

Summary: The Department of Defense (DOD) is planning to invest $1.3 trillion between 2005 and 2009 in researching, developing, and procuring major weapon systems. How DOD manages this investment has been a matter of congressional concern for years. Numerous programs have been marked by cost overruns, schedule delays, and reduced performance. Over the past 3 decades, DOD's acquisition environment has undergone many changes aimed at curbing cost, schedule, and other problems. In order to determine if the policy DOD put in place is achieving its intended goals, we assessed the outcomes of major weapons development programs initiated under the revised policy. Additionally, we assessed whether the policy's knowledge-based, evolutionary principles are being effectively implemented, and whether effective controls and specific criteria are in place and being used to make sound investment decisions. Changes made in DOD's acquisition policy over the past 5 years have not eliminated cost and schedule problems for major weapons development programs. Of the 23 major programs we assessed, 10 are already expecting development cost overruns greater than 30 percent or have delayed the delivery of initial operational capability to the warfighter by at least 1 year. The overall impact of these costly conditions is a reduction in the value of DOD's defense dollars and a lower return on investment. Poor execution of the revised acquisition policy is a major cause of DOD's continued problems. DOD frequently bypasses key steps of the knowledge-based process outlined in the policy, falls short of attaining key knowledge, and continues to pursue revolutionary--rather than evolutionary or incremental--advances in capability. Nearly 80 percent of the programs GAO reviewed did not fully follow the knowledge-based process to develop a sound business case before committing to system development. Most of the programs we reviewed started system development with immature technologies, and half of the programs that have held design reviews did so before achieving a high level of design maturity. These practices increase the likelihood that problems will be discovered late in development when they are more costly to address. Furthermore, DOD's continued pursuit of revolutionary leaps in capability also runs counter to the policy's guidance. DOD has not closed all of the gaps in the policy that GAO identified nearly 3 years ago, particularly with regard to adding controls and criteria. Effective controls require decision makers to measure progress against specific criteria and ensure that managers capture key knowledge before moving to the next acquisition phase. However, DOD's policy continues to allow managers to approach major investment decisions with many unknowns. Without effective controls that require program officials to satisfy specific criteria, it is difficult to hold decision makers or program managers accountable to cost and schedule targets. In this environment, decision-making transparency is crucial, but DOD is lacking in this area as well.