Traditionally, DOD’s strategy for acquiring major weapon systems has been to plan programs that would achieve a big leap forward in capability. However, because the needed technologies often are not yet mature, programs stay in development for years until the technologies are demonstrated. As a result, weapon systems have frequently been characterized by poor cost, schedule, and performance outcomes. This has slowed modernization efforts, reduced the buying power of the defense dollar, delayed capabilities for the warfighter, and forced unplanned—and possibly unnecessary—trade-offs among programs. Our extensive body of work shows that leading companies use a product development model that helps reduce risks and increase knowledge when developing new products. This best practices model enables decision makers to be reasonably certain about their products at critical junctures during development and helps them make informed investment decisions. This knowledge-based process can be broken down into three cumulative knowledge points. Knowledge point 1: A match must be made between the customer’s needs and the developer’s available resources—technology, engineering knowledge, time, and funding—before a program starts. Knowledge point 2: The product’s design must be stable and must meet performance requirements before initial manufacturing begins. Knowledge point 3: The product must be producible within cost, schedule, and quality targets and demonstrated to be reliable before production begins. To bolster the knowledge-based process, leading companies use evolutionary product development, an incremental approach that enables developers to rely more on available resources rather than making promises about unproven technologies. While the user may not initially receive the ultimate capability under this approach, the initial product is available sooner and at a lower, more predictable cost. Also, leading companies know that invention cannot be scheduled and its cost is difficult to estimate. They do not bring technology into new product development unless that technology has been demonstrated to meet the user’s requirements. Allowing technology development to spill over into product development puts an extra burden on decision makers and provides a weak foundation for making product development estimates. DOD understands that it must improve acquisition process outcomes if it is to modernize its forces within currently projected resources. To help achieve this goal, DOD has revised its acquisition policy, called the 5000 series, to reflect best practices from successful commercial and DOD programs. The policy covers most—but not all—major acquisitions. The Secretary of Defense has delegated authority to the Missile Defense Agency and to the National Security Space Team to develop separate guidance for missile defense and space systems, respectively.Approximately 35 percent of DOD’s development funds in 2003 went to these systems. (Figure 1 shows how $43.1 billion in development funds were distributed across space, missile defense, and systems covered by the 5000 series.) This report addresses policy for the defense programs covered exclusively under the 5000 series. DOD’s leaders have made significant improvements to DOD’s acquisition policy by adopting the knowledge-based, evolutionary approach used by leading commercial companies. The revised policy has the potential to transform DOD’s acquisition process by reducing risks and increasing the chances for successful outcomes. The policy provides a framework for developers to ask themselves at key decision points whether they have the knowledge they need to move to the next phase of acquisition. If rigorously applied, this knowledge-based framework can help managers gain the confidence they need to make significant and sound investment decisions for major weapon systems. In placing greater emphasis on evolutionary product development, the policy sets up a more manageable environment for achieving knowledge. Another best practice reflected in the policy’s framework is separating technology development from product development, which reduces technological risk at the start of a program. As shown in table 1, DOD’s policy emphasizes best practices used by leading companies. Similar to the best practices model, DOD’s policy divides its acquisition process into phases, as shown in figure 2. Key decisions are aligned with the three critical junctures of a product’s development, or knowledge points. In other similarities, DOD’s framework pinpoints program start at milestone B, about the same point as program start on the best practices model. At the midway point on both approaches, a stable product design should be demonstrated. With DOD’s framework, managers are required to know—by the time full-rate production decision review occurs—whether the product can be produced within cost, schedule, and quality targets. This requirement occurs earlier in the best practices model, before production begins, or at knowledge point 3. Leading companies have used this approach to reduce risks and to make costs and delivery dates more predictable. While DOD has strengthened its acquisition policy with a knowledge- based, evolutionary framework, the policy does not include many of the same controls that leading companies rely on to attain a high level of knowledge before making additional significant investments. Controls are considered effective if they are backed by measurable criteria and if decision makers are required to consider them before deciding to advance a program to the next level. Controls used by leading companies help decision makers gauge progress in meeting cost, schedule, and performance goals and ensure that managers will (1) conduct activities to capture relevant product development knowledge, (2) provide evidence that knowledge was captured, and (3) hold decision reviews to determine that appropriate knowledge was captured to move to the next phase. To determine if DOD has the necessary controls, we compared controls in DOD’s policy with those used in the best practices model at three critical junctures. Table 2 shows the presence or absence of controls for various versions of DOD policy since 1996, including the May 2003 revision. At all three knowledge points, DOD’s policy does not provide all the necessary controls used by commercial companies. For example, at program launch (milestone B) or when knowledge point 1 should be reached, the policy requires decision makers to identify and validate a weapon system’s key performance requirements and to have a technical solution for the system before program start. This information is then used to form cost and schedule estimates for the product’s development. However, the policy does not emphasize the use of a disciplined systems engineering process for balancing a customer’s needs with resources to deliver a preliminary design. The lack of effective controls at knowledge point 1 could result in gaps between requirements and resources being discovered later in development. At the design readiness review or when knowledge point 2 should be reached, DOD’s policy does not require specific controls to document that a product is ready for initial manufacturing and demonstration. DOD’s policy suggests appropriate criteria, such as number of subsystem and system design reviews completed, percentage of drawings completed, planned corrective actions to hardware and software deficiencies, adequate development testing, completed failure modes and effects analysis, identification of key system characteristics and critical manufacturing processes, and availability of reliability targets and growth plans. However, these criteria are not required. For example, we found that a key indicator of a product’s design stability is the completion of 90 percent of the engineering drawings supported by design reviews. DOD’s policy does not require that a certain percentage of drawings or design reviews be completed to ensure the design is mature enough to enter the system demonstration phase. As a result, a decision maker has no benchmark to consider when deciding to advance a program to the next level of development. Finally, at production commitment or when knowledge point 3 should be reached, DOD’s policy does not require specific controls to document that a product can be manufactured to meet cost, schedule, and quality targets before moving into production. For example, the policy states there should be “no significant manufacturing risks” at the start of low-rate production but does not define what this means or how it is to be measured. DOD’s policy does not require the demonstrated control of manufacturing processes and the collection of statistical process control data until full-rate production begins but even then fails to specify a measurable control. Given that low-rate production can last several years, a significant number of products can be manufactured before processes are brought under control, creating a higher probability of poor cost and schedule outcomes. While supporting efforts to build more flexibility into the DOD acquisition process and to develop weapon systems using an evolutionary approach, Congress asked DOD to be more disciplined in its approach. The Defense Authorization Act for Fiscal Year 2003 required DOD to address (1) the way it plans to meet certain statutory and regulatory requirements for managing its major acquisition programs, (2) needed guidance for implementing spiral developments, and (3) technology readiness (at acquisition program initiation). DOD was responsive to all three requirements. With regard to the second requirement, a description of the process that would be used to independently validate that measurable exit criteria for applying a spiral development process have been met was unclear. DOD stated that the milestone decision authority provides that independent validation as part of DOD’s milestone approval process. DOD’s responses to the relevant sections of the act are summarized below. More detailed comparisons are provided in appendixes I, II, and III. Requirements: This section directed DOD to report on its plan to meet certain statutory and regulatory requirements for managing its major acquisition programs applying an evolutionary acquisition process. These include establishing and approving operational requirements and cost and schedule goals for each increment, meeting requirements for operational and live fire testing for each increment, and optimizing total system performance and minimizing total ownership costs. DOD response: In April 2003, DOD submitted its report reflecting how these requirements are addressed in its acquisition policy. According to the report, the policy addresses the statutory and other requirements applicable to all major defense acquisition programs, including each increment of evolutionary acquisition programs. For example, the policy requires that each program or increment of an evolutionary acquisition have a milestone B decision to approve program initiation and to permit entry into system development and demonstration. The policy specifies the statutory and regulatory information necessary to support the decision. Requirements: This section authorizes DOD to conduct a research and development program for a major defense acquisition program using spiral development only if approved by the Secretary of Defense or authorized high-level designee.