In general, a quality product is one that is delivered on time, performs as expected, and can be depended on to perform when needed, at an affordable cost. This applies whether the customer is an individual purchasing a simple consumer good, such as a television, a hospital purchasing medical imaging equipment to help doctors treat cancer patients, or DOD purchasing sophisticated weapons for its warfighters to use on the battlefield. For about 3 decades, DOD based its quality requirements on a military standard known as MIL-Q-9858A, and its quality assurance practices were oriented toward discovering defects through inspections. In 1994, the Secretary of Defense announced that commercial quality standards should replace MIL-Q-9858A. The intent was to remove military-unique requirements that could present barriers to DOD in accessing the commercial supplier base. Currently, responsibilities for quality policy and oversight fall under the Systems and Software Engineering organization, within the Office of the Secretary of Defense. Over the past 20 years, commercial companies have had to dramatically improve quality in response to increased competition. Many companies moved from inspection-oriented quality management practices—where problems are identified and corrected after a product is produced—to a process in which quality is designed into a product and manufacturing processes are brought in statistical control to reduce defects. Many companies have also adopted commercial quality standards, such as ISO 9001. This standard was developed by the International Organization for Standardization, a non-governmental organization established in 1947 to facilitate the international coordination and unification of industrial standards. Similar to DOD’s MIL-Q-9858A, ISO 9001 includes requirements for controlling a product’s design and development, and production, as well as processes for oversight and improvement. Some industries, such as the automotive and aerospace industries, also have standards specific to their sector based on the ISO 9001. Because supplier parts account for a substantial amount of the material value of many companies’ products, companies may require their suppliers to adopt the same standards. In practice, DOD and its prime contractors both participate in activities that contribute to weapon system quality. DOD plays a large role in quality when it sets key performance parameters, which are the most important requirements DOD wants prime contractors to focus on during development. For example, if reliability is one of those key performance parameters, then prime contractors are expected to focus on it during weapon system design. Prime contractors employ quality assurance specialists and engineers to assess the quality and reliability of parts they receive from suppliers, as well as the overall weapon system. DOD has its own quality specialists within the Defense Contract Management Agency and the military services, such as the Navy’s Supervisor of Shipbuilding organization. DOD’s quality specialists oversee prime contractors’ design, manufacturing, and supplier management activities; oversee selected supplier manufacturing activities; and conduct final product inspections prior to acceptance. GAO previously reported on DOD quality practices in 1996. At that time, we reported that numerous weapon system programs had historically had quality problems in production because designs were incomplete. The B-2 bomber program and the C-17 Airlifter program, for example, encountered major manufacturing problems because they went forward with unstable designs and relied on inspections to find defects once in production. Since 1996, GAO has recommended several times that DOD adopt a knowledge- based acquisition approach used by leading commercial companies to develop its weapon systems. Under this approach, high levels of knowledge are demonstrated at critical decision points in the product development process, which results in successful product development outcomes. Systems engineering is a key practice that companies use to build quality into new products. Companies translate customers’ broad requirements into detailed requirements and designs, including identifying requisite technological, software, engineering, and production capabilities. Systems engineering also involves performing verification activities, including testing, to confirm that the design satisfies requirements. Products borne out of a knowledge-based approach stand a significantly better chance to be delivered on time, within budget, and with the promised capabilities. Related GAO products, listed at the back of this report, provide detailed information about the knowledge-based approach. Although major defense contractors have adopted commercial quality standards in recent years, quality and reliability problems persist in DOD weapon systems. On the 11 weapon systems GAO reviewed, these problems have resulted in billions of dollars in cost overruns, years of schedule delays, and reduced weapon system availability. Prime contractors’ poor systems engineering practices related to requirements analysis, design, and testing were key contributors to these quality problems. We also found problems with manufacturing and supplier quality that contributed to problems with DOD weapon systems. Senior officials from the prime contractor companies we contacted said that they agreed with our assessment of the causes of the quality problems of weapon system programs we reviewed and that disciplined processes help improve overall quality. Quality problems caused significant cost and/or schedule delays in the 11 weapon systems we reviewed. Figure 1 shows the types of problems we found and the resulting impacts. Appendix II contains detailed information about each of the programs’ quality problems. Quality problems occurred despite the fact that each of the prime contractors for these programs is certified to commercial quality standards and most provided us with quality plans that address systems engineering activities such as design, as well as manufacturing, and supplier quality. However, quality problems in these areas point to a lack of discipline or an inconsistency in how prime contractors follow through on their quality plans and processes. GAO’s past work has identified systems engineering as a key practice for ensuring quality and achieving successful acquisition outcomes. Systems engineering is a sequence of activities that translates customer needs into specific capabilities and ultimately into a preferred design. These activities include requirements analysis, design, and testing in order to ensure that the product’s requirements are achievable and designable given available resources, such as technologies. In several of the DOD weapon programs we reviewed, poor systems engineering practices contributed to quality problems. Examples of systems engineering problems can be found on the Expeditionary Fighting Vehicle, Advanced Threat Infrared Countermeasure/Common Missile Warning System, and Joint Air-to- Surface Standoff Missile programs. Design problems have hampered the development of the Marine Corps’ Expeditionary Fighting Vehicle. The system, built by General Dynamics, is an amphibious vehicle designed to transport troops from ships offshore to land at higher speeds and from farther distances than its predecessor. According to program officials, prime contractor design and engineering changes were not always passed to suppliers, resulting in supplier parts not fitting into assemblies because they were produced using earlier designs. Systems engineering problems have also contributed to poor vehicle reliability, even though reliability was a key performance parameter. Consequently, the prime contractor was only able to demonstrate 7.7 hours between mission failures, which was well short of the 17 hours it needed to demonstrate in pre-production testing. Subsequently, the vehicle’s development phase has been extended. Program officials estimate that this extension, which will primarily focus on improving reliability, will last an additional 4 years at an estimated cost of $750 million. For several other weapon systems, inadequate testing was another systems engineering problem. The Army’s Advanced Threat Infrared Countermeasure/Common Missile Warning System program, developed by BAE Systems, is designed to defend U.S. aircraft from advanced infrared- guided missiles. Reliability problems related to the Advanced Threat Infrared Countermeasure jam head forced the Army to initiate a major redesign of the jam head in fiscal year 2006, and fielding of the subsystem has been delayed until fiscal year 2010. According to a prime contractor official, the reliability problems were caused, at least in part, by inadequate reliability testing. Likewise, the Joint Air-to-Surface Standoff Missile program, developed by Lockheed Martin, has experienced a number of flight test failures that have underscored product reliability as a significant problem. Ground testing, which prime contractor officials said could have identified most of the failure modes observed in flight testing, did not occur initially. Prime contractor officials indicated that ground testing was not considered necessary because the program was a spin-off of a previous missile program and there was an urgent need for the new missile. As a result of the test failures, the program has initiated a reliability improvement effort that includes ground and flight testing. A program official reported that the cost of reliability improvements for fiscal years 2006 and 2007 totaled $39.4 million. GAO’s past work addresses the importance of capturing manufacturing knowledge in a timely manner as a means for ensuring that an organization can produce a product within quality targets. Prime contractor activities to capture manufacturing knowledge should include identifying critical characteristics of the product’s design and then the critical manufacturing processes to achieve these characteristics. Once done, those processes should be proven to be in control prior to production. This would include making work instructions available, preventing and removing foreign object debris in the production process, and establishing criteria for workmanship. However, prime contractors’ lack of controlled manufacturing processes caused quality problems on several DOD weapon programs, including the F-22A and LPD 17 programs. The F-22A, a fighter aircraft with air-to-ground attack capability being built by Lockheed Martin, entered production with less than 50 percent of the critical manufacturing processes in control. In 2000, citing budgetary constraints and specific hardware quality problems that demanded attention, the Air Force abandoned its efforts to get manufacturing processes in control prior to the start of production. Subsequently, the contractor experienced a scrap, rework, and repair rate of about 30 percent on early-production aircraft. The contractor also experienced major problems with the aircraft canopy. According to program officials, the aircraft uses a first-of-a-kind canopy, with an external metallic stealth layer. The contractor did not bring its manufacturing processes in control and the canopy cracked near the mounting holes. This problem was discovered in March 2000 and temporarily grounded the flight test aircraft. In addition, in 2006 a pilot was trapped in an F-22A for 5 hours when a defective activator prevented him from opening the canopy.