Because computer software controls most functions of modern defense systems, the systems’ performance depends largely on the quality of that complex and increasingly costly software. In fact, many major weapon systems may be inoperable if software fails to function as required. Mission-critical computer software, which is integral to most military applications, tends to be more difficult to develop than software for other types of applications primarily because it must operate in real time under very unique environments. Accordingly, software quality has become a primary concern in emerging defense acquisition programs, including weapon systems; automated information systems; and command, control, communications, and intelligence systems. Software-intensive systems are, by nature, highly complex and often require millions of lines of code. These significant factors increase the overall costs of software. Although the Department of Defense (DOD) does not know precisely how much it spends on software, the Defense Systems Management College projected that DOD would spend about $36.2 billion for software in 1992. The Management College expects software costs to continue to rise at a rate proportionately higher than computer hardware costs. According to the DOD Inspector General, the costs of computer hardware components integral to weapon systems and other critical military and intelligence systems, are expected to remain stable at about $6 billion annually between 1990 and 1995, whereas corresponding software costs are expected to grow from about $30 billion to $42 billion. (See fig. 1.1.) DOD estimates that about 30 percent of its software life-cycle expenditures are for initial development and 70 percent are for post-deployment software support, that is, maintaining, upgrading, and modifying existing software to correct deficiencies, respond to mission changes, or enhance technology. Up-front improvements in the quality of software development processes and more effective software test and evaluation may play a significant role in controlling these costs, which are incurred largely after systems have been fielded. The primary purpose of test and evaluation during the acquisition process is to reduce the risk that the system or equipment either will not meet performance specifications or cannot be effectively used in its intended operational environment. Test and evaluation is therefore designed to detect errors in both software and hardware before a system is fielded and to provide essential information to decisionmakers for assessing acquisition risk. Early in the acquisition cycle, development test and evaluation (DT&E) primarily measures a system’s technical performance and compliance with contractual specifications. DT&E, which starts at the systems requirements phase and proceeds through the design phase, is designed to detect errors in software and hardware prior to operational test and evaluation (OT&E). Later, OT&E focuses on the system’s effectiveness and suitability. (See table 1.1.) Before a system is certified as ready for OT&E, any major deficiency is expected to be identified and corrected during DT&E. Deficiencies discovered during developmental and operational testing affect a system’s cost, schedule, and performance. However, problems that are not identified and resolved until operational testing and production begins are generally more difficult and costly to correct. Test and evaluation is the key internal control to ensure that decisionmakers have valid, credible information for making development and production decisions. OT&E results contribute to decisions not only on acquiring new systems but also on modifying systems deployed in the field and upgrading the software or hardware of systems already in production. The Congress and senior DOD officials have long been concerned with DOD’s inability to field software-intensive defense acquisition programs on time and within budget. Because of these concerns, we initiated this review to identify (1) the extent to which software-related problems affect the performance of defense acquisition programs during OT&E, (2) pervasive barriers in the acquisition process that limit the effectiveness of test and evaluation of software-intensive systems, and (3) DOD’s efforts to resolve software test and evaluation problems. A wide range of technical and management challenges impact the development, testing, and fielding of software-intensive systems. However, this report is not intended to address the technical aspects of the software development process, which, at best, is a difficult and complex undertaking. Rather, the report focuses more directly on those barriers that require the attention of DOD acquisition and technology management officials and that DOD believes limit the effectiveness of OT&E of software-intensive systems. To accomplish our objectives, we reviewed defense acquisition, software development, and test and evaluation policy documents. To determine the status of systems’ software during OT&E, we analyzed the OT&E results of 27 systems that represented the total population of major programs the services had identified as having undergone OT&E during the 2-year period from January 1990 to December 1992. We also visited several prime contractors identified by DOD and service officials to obtain an overview of industry practices. The organizations we visited included the Office of the Under Secretary of Defense for Acquisition, Washington, Office of the Director, Defense Research and Engineering, Washington, Office of the Director, Operational Test and Evaluation, Washington, D.C.; Army’s Director of Information Systems for Command, Control, Communications, and Computers, Washington, D.C.; Test and Evaluation Management Agency, Washington, D.C.; Army Operational Test and Evaluation Command, Alexandria, Virginia; U.S. Army Communications-Electronics Command, Fort Monmouth, New Navy Operational Test and Evaluation Force, Norfolk, Virginia; Fleet Combat Direction Support System Activity, Dam Neck, Virginia; Marine Corps Operational Test and Evaluation Activity, Quantico, Virginia; Air Force Operational Test and Evaluation Center, Albuquerque, Sacramento Air Logistics Center, California; Jet Propulsion Laboratory, Pasadena, California; Hughes Aircraft Corporation, Ground Systems Group, Fullerton, Hughes Aircraft Corporation, Radar Systems Group, Torrance, California; General Dynamics Electronics Division, San Diego, California; Science Applications International Corporation, San Diego, California; and TRW Systems Integration Group, Carson, California. We conducted our review between April 1992 and April 1993 in accordance with generally accepted government auditing standards. Since the 1970s, software problems discovered during OT&E have adversely affected the cost, schedule, and performance of major defense acquisition systems. Because many systems do not undergo rigorous DT&E and therefore begin OT&E before their software is fully mature (i.e., the software is able to satisfy all documented user requirements), they often fall short of system performance expectations. The readiness of such systems for OT&E is therefore questionable. Although DOD recognizes these problems, it has made only limited progress in adopting solutions. Fundamentally, DOD has not (1) acknowledged or adequately addressed the criticality of software to systems’ operational requirements early enough in the acquisition process; (2) developed, implemented, or standardized decision-making tools and processes (e.g., metrics) to help measure or project weapon system software cost, schedule, and performance risk; (3) developed test and evaluation policy that provides specific guidance regarding software maturity; and (4) adequately defined and managed requirements for its increasingly complex software (see ch. 3). To ensure no surprises during OT&E, defense systems are expected to be subjected to rigorous DT&E. Formal operational test readiness reviews also address the readiness of systems for OT&E. However, software-intensive systems have repeatedly failed to meet users’ requirements during OT&E and, in some cases, during operations in the field. This has been recognized in DOD and industry as a major contributor to DOD’s “software crisis.” In general, the thoroughness of DT&E and the readiness of such systems for operational testing has been questionable. According to a 1992 report by the Secretary of the Air Force, virtually all software-intensive defense systems suffer from difficulties in achieving cost, schedule, and performance objectives. Our prior reports, the comments of senior officials responsible for various aspects of software development, and the reports of the services’ operational test agencies (27 such reports were analyzed during our review) corroborate the existence of significant software problems. Our review of the services’ OT&E reports from January 1990 to December 1992 showed that 23 of 27, or 85 percent, of the software-intensive systems tested were immature, ineffective in a threat environment, or difficult to maintain in field operations. Table 2.1 contains some typical examples of software problems found in these systems, and appendix I provides a more complete list of the problems. Since the early 1970s, we have reported that defense systems have begun production without timely or realistic OT&E. More recently, we have reported on software shortfalls in individual systems (see table 2.2). For example, in December 1992 we reported that DOD’s mission-critical computer systems continued to have significant software problems due in part to a lack of management attention, ill-defined requirements, and inadequate testing. DOD officials cited the following reasons for immature software during OT&E: In many cases, rigorous DT&E is not being done before systems begin OT&E; nonetheless, the systems are being certified as ready before they have achieved appropriate levels of maturity. Problems are not identified until it is too late to address them effectively and economically because some program managers may not fully report program weaknesses that could lead decisionmakers (i.e., the Office of the Secretary of Defense, Congress, or the military services’ acquisition executive) to adjust program funding. The career success of participants in the acquisition process is perceived to depend more on getting programs into production than on achieving successful program outcomes. Thus, program managers have incentives to delay testing and take chances that immature systems might succeed in OT&E. The congressional appropriations process forces programs to be calendar-driven rather than event-driven, causing program managers to prematurely certify systems as ready for OT&E to avoid losing funding or slipping schedules. Some program managers give priority to developing software that will support a system production decision and give less attention to the post-deployment support element of life-cycle costs.