In fiscal year 1995, the Department of Defense (DOD) plans to spend over $79 billion for research, development, test, evaluation, and production of weapon systems. While DOD has acquired some of the most technologically advanced and effective weapon systems, DOD has often been criticized for not acquiring the systems in the most efficient manner. As weapon system programs progress through the phases of the acquisition process, they are subject to review at major decision points called milestones. The milestone review process is predicated on the principle that systems advance to higher acquisition phases by demonstrating that they meet prescribed technical specifications and performance thresholds. Figure 1.1 illustrates the DOD's weapon system acquisition process. At milestone 0, a determination is made about whether an identified mission need warrants a study of alternative concepts to satisfy the need. If warranted, the program is approved to begin the concept exploration and definition phase. At milestone I, a determination is made about whether a new acquisition program is warranted. If warranted, initial cost, schedule, and performance goals are established for the program, and authorization is given to start the demonstration and validation phase. At milestone II, a determination is made about whether continuation of development, testing, and preparation for production is warranted. If warranted, authorization is given to start the engineering and manufacturing development phase. Also, approval of this phase will often involve a commitment to low-rate initial production (LRIP). At milestone III, a determination is made about whether the program warrants a commitment to build, deploy, and support the system. DOD acquisition policy states that program risks shall be assessed at each milestone decision point before approval is granted for the next phase. The policy adds that test and evaluation shall be used to determine system maturity and identify areas of technical risk. Operational test and evaluation (OT&E) is a key internal control to ensure that decisionmakers have objective information available on a weapon system's performance, to minimize risks of procuring costly and ineffective systems. OT&E has been defined as (1) the field test, under realistic conditions, of any item of (or key component of) weapons, equipment, or munitions for the purpose of determining its effectiveness and suitability for use in combat by typical military users and (2) the evaluation of the results of such a test. Over a period of many years, the Congress has been concerned about the performance of weapon systems being acquired by DOD. As early as 1972, the Congress required DOD to provide it with information on the OT&E results of major weapon systems before committing them to production. However, the Congress continued to receive reports from the DOD Inspector General (DOD-IG), us, and others that (1) weapon systems were not being adequately tested before beginning production, (2) fielded systems were failing to meet their performance requirements, and (3) OT&E being conducted on weapon systems was of poor quality. In the late 1970s and early 1980s, the Congress enacted a series of laws to ensure that U.S. military personnel receive the best weapon systems possible and that the U.S. government receives best value for the defense procurement dollar. Among other things, these laws specified that independent OT&E be conducted; established the Office of the Director, Operational Test and Evaluation (DOT&E), and assigned it specific oversight duties and responsibilities; specified that OT&E of a major defense acquisition program may not be conducted until DOT&E approves the adequacy of the plans for that OT&E; required that a major system may not proceed beyond LRIP until its initial OT&E is completed; and required that DOT&E analyze the results of OT&E conducted for each major defense acquisition program and, prior to a final decision to proceed beyond LRIP, report on the adequacy of the testing and whether the results confirm that the items tested are operationally effective and suitable for combat. In the late 1980s, the Congress found that DOD was acquiring a large portion of the total program quantities, using the LRIP concept, without successfully completing OT&E. In the National Defense Authorization Act for Fiscal Years 1990 and 1991 (P.L. 101-189), the Congress addressed this situation by including a definition of LRIP and a requirement that the determination of the LRIP quantities to be procured be made when a decision is made to enter engineering and manufacturing development. According to the act, LRIP was defined as the minimum quantity needed to (a) provide production-representative articles for OT&E, (b) establish an initial production base, and (c) permit orderly ramp-up to full-rate production upon completion of OT&E. In the conference report for the act, the conferees indicated that they did not condone the continuous reapproval of LRIP quantities that eventually total a significant percentage of the total planned procurement. Also, the conferees granted an exception to the LRIP legislation for ship and satellite programs because of their inherent production complexity, small number, high unit cost, and long unit production periods. However, they directed the Secretary of Defense to develop regulations that capture the spirit of the LRIP legislation as it applies to these programs. This special consideration for ships and satellites carries with it additional reporting requirements to improve the oversight of these programs. Finally, in the National Defense Authorization Act for Fiscal Year 1994, the Congress required that the Secretary of Defense ensure that appropriate, rigorous, and structured testing be completed prior to LRIP of any electronic combat or command, control, and communications countermeasure system. Senators David Pryor and William V. Roth, Jr., requested that we review DOD's use of LRIP in the acquisition of major defense programs. Specifically, the Senators asked that we determine whether LRIP policies were resulting in the production of systems with adequate performance capabilities and the legislation underlying the LRIP policies was adequate. We analyzed the legislation and DOD policies governing the production and testing of weapon systems, particularly those dealing with (1) the purposes of LRIP, (2) the criteria or requirements for entering LRIP and full-rate production, and (3) the testing requirements related to this process. We used the results of our extensive body of work from the past decade or so on defense acquisition programs and the acquisition processto determine whether the LRIP concept, as currently authorized and practiced by DOD, has resulted in a premature commitment to production of both major and nonmajor systems. We reviewed the 1993 report of the DOD-IG on LRIP and held discussions with the DOD-IG staff. We gathered and summarized data on numerous ongoing system acquisition programs (both major and nonmajor programs) and supplemented that information with discussions with officials from the Office of the Secretary of Defense and the military services. In addition, we gathered and analyzed information on the advantages and disadvantages of conducting OT&E before LRIP (for both major and nonmajor systems). We also held discussions with those officials on DOD's current acquisition strategies and OT&E policies and practices. This review was conducted from April 1993 to May 1994 in accordance with generally accepted government auditing standards. Our extensive body of work over the years has amply demonstrated that improper usage of LRIP has been widespread. Many major and nonmajor systems from each of the services have been prematurely committed to production, which often results in problems being found after a substantial number of units have been produced and a significant commitment made to the entire procurement program. In addition, contrary to the statutory emphasis on minimum LRIP quantities and conferee statements, many programs continue in LRIP for prolonged periods. DOD's continuing reluctance to employ the discipline of early OT&E is evident in each of the services and in many major and nonmajor programs. Adequate controls have not been established over the start and continuation of LRIP. A requirement to successfully complete enough independent testing in an operational environment to ensure that the item meets its key performance parameters before LRIP starts would be feasible in most cases and would be an effective management control over the premature start of production. Over the years, we have found numerous instances from all three services in which production of major and nonmajor systems was permitted to begin and continue based not on the systems' technical maturity, but on schedule or other considerations. DOD has frequently committed programs to production without assurance that the systems would perform satisfactorily. Many of the weapon systems that start production prematurely later experience significant operational effectiveness and/or suitability problems. As a result, major design changes were often needed to correct the problems, additional testing was needed to verify that the corrective action was effective, and costly retrofits were needed for any delivered units. A few of the many examples of premature and extensive commitments to production of major and nonmajor systems are shown in the following tables. Table 2.1 shows systems that entered LRIP before any operational tests were conducted and later experienced significant problems during the tests. Table 2.2 shows systems that were subjected to early operational tests but were allowed to enter LRIP even though the performance deficiencies were not corrected. Programs that enter production prematurely often require more time and resources than originally planned to correct problems and to meet the requirements for full-rate production. LRIP is often continued, despite the evidence of technical problems, well beyond that needed to provide test articles and to establish an initial production capability. As a result, major production commitments are often made during LRIP. In the conference report for the LRIP legislation, the conferees stated that they did not intend to authorize the continuance of LRIP on an indefinite basis. Nevertheless, the existing LRIP legislation does not include any specific principles or guidelines on when and how programs should begin LRIP, on the type and amount of testing to be done before LRIP, on how much LRIP can or should be done, or under what circumstances LRIP should be curtailed or stopped. Instead, the emphasis has been placed almost entirely on the full-rate production decision, at which point the law requires, among other things, that a report be provided on the adequacy of the testing conducted and an assessment be made of the system's operational effectiveness and suitability. Although programs are delayed getting approval for full-rate production, LRIP is rarely stopped or slowed significantly. As a result, the decision to start LRIP, in many cases, is also the de-facto full-rate production decision. DOD's written policies provide that acquisition strategies be event-driven and link major contractual commitments and milestone decisions to demonstrated accomplishments in development, test, and initial production. However, DOD policies state that a primary goal in developing an acquisition strategy shall be to minimize the time and cost of satisfying a need consistent with common sense and sound business practices. In addition, DOD's policies state, but without detailed requirements, that OT&E should be conducted throughout the acquisition process. However, while DOD is statutorily required to conduct OT&E before full-rate production is approved, DOD's policies permit LRIP to begin before any OT&E is conducted. The point at which LRIP begins is not a required milestone under DOD policy. As a result, for many major defense acquisition programs, the services do not plan to conduct any OT&E prior to the start of LRIP. It has been and continues to be the exception, rather than the rule, for programs to include OT&E before LRIP starts. In some instances, the services plan to start LRIP even though they plan to use developmental or prototype units for their initial OT&E, not LRIP units. Although not required by written DOD or Navy policy, the Navy now performs a limited phase of OT&E before LRIP to prepare for later phases of OT&E on some of its programs. However, these programs are not required to meet specific testing-related criteria before entering LRIP. As shown in table 2.2, even when some OT&E was conducted prior to the start-up of production, identified problems were not verified as corrected, and significant performance problems emerged later in the program. Over the past several years, DOD has stated that it planned to reemphasize the need for OT&E as early as possible in the acquisition process. However, we have not detected any reemphasis on early OT&E, and DOD's 1991 revision of its key acquisition directives did not address this issue. DOD acquisition and testing officials concede that there has not been any major reemphasis on early OT&E. In fact, DOD has recently supported legislative proposals that would reduce the current overall requirements to conduct OT&E. DOD has recognized that reducing the amount of production prior to completing development provides for greater design maturity, which increases the likelihood of meeting system requirements and avoiding retrofit costs. In commenting on our 1992 report, DOD officials said they were lessening the amount of concurrent development and production in weapon programs due to the end of the Cold War. In 1992, the Under Secretary of Defense for Acquisition also stated that the need to replace existing weapon systems in order to maintain a significant technological advantage was no longer as urgent. However, acquisition strategies of many current programs do not reflect these positions. DOD's acquisition practices continue to stress the importance of minimizing the time to deploy new or improved weapon systems. Highly concurrent acquisition strategies continue to be featured in many current major and nonmajor programs, with little, if any, OT&E expected until well after the start of production and a significant commitment is made to the procurement of the system. Our analysis of the current selected acquisition reports shows that many programs continue to postpone initial OT&E until well after the start of production. LRIP is expected to be approved in February 1996 for the Army's Secure Mobile Anti-Jam Reliable Tactical Terminal. Initial OT&E will not be completed until July 1998, by which time a total of 125 units, or 3 years of LRIP, is planned to be approved out of a total program quantity of 367 units. The LRIP decision for the Air Force's F-22 aircraft program is expected in June 1998, and initial OT&E is to be conducted from March to November 2001. Thus, 1 year of preproduction verification and 4 years of LRIP--80 aircraft out of a total quantity of 442 units--are planned to be approved before completion of OT&E. The Navy plans to procure 106 of the 630 planned Multifunctional Information Distribution Systems before OT&E is completed in December 2000 and a full-rate production decision is made in June 2001. In addition, 42 prototype systems are to be built as part of the system development effort. These programs feature major commitments to LRIP before development is completed and before any OT&E is completed, even though developmental prototypes are expected to be available for testing in these programs. Accordingly, a substantial and frequently irreversible commitment to production will have been made before the results of independent testing are available to decisionmakers. In its 1993 report, the DOD-IG found that major defense acquisition programs were entering LRIP without meeting development, testing, and production readiness prerequisites. As a result, the DOD-IG concluded that DOD incurred excessive program risk of overcommitment to production of systems without obtaining assurance that the design is stable, potentially operationally acceptable, and capable of being produced efficiently. Among other things, the DOD-IG recommended that DOD (1) provide guidance on the specific minimum required program accomplishments for entry into and continuation of LRIP and (2) require that program-specific exit criteria be established for entry into and continuation of LRIP. DOD is currently considering what, if any, actions will be taken in light of the DOD-IG's recommendations. The decision to begin LRIP should be given much more attention because decisionmakers find it very difficult to stop or slow down programs once they are in production. Given the cost risks involved and DOD's inability or unwillingness to curtail production after it starts, we agree with the DOD-IG that controls are urgently needed over the start and continuation of LRIP. A key criterion for all programs beginning LRIP should be the completion of a phase of independent testing in an operational environment. During such testing, some problems should be expected. However, enough realistic testing should be conducted for the services' independent testing agencies and/or DOT&E to be able to certify to the decision authority that (1) the system's developmental testing is essentially complete and the basic results have been validated in an operational environment, (2) the system has clearly shown that it can meet the key parameters among its minimum acceptable performance requirements, (3) the system has clearly demonstrated the potential to fully meet all of its minimum acceptable requirements for performance and suitability without major or costly design changes, and (4) the system should be able to readily complete its remaining OT&E in time to support the planned full-rate production decision. Comprehensive testing of a system's operational suitability features, such as supportability, may not be possible during early independent testing. However, the testing should be sufficient to reveal major suitability problems. Conducting OT&E before LRIP will not, by itself, result in a better weapon system, but it is the best means available to guard against the premature start of production. Decisionmakers need verifiable information on system design maturity and where corrective actions are needed before production start-up. Every effort should be made to correct problems in development, not in production, because early fixes are less expensive, easier to implement, and less disruptive. In today's national security environment, there should be very few cases in which an urgent need dictates that DOD start production without assurance that the system will work as intended. We realize that, for some programs, a significant effort (personnel and facilities) may be needed to produce one or more prototypes for a phase of early OT&E. These programs would typically involve inherent fabrication complexity, small procurement quantities, high unit cost, and long unit production periods. To suspend that type of effort while OT&E is underway could be costly and disruptive. Alternatively, key subsystems should be independently tested on surrogate platforms before production. Once underway, production should be limited until acceptable OT&E results are obtained on the entire system. We believe that LRIP should be used to focus on (1) addressing producibility and product quality issues; (2) producing just enough systems to support initial OT&E, to prove out the production process, and to sustain the production line; and (3) testing those systems and correcting any deficiencies. A limit on the quantity that can be produced under LRIP would provide an opportunity to correct problems that are identified during initial OT&E, without incurring the risk of overproducing under the LRIP phase. We recommend that the Secretary of Defense revise DOD's acquisition policies in the following ways: Require that, before entry into LRIP, programs (with the exception of ships, satellites, and those other programs that involve inherent fabrication complexity, small procurement quantities, high unit costs, and long unit production periods) plan, buy prototypes for, and conduct enough realistic testing for the service's independent testing agency and/or DOT&E to be able to certify to the decision authority that (1) the system's developmental testing is essentially complete and the basic results of that testing have been validated in an operational environment; (2) the system has clearly shown that it can meet the key parameters among its minimum acceptable performance requirements; (3) the system has clearly demonstrated the potential to fully meet all of its minimum acceptable requirements for performance and suitability without major or costly design changes; and (4) the system should be able to readily complete its remaining OT&E in time to support the planned full-rate production decision. Require that those programs excluded from the requirement to test prototypes instead test all key subsystems in an operational environment before entry into LRIP. Adopt the recommendations made by the DOD-IG regarding controls over the start and continuation of LRIP such as (1) providing guidance on the specific minimum required program accomplishments for entry into and continuation of LRIP and (2) requiring that program-specific exit criteria be established for entry into and continuation of LRIP. We also recommend that the Secretary of Defense work with the service secretaries to ensure that these policies are implemented for the acquisition of both major and nonmajor systems. The legislation defining LRIP has not been effective in accomplishing its purpose, which was to limit the commitment to major production quantities pending satisfactory completion of OT&E. Therefore, we recommend that the Congress legislatively mandate (1) that certain OT&E requirements be met before LRIP may start and (2) specific limits on the number of units allowed to be produced during LRIP. Specifically, the Congress may wish to require that all defense acquisition programs (major and nonmajor) conduct enough realistic testing on the entire system or key subsystems to ensure that its key performance parameters are met before LRIP is permitted to start. In addition, the Congress may wish to (1) specify a percentage (10 percent, for example) of a system's total procurement beyond which a program may not proceed during LRIP and/or (2) amend 10 U.S.C. 2400 (by deleting subsection (b)(3)) to preclude the use of LRIP authority to ramp-up the production rate prior to the successful completion of OT&E.

Summary:
Pursuant to a congressional request, GAO reviewed the Department of Defense's (DOD) use of low-rate initial production (LRIP) in its systems acquisition programs, focusing on whether: (1) DOD LRIP practices result in the production of adequate systems; and (2) the legislation underlying LRIP policies is adequate. GAO found that: (1) despite congressional emphasis on the need for operational test and evaluation (OT&E) prior to system production, legislation and DOD policies permit LRIP to start before any OT&E is conducted because there are no specific guidelines on the type and amount of testing required prior to LRIP; (2) the lack of guidelines has resulted in substantial inventories of unsatisfactory weapons that need costly modifications and some deployments of substandard systems to combat forces; (3) correction of system deficiencies in prematurely produced systems lengthens production schedules and increases resource consumption; (4) major production decisions are often made during LRIP; (5) LRIP severely limits Congress' and DOD decisionmakers' options for dealing with deficient systems; (6) DOD needs accurate, independent information on system performance and suitability to minimize the risks of procuring costly and ineffective systems; and (7) in light of the current national security environment, there should not be an urgent need to start LRIP before system capabilities are adequately tested.