Concurrency is broadly defined as the overlap between development and production of a system. The stated rationale for concurrency is to introduce systems in a more timely manner or to fulfill an urgent need, to avoid technology obsolescence and/or to maintain an efficient industrial development/production work force. For measuring the degree of concurrency in this report we used a statutorily required guide issued by the Department of Defense (DOD) in April 1990 for assessing concurrency and associated risk in major acquisition programs. Its measure of concurrency is the amount of initial operational testing and evaluation (IOT&E) completed before entering production of a system. Initial operational tests are field tests intended to demonstrate a system's effectiveness and suitability for military use. IOT&E is a key internal control to ensure that decisionmakers have objective information available on a weapon system's performance and to minimize risks of procuring costly and ineffective systems. In the late 1980s, the Congress found that DOD was acquiring a large portion of total program quantities, using the low-rate initial production (LRIP) concept, without successfully completing IOT&E. As a result, legislation was enacted in 1989 to limit LRIP quantities for major systems. The law, 10 U.S.C. 2400, defined LRIP as the minimum production quantity needed to provide production representative articles for IOT&E, establish an initial production base, and permit an orderly increase in the production rate sufficient to lead to full-rate production after completion of IOT&E. In the conference report supporting the National Defense Authorization Act for Fiscal Years 1990 and 1991 (P.L. 101-189), the conferees indicated that LRIP quantities should not total a significant percentage of a total planned procurement. Later, the Federal Acquisition Streamlining Act of 1994 prescribed new controls for LRIP. The act states that the Secretary of Defense must specifically explain to the Congress why any planned LRIP quantities exceed 10 percent of a planned production quantity of a system, as defined at the milestone II or development decision. This provision, however, was not in effect when the F-22 program reached milestone II. The F-22 passed milestone II in 1991. At that time, the Air Force planned to acquire 648 F-22 operational aircraft at a cost of $86.6 billion. After the Bottom Up Review, completed by DOD in September 1993, the planned quantity of F-22s was reduced to 442 at an estimated cost of $71.6 billion. We recently reported that aircraft systems, including the T-45 trainer aircraft, B-1B bomber, and the C-17 cargo aircraft, as well as many other smaller systems, entered LRIP before successfully completing any IOT&E.This resulted in the purchase of systems requiring significant and sometimes costly modifications to achieve satisfactory performance, acceptance of less capable systems than planned, and in some cases deployment of substandard systems to combat forces. The LRIP contract award is scheduled for September 1997. LRIP aircraft are those to be procured during the period of concurrency. In 1990, DOD performed a statutorily required analysis of the concurrency in acquisition programs partly to define the appropriate measures for evaluating the degree of concurrency and associated risk in programs. The Office of the Secretary of Defense defined a highly concurrent program as one that proceeds into LRIP before significant IOT&E is complete. Using DOD guidelines, concurrency in the F-22 program is high because the F-22 program is scheduled to proceed into LRIP well before any IOT&E is started. Further, considering the new technology advancements being developed for use in the aircraft, the level of concurrency increases the cost, schedule, and technical risks of the program. We found that development flight tests of critical F-22 technology advances are not scheduled to begin until about 1 year after LRIP is scheduled to start and over $2 billion will have been committed to procure F-22 aircraft. According to the F-22 acquisition plan, the Air Force will commit to LRIP quantities that increase from 4 aircraft a year to 36 a year (an 800-percent increase), totaling 80 aircraft, before completion of IOT&E. Production of 36 aircraft a year under LRIP represents 75 percent of the planned full-production rate. The estimated cost of those 80 aircraft is $12.4 billion. Figure 1 shows the planned schedule of commitments to procurement of F-22 aircraft and the estimated cumulative costs prior to completion of IOT&E. A first set of hardened tooling is required initially to produce the developmental aircraft for testing. Program office officials told us that the maximum quantity of F-22s that can be produced with the first set of tooling is about 6 to 8 aircraft a year. The concurrency of development, testing and production in the F-22 program is shown in figure 2, which shows concurrent development and production from September 1997 through February 2002. Low-rate production of the F-22 is scheduled to begin in September 1997. However, IOT&E is not scheduled to take place until December 1999 through February 2002. Thus, the testing is not scheduled to be complete until over 4 years after the start of production and the commitment at an estimated cost of $12.4 billion to procure 80 aircraft (4 preproduction aircraft and 76 production aircraft), or 18 percent of all 442 aircraft to be procured. Although laboratory tests are underway and simulations of the avionics are planned, the Air Force does not plan to flight test several of the critical F-22 technology advances on an F-22 until well after the start of production in September 1997. Flight tests of low observability are not scheduled to begin until September 1998. Although the highest risk element of the F-22 program was reported to be the integrated avionics, the first flight test of an F-22 equipped with a complete integrated avionics system is not scheduled to begin until September 1999, 2 years after the start of production. By the time that testing begins, the Air Force will have already made commitments to procure 20 aircraft and long lead materials for an additional 24. For programs entering the engineering and manufacturing phase of the acquisition cycle, the Federal Acquisition Streamlining Act of 1994 requires the Secretary of Defense to explain to the Congress any plans to procure more than 10 percent of the total procurement quantity in the LRIP phase. This provision of the act is not retroactive to the F-22 program. In 1991, when milestone II was approved for the F-22 program, the total aircraft procurement quantity planned was 648. Accordingly, 10 percent would have been 65 aircraft. Currently, 442 aircraft are to be procured, meaning 10 percent would be 44 aircraft. The number of F-22 LRIP aircraft currently planned is 80, exceeding 10 percent, in either case. The Air Force's planned commitment to production of F-22's prior to completion of IOT&E, as a percentage of total production, exceeds the commitments made for recent fighter programs except the F-15, in which the percent is about the same as the F-22. Figure 3 compares the planned percentage of aircraft committed to production before completion of IOT&E for the F-22 and percentages committed for other recent fighter programs. Although the actual number of F-22 aircraft to be acquired before completion of IOT&E is lower than in the F-14, F-15, F-16, and F/A-18, the other fighters were acquired before the end of the Cold War when a greater degree of urgency existed for procuring aircraft. The Air Force plans to use advances in technologies and innovations to provide high performance and increased reliability and maintainability for the F-22. The integrated avionics, engine, and stealth characteristics are the primary areas that increase the cost, schedule, and technical risk in the F-22 program. After reviewing the program, the DOD Defense Science Board (DSB) concluded that concurrency was acceptable and risks were readily controllable, but noted that the F-22 program is very ambitious technically. Descriptions of some of the problems that have occurred in the development program are included below. The purpose of these descriptions is to illustrate that there remain important cost, schedule, and technical risks in the F-22 program. The F-22 Program Office has taken a number of steps to reduce the technical risks of the program, including a 54-month demonstration/validation phase using an F-22 prototype, and a risk management program for engineering and manufacturing. Some deficiencies associated with the higher risk features of the F-22 have been experienced during ground tests, requiring expensive redesigns. The F-22's integrated avionics are expected to provide unprecedented situational awareness to the pilot. The F-22 is the first aircraft to use integrated avionics, that is, critical systems such as the radar, the weapons management system, and electronic warfare sensors that work as one unit. The key to achieving the necessary performance is the successful development of highly advanced integrated computer processors, known as the common integrated processors, and large amounts of software. Avionics and software integration has been characterized by the DOD Defense Acquisition Board as one of the highest risks to the successful development of the F-22. The risk assessment was prepared for the DOD Defense Acquisition Board to evaluate the readiness of the F-22 to begin the engineering and manufacturing development phase of the acquisition cycle in 1991. This report in June 1991 explained that the estimated 1.3 million single lines of software code needed for the F-22 represented the largest software task ever for an attack/fighter onboard software program. Further, the DSB in 1993 rated the integrated avionics as the highest technical risk in the F-22 program. Program managers for the F-22 agreed in October 1994 that the avionics and software integration are the most risky tasks facing the contractors. In a separate report, we concluded in 1994 that although the Air Force's planned strategy for the F-22 software was generally sound in concept, some significant features of the strategy were not being followed. For example, the independent verification and validation of software products--part of the quality assurance process--was less rigorous than planned. In addition, the technical risks being encountered with the system/software engineering environment and common integrated processor were not being formally reported to DOD management. Finally, we indicated that the Air Force had begun actions to respond to our concerns. DOD responded to that report in February 1995, indicating that the quality assurance program is now being complied with as planned. DOD also stated that common automated tools had matured and would support completion of the software development effort through the engineering and manufacturing development phase of the program. We have not verified the DOD response. The F-22's engine has not been flight tested, but has experienced problems during ground tests. The F-22's engine is expected to be the first to provide the ability to fly faster than the speed of sound for an extended period of time without the high fuel consumption characteristic of aircraft that use afterburners to achieve supersonic speeds. It is expected to provide high performance and high fuel efficiency at slower speeds as well. Problems with performance of the F-22's engine first surfaced after the initial engine ground tests began in December 1992. The contractor is conducting a series of interim tests, with a goal of having a complete engine with a redesigned turbine and other changes qualified for flight by December 1996 if tests now planned for 1995 are successful. If not, F-22 flight tests will be started with an engine that is not fully representative of the current approved configuration. An Executive Independent Review Team was formed to provide advice on engine development issues, including a turbine problem. The team stated that it did not consider the nature and number of engine problems to be excessive for a highly sophisticated engine at this stage of development. They also stated that the proposed solutions can only be proven by exposing authentic hardware to the full range of realistic testing. Through November 1994, the Air Force had identified engine problems that may cost as much as $479 million to remedy. The Air Force increased the target cost of the engine development contract by $218 million to design and test solutions to the engine problems. The incorporation of corrective modifications to future production engines is expected to increase production costs by $123 million. The Air Force believes its current program estimate can cover the $341 million increase ($218 million plus $123 million), but the Air Force has identified other potential design changes that may add $138 million to development and production costs. The other potential design changes are not currently part of the planned program. The low observability or stealth characteristics of the F-22 is another risk area. The F-22 is to be the first supersonic, highly maneuverable fighter that uses low observable technologies to reduce radar, infrared, acoustic and optical signatures of the aircraft, making it difficult for an adversary to detect. An evaluation of the complete F-22 radar signature using computer models and a scale version of the aircraft concluded that the aircraft's radar signature did not meet the Air Force's operational requirement. Although DOD advised us that these problems were not considered major, design changes, such as reducing the number of aircraft maintenance access panels and fuel drain holes, and reshaping the airframe were evaluated through December 1994 to determine if these changes were successful in reducing the signature. DOD further stated that the contractual specifications are being revised. The estimated development cost to resolve these problems is about $20 million according to the F-22 program office. Additional production costs of about $110 million could also be required, however, program officials told us the total estimated cost ($71.6 billion) of the F-22 program should not be affected. The DSB, in its review of the F-22 program's concurrency and technical risks, identified a number of other concerns. Examples of concerns mentioned by the DSB include control of excess aircraft weight; use of new materials and fabrication processes; uncertain durability of composite materials in the F-22 application; probable inability of the engine to meet performance and durability goals design of certain low observable features and applicable manufacturing very challenging development of electronic warfare system; late scheduling of tests relative to increasing production to 12 aircraft a year; and the need for a long, evolutionary software development. Overall, the DSB characterized the F-22 program as very ambitious technically. We recommend that the Secretary of Defense reduce the degree of concurrency in the program because independent testing of technology advances (IOT&E), will not be completed before significant commitments are made to produce F-22s; the percentage of planned F-22s to be committed to production before completion of IOT&E is higher than most recent fighter programs; and the need for the F-22 is not urgent. To minimize commitments to production of F-22s until after successful completion of IOT&E, we recommend that the Secretary of Defense limit LRIP quantities to that which can be produced using the first set of hard tooling, about six to eight aircraft a year. DOD partially agreed with the findings in this report, but disagreed with the recommendations. DOD indicated it believed that the F-22 program had an acceptable degree of concurrency based on the DSB's evaluation that risks associated with premature entry into successively higher rates of production were readily controllable through insistence on meeting certain key events and test criteria already built into the F-22 plan. The record shows, however, that DOD has often been unwilling or unable to curtail production of other systems after it starts, despite discovery of significant problems in development or operational tests. We believe the degree of concurrency in the program should be addressed now, because (1) independent testing of important technology advances is not planned until after commitments are made to produce F-22s, (2) program concurrency is high according to DOD's prescribed measure, and (3) the need for the F-22 is not urgent. DOD disagreed with (1) our use of the completion of IOT&E as a measure of concurrency and risk in the program, (2) our positions on the level of risk in the F-22 program, and (3) the comparison of the F-22 to prior fighter programs using the percentage of planned aircraft procured during LRIP as a measurement. We first applied DOD's own guidance for measuring the degree of concurrency in the program, that is, the amount of IOT&E completed prior to entering productions. We also used other metrics, such as the percent of the total program committed to production before completion of IOT&E. Further, DOD's comments appear to discount the risks in the program identified by the DSB. Our comparison of the F-22 to recent fighter programs, although not the same as comparisons made by the DSB, provides an important historical perspective that DOD's planned LRIP meets or exceeds the fighter programs undertaken during the Cold War. In conducting our work, we obtained information and interviewed officials from the Office of the Secretary of Defense, DSB, and Air Force Headquarters, Washington, D.C.; F-22 System Program Office, Wright-Patterson Air Force Base, Ohio; the Air Force Air Combat Command, Langley Air Force Base, Virginia; and the Air Force Operational Test and Evaluation Center, Kirtland Air Force Base, New Mexico. We interviewed officials in charge of program management, the operation of tactical fighter aircraft, risk assessment and the operational testing of Air Force weapon systems. We reviewed documents, including program office briefings, program schedules, test plans and reports, technology risk assessments, requirements documents and cost reports. We used these interviews and documents to determine the program management philosophy, the amount of program concurrency, the planned flight testing of F-22 technologies, program technology requirements and program risk assessments. We also reviewed DOD instructions, Air Force regulations, Office of Secretary of Defense guidance, publications from the Defense Systems Management College, our prior reports, a report of another audit organization, congressional reports, an Institute for Defense Analyses report, a DSB report, a report prepared for the Defense Acquisition Board, executive summaries, and monthly program reports. In addition, we interviewed officials from the Air Force's Air Combat Command and examined the F-22 System Operational Requirements Document, Statements of Need, and the Mission Element Need Statement for new fighter aircraft. We performed our work from August 1994 through February 1995 in accordance with generally accepted government auditing standards. We are sending copies of this report to the Secretaries of Defense and the Air Force and the Director of the Office of Management and Budget. Copies will also be made available to others on request. Please contact me on (202) 512-4841 if you or your staff have any questions concerning this report. Major contributors to this report are listed in appendix II. The following are GAO's comments on the Department of Defense's (DOD) letter dated February 24, 1995. 1. These comments are dealt with on pages 12 and 13 of the report and in our responses to the DOD specific comments that follow. 2. For the most part, the risk/concurrency guidelines listed in the Office of the Secretary of Defense's April 1990 guide are specific requirements that should be met before a program progresses. We are aware of many of those requirements that are incorporated in the F-22 program. However, the only assessment provided for in the guide for measuring the degree of concurrency is the amount of initial operational testing and evaluation (IOT&E) completed at the time low-rate initial production (LRIP) begins. By that measure, the F-22 program clearly has a high degree of concurrency. In our opinion, the ramp up of production from 4 a year to 36 a year under the LRIP phase, and initiation of long lead for 48 a year essentially represents a plan to achieve a full-rate production schedule (now defined as 48 a year) before IOT&E is completed. 3. The F-22 program, as currently planned, schedules procurement of 80 LRIP aircraft at an estimated cost of $12.4 billion. We believe that exceeds the minimum needed to successfully complete the LRIP phase of the program and that the production rates should be restricted during LRIP. Although many important F-22 development tests are scheduled prior to the acceleration of production rates, many other critical developmental tests and most IOT&E testing are not scheduled to be complete until after significant commitments are made to production. 4. We adjusted the report to reflect this information. However, it should be noted that the total number of each type of aircraft produced was much higher than planned for the F-22. This results in a higher degree of concurrency in the F-22 program when using the percentage of aircraft procured at completion of IOT&E as a measure of concurrency. 5. Our report does not, either explicitly or implicitly, suggest "total avoidance" of concurrency. 6. The Defense Science Board (DSB) portrayal of the F-22 program as relatively conservative was based on the amount of development testing to be completed at early production decision points. However, using the measure called for by DOD's own 1990 guidance--the amount of IOT&E completed at the time LRIP begins--shows that the F-22 program is far from conservative. 7. Production ramp up from 4 aircraft a year to 36 aircraft a year appears to provide a more rapid acceleration than we believe is necessary in the LRIP phase of the program. In our opinion the ramp up of production from 4 a year to 36 a year under the LRIP phase, and initiation of long lead to support 48 a year, essentially represents a plan to achieve a full-rate production schedule before IOT&E is completed. 8. This material has been deleted from the report. 9. Additional information concerning this matter has been added to the body of the report. 10. DOD response to our prior report on embedded computers has been recognized in the body of this report. 11. We did not attempt to quantify potential cost growth in the F-22 program that may result from a change in the program schedule. However, the thrust of the LRIP legislation is to authorize only minimum necessary quantities. DOD acquisition profiles created for other weapon programs have often proven to be optimistic and are rarely carried out as initially planned because of technical, financial, or test problems. If the baseline against which to compare potential growth of costs is optimistic, an estimate of cost growth would have limited meaning at this point because those problems are likely to occur in highly concurrent programs that involve substantial advances in technology. Brenda Waterfield, Evaluator The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (301) 258-4097 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.

Summary:
Pursuant to a legislative requirement, GAO assessed the concurrency between the development and production phases of the Air Force's F-22 fighter program and the risks associated with that concurrency. GAO found that: (1) the F-22 program has a high degree of concurrency because it will enter production before initial operational testing and evaluation (IOT&E) is completed; (2) F-22 concurrency poses substantial production and operational risks because the aircraft may be procured before technological advances are flight-tested; (3) the Air Force plans to procure 80 F-22 aircraft at a cost of $12.4 billion before completing IOT&E; (4) the F-22 low-rate initial production (LRIP) quantities substantially exceed the 10-percent guideline included in federal acquisition streamlining requirements; (5) the percentage of F-22 committed to production before IOT&E is higher than most recent fighter programs; (6) the Air Force plans to accelerate F-22 production rates in the LRIP phase of the program so that 75 percent of the full production rate will be achieved; (7) the planned rate of acceleration appears to exceed the amount that is needed to complete the program's LRIP phase and represents a plan to commit to a full-rate production schedule before IOT&E is completed; (8) the Air Force should limit LRIP quantities each year given the program's high degree of concurrency; (9) technology advances and innovations are critical to F-22 operational success; (10) the need for F-22 aircraft is not urgent and its procurement could be deferred; and (11) existing operational and technological problems need to be addressed before significant commitments are made to F-22 production.