Boeing and TRW disclosed the key results and limitations of Integrated Flight Test 1A in written reports released between August 13, 1997, and April 1, 1998. The contractors explained in a report issued 60 days after the June 1997 test that the test achieved its primary objectives, but that some sensor abnormalities were noted. For example, while the report explained that the sensor detected the deployed targets and collected some usable target signals, the report also stated that some sensor components did not operate as desired and the sensor often detected targets where there were none. In December 1997, the contractors documented other test anomalies. According to briefing charts prepared for a December meeting, the Boeing sensor tested in Integrated Flight Test 1A had a low probability of detection; the sensor’s software was not always confident that it had correctly identified some target objects; the software significantly increased the rank of one target object toward the end of the flight; and in-flight calibration of the sensor was inconsistent. Additionally, on April 1, 1998, the contractors submitted an addendum to an earlier report that noted two more problems. In this addendum, the contractors disclosed that their claim that TRW’s software successfully distinguished a mock warhead from decoys during a post-flight analysis was based on tests of the software using about one-third of the target signals collected during Integrated Flight Test 1A. The contractors also noted that TRW reduced the software’s reference data so that it would correspond to the collected target signals being analyzed. Project office and Nichols Research officials said that in late August 1997, the contractors orally communicated to them all problems and limitations that were subsequently described in the December 1997 briefing and the April 1998 addendum. However, neither project officials nor contractors could provide us with documentation of these communications. Although the contractors reported the test’s key results and limitations, they described the results using some terms that were not defined. For example, one written report characterized the test as a “success” and the sensor’s performance as “excellent.” We found that the information in the contractors’ reports, in total, enabled officials in the Ground Based Interceptor Project Management Office and Nichols Research to understand the key results and limitations of the test. However, because such terms are qualitative and subjective rather than quantitative and objective, their use increased the likelihood that test results would be interpreted in different ways and might even be misunderstood. As part of our ongoing review of missile defense testing, we are examining the need for improvements in test reporting. Appendix I provides details on the test and the information disclosed. The Ground Based Interceptor Project Management Office relied on an on- site engineer and Nichols Research Corporation to provide insight into Boeing’s work. The project office also relied on Boeing to oversee the performance of its subcontractor, TRW. Oversight was limited by the ongoing competition between Boeing and another contractor competing for the exoatmospheric kill vehicle contract because the Ground Based Interceptor Project Management Office and its support contractors had to be careful not to affect competition by assisting one contractor more than another. Project officials said that they relied more on “insight” into the contractors’ work rather than oversight of that work. Nichols gained program insight by attending technical meetings, assessing test reports, and sometimes evaluating technologies proposed by Boeing and TRW. For more information on how the project office exercised oversight over its contractors’ technical performance, see appendix II. Boeing and TRW reported that post-flight testing and analysis of data collected during Integrated Flight Test 1A showed that deployed target objects displayed distinguishable features when observed by an infrared sensor. The contractors reported the test also showed that Boeing’s exoatmospheric kill vehicle sensor could collect target signals from which TRW’s software could extract distinguishable features and that the software could identify the mock warhead from other objects by comparing the extracted features to the features that it had been told to expect each object to display. However, there has been no independent verification of these claims. We talked with Dr. Mike Munn, who was, during the 1980s, the Chief Scientist for missile defense programs at Lockheed Missiles and Space Company. He agreed that a warhead and decoys deployed in the exoatmosphere likely display distinguishable differences in the infrared spectrum. However, the differences may not be fully understood or there may not presently be methods to predict the differences. Dr. Munn added that the key was in the ability to make both accurate and precise measurements and also to predict signatures accurately. He emphasized that robust discrimination depends on the ability to predict signatures and then to match in-space measurements with those predictions. The Phase One Engineering Team and Nichols Research Corporation have noted that TRW's software used prior knowledge of warhead and decoy differences, to the maximum extent available, to discriminate one object from the other and cautioned such knowledge may not always be available in the real world. National Missile Defense program officials said that after considerable debate among themselves and contractors, the program manager reduced the number of decoys planned for intercept flight tests in response to a recommendation by an independent panel, known as the Welch Panel.The panel, established to reduce risk in ballistic missile defense flight test programs, viewed a successful hit-to-kill engagement as a difficult task that should not be further complicated in early tests by the addition of decoys. After contemplating the advice of the Welch panel and considering the opinions of program officials and contractors who disagreed over the number and complexity of decoys that should be deployed in future tests, the program manager decided that early tests should include only one decoy, a large balloon. See appendix III for more information on the reduction of decoys in later tests. The Phase One Engineering Team was tasked by the National Missile Defense Joint Program Office to assess the performance of TRW’s software and to complete the assessment within 2 months using available data. The team's methodology included determining if TRW’s software was based on sound mathematical, engineering, and scientific principles and testing the software’s critical modules using data from Integrated Flight Test 1A. The team reported that although the software had weaknesses, it was well designed and worked properly, with only some changes needed to increase the robustness of the discrimination function. Further, the team reported that the results of its test of the software using Integrated Flight Test 1A data produced essentially the same results as those reported by TRW. Based on its analysis, team members predicted that the software would perform successfully in a future intercept test if target objects deployed as expected. Because the Phase One Engineering Team did not process the raw data from Integrated Flight Test 1A or develop its own reference data, the team cannot be said to have definitively proved or disproved TRW’s claim that its software successfully discriminated the mock warhead from decoys using data collected from Integrated Flight Test 1A. A team member told us its use of Boeing- and TRW-provided data was appropriate because the former TRW employee had not alleged that the contractors tampered with the raw test data or used inappropriate reference data. Appendix IV provides additional details on the Phase One Engineering Team evaluation. In commenting on a draft of this report, the Department of Defense concurred with our findings. It also suggested technical changes, which we incorporated as appropriate. The Department's comments are reprinted in appendix VII. We conducted our review from August 2000 through February 2002 in accordance with generally accepted government auditing standards. Appendix VI provides details on our scope and methodology. The National Missile Defense Joint Program Office’s process for releasing documents significantly slowed our work. For example, the program office took approximately 4 months to release key documents such as the Phase One Engineering Team’s response to the professor’s allegations. We requested these and other documents on September 14, 2000, and received them on January 9, 2001. As arranged with your staff, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from its issue date. At that time, we plan to provide copies of this report to the Chairmen and Ranking Minority Members of the Senate Committee on Armed Services; the Senate Committee on Appropriations, Subcommittee on Defense; the House Committee on Armed Services; and the House Committee on Appropriations, Subcommittee on Defense; and the Secretary of Defense; and the Director, Missile Defense Agency. We will make copies available to others upon request. If you or your staff have any questions concerning this report, please contact Bob Levin, Director, Acquisition and Sourcing Management, on (202) 512-4841; Jack Brock, Managing Director, on (202) 512-4841; or Keith Rhodes, Chief Technologist, on (202) 512-6412. Major contributors to this report are listed in appendix VIII. Boeing and TRW disclosed the key results and limitations of an early sensor flight test, known as Integrated Flight Test 1A, to the Ground Based Interceptor Project Management Office. The contractors included some key results and limitations in written reports submitted soon after the June 1997 test, but others were not included in written reports until December 1997 or April 1998. However, according to project office and Nichols officials, all problems and limitations included in the written reports were communicated orally to the project management office in late August 1997. The deputy project office manager said his office did not report these verbal communications to others within the Program Office or the Department of Defense because the project office was the office within the Department responsible for the Boeing contract. One problem that was included in initial reports to program officials was a malfunctioning cooling mechanism that did not lower the sensor’s temperature to the desired level. Boeing characterized the mechanism’s performance as somewhat below expectations but functioning well enough for the sensor’s operation. We hired experts to determine the extent to which the problem could affect the sensor’s performance. The experts found that the cooling problem degraded the sensor’s performance in a number of ways, but would not likely result in extreme performance degradation.