Joint STARS is a joint Air Force and Army wide-area surveillance and target attack radar system designed to detect, track, classify, and support the attack of moving and stationary ground targets. This $11 billion major defense acquisition program consists of air and ground segments— refurbished 707 aircraft (designated the E-8) equipped with radar, operation and control, data processing, and communications subsystems, together with ground stations equipped with communications and data processing subsystems. Low-rate initial production (LRIP) of the Joint STARS aircraft began in fiscal year 1993. In line with 10 U.S.C. 2399, DOD’s final decision to proceed beyond LRIP first required the DOD Director of Operational Test and Evaluation (DOT&E) to submit a report to Congress, referred to as the Beyond LRIP report, stating whether (1) the test and evaluation performed was adequate and (2) testing demonstrated that the system is effective and suitable for combat, that is, operationally effective and suitable. The Joint STARS aircraft was scheduled to begin its initial operational test and evaluation—referred to as the Joint STARS multi-service operational test and evaluation—in November 1995. That testing was delayed and then changed because of the deployment of Joint STARS assets to the European theater to support Operation Joint Endeavor in Bosnia. The Air Force Operational Test and Evaluation Center (AFOTEC) and the U.S. Army Operational Test and Evaluation Command conducted a combined development and operational test of Joint STARS from July through September 1995 and an operational evaluation of the system during Operation Joint Endeavor from January through March 1996. Two Air Force Joint STARS aircraft and 13 Army Joint STARS ground station modules were deployed to support Operation Joint Endeavor and operationally evaluated from January through March 1996. After analyzing the data from the combined development and operational test and the operational evaluation performed during Operation Joint Endeavor, AFOTEC issued its Joint STARS multi-service operational test and evaluation final report on June 14, 1996. DOT&E staff analyzed the same and additional data and the Director issued his Beyond LRIP report to Congress on September 20, 1996. On September 25, 1996, the Under Secretary of Defense for Acquisition and Technology signed an acquisition decision memorandum approving the Joint STARS program’s entry into full-rate production with a total planned quantity of 19 aircraft. LRIP of the Joint STARS aircraft began in fiscal year 1993. By statute, 10 U.S.C. 2399, the “Secretary of Defense shall provide that a major defense acquisition program may not proceed beyond low-rate initial production until initial operational test and evaluation of the program is completed.” Operational test and evaluation is the primary means of assessing weapon system performance in a combat-representative environment. It is defined as the (1) field test, conducted under realistic combat conditions, to determine an item’s effectiveness and suitability for use in combat by typical military users and (2) evaluation of the results of such a test. If used effectively, operational test and evaluation is a key internal control measure to ensure that decisionmakers have objective information available on a weapon system’s performance, thereby minimizing risks of procuring costly and ineffective systems. Joint STARS was moved from low-rate to full-rate production even though (1) it performed poorly during both the combined development and operational test and the operational evaluation in Bosnia, (2) excessive contractor effort was needed to support Operation Joint Endeavor, (3) the suitability and sustainability of the system is questionable since it uses refurbished 25-30 year old airframes, and (4) operational software is considered significantly immature. In DOT&E’s Beyond LRIP report, the DOT&E stated that Joint STARS had only demonstrated effectiveness for operations other than war. The report indicated that of three critical operational issues to judge effectiveness, only one had been demonstrated as met “. . . with limitations.” Those critical operational issues related to (1) performance of the tactical battlefield surveillance mission, that is, surveillance—“met with limitations”; (2) support of the execution of attacks against detected targets, that is, target attack support; and (3) the provision of information to support battlefield management and target selection, that is, battle management. The effectiveness critical operational issues were judged based on seven supporting measures. In its report to Congress, DOT&E listed four of those measures of effectiveness as “not met” during the system’s combined development and operational test and did not list any as having been demonstrated during the Operation Joint Endeavor operational evaluation. “In the current configuration, the [Joint STARS] aircraft has not demonstrated the ability to operate at the required maximum altitude; adequate tactics, techniques, or procedures to integrate [Joint STARS] into operational theaters have not been developed; [Joint STARS] exceeded the break rate and failed the mission reliability rate during [Operation Joint Endeavor]. During , [Joint STARS] did not achieve the effective time-on-station requirement.” He concluded that without corrective actions, “[Joint STARS] would not be suitable in higher intensity conflict” and later in the report judged that the system “as tested is unsuitable.” Analysis of DOT&E’s Beyond LRIP report indicates that not only did Joint STARS have disappointing test results but also that extensive follow-on operational testing of Joint STARS is needed. In its Beyond LRIP report, DOT&E presented a table that reported its findings of the combined development and operational test and Joint STARS Operation Joint Endeavor operational evaluation and indicates where further testing is required. Our analysis of that table indicates that at most only 25 of 71 test criteria could be judged met. DOT&E considers 18 of those 25 to require no further testing, that is, DOT&E judges them clearly met. However, our analysis also indicates that 19 test criteria were clearly not met and that as many as 26 might not have been met. Twenty-seven of the criteria could not be determined in either the combined development and operational test or the Operational Joint Endeavor operational evaluation. Of the 71 Joint STARS operational test and evaluation criteria listed, DOT&E indicates that 53, or about 75 percent, require further testing. In addition to the above, DOT&E also noted that there were several operational features present during Joint STARS Operation Joint Endeavor deployment that were essential to its mission accomplishment but were not included in the recent production decision. It provided two specific examples—satellite communications and a deployable ground support station. DOT&E believes these features “will be a necessary part of the production decision to achieve a capable [Joint STARS] system.” It also noted the need for other features—moving target indicator clutter suppression, communications improvements, terrain masking tools for ground station module operators, and linkage to operational theater intelligence networks. Since at least two of the features present during Operation Joint Endeavor were “essential” to its mission accomplishment have already been developed, and may be needed “to achieve a capable Joint STARS system,” those features should also be tested during the planned Joint STARS follow-on test and evaluation. “ must yield the most credible and objective results possible. All facets of the test effort must operate under the rules that support total objectivity and prevents improper data manipulation.” The test plan also states that interim contractor support “will be limited to perform ground maintenance only; no in-flight support.” Regarding the Army’s ground station modules, it states that “the Army maintenance concept does not call for at any level . . . .” “Approximately 80 contractors were deployed to support the E-8C. However, three or four systems engineers flew on each flight to ensure they could provide system stability and troubleshooting expertise during missions. Additionally, three or four software engineers were on the ground full time, researching and developing fixes to software problems identified during the deployment.” AFOTEC also reported that “Each of the had one contractor representative on site and on call with additional help available as necessary. Five contractor representatives remained at [Rhein-Main Air Base] and functioned as a depot.” The AFOTEC report stated that the “test director agreed to contractor participation in the to a greater extent permitted under US Public Law, Title 10, Section 2399.” When we formally expressed our concerns about the significant contractor involvement in Operation Joint Endeavor, DOD did not directly acknowledge that contractors were utilized beyond the constraints of the law governing operational test and evaluations. It stated that “were this solely an , contractors would not have been utilized beyond the constraints of 10 U.S.C. §2399,” and noted that the contractors were involved in the Joint STARS operation to support the mission. It further stated that employing Joint STARS in Operation Joint Endeavor “allowed the system to be operated and tested at a greater operational tempo than the system would have undergone in traditional testing.” DOD also stated that “because of the developmental nature of the aircraft, we needed to have more contractor personnel involved than we would otherwise have had.” It is understandable that DOD wanted to provide the best support possible in Operation Joint Endeavor. However, such significant contractor use neither supports a conclusion that the system is operationally effective or suitable for combat, nor is it indicative of a level of system maturity that justifies full-rate production. Joint STARS failure to meet its maintainability criteria during an operation less demanding than combat, even with such significant contractor involvement beyond that planned for in combat, also raises the question of the Air Force’s ability to develop a cost-effective maintenance plan for the system. This issue is recognized in the Under Secretary’s acquisition decision memorandum approving Joint STARS full-rate production. In that memorandum, the Under Secretary called for the Air Force to fully examine Joint STARS affordability, sustainability, and life-cycle costs, including the scope of contractor support.