In 1983, Congress established DOT&E to coordinate, monitor, and evaluate operational testing of major weapon systems. As part of the Office of the Secretary of Defense (OSD), DOT&E is separate from the acquisition community that conducts developmental and operational testing and therefore is in a position to provide the Secretary and Congress with an independent view. Congress created DOT&E in response to reports of conflicts of interest in the acquisition community’s oversight of operational testing leading to inadequate testing of operational suitabilityand effectiveness and the fielding of new systems that performed poorly. (DOD’s system acquisition process is described in app. III.) By law, DOT&E serves as the principal adviser on operational test and evaluation in DOD and bears several key responsibilities, including monitoring and reviewing all operational test and evaluation in DOD, reporting to the Secretary of Defense and congressional committees whether the tests and evaluations of weapon systems were adequate and whether the results confirmed that the system is operationally suitable and effective for combat before a decision is made to proceed to full-rate production, and submitting to the Secretary of Defense and congressional decisionmakers an annual report summarizing operational test and evaluation activities during the preceding fiscal year. In 1993, DOD’s advisory panel on streamlining and codifying acquisition laws concluded that DOT&E was impeding the goals of acquisition reform by (1) promoting unnecessary oversight, (2) requiring excessive reporting detail, (3) inhibiting the services’ discretion in testing, and (4) limiting participation of system contractors in operational tests where such involvement is deemed necessary by the services. The following year, DOD proposed legislative changes that would have reduced the scope and authority of DOT&E. In testimony, we opposed these changes because they were directed at perceived rather than documented problems and would undermine a key management control over the acquisition process—independent oversight of operational test and evaluation. Although the legislative proposals were not adopted, in 1995 the Secretary of Defense implemented several operational test and evaluation initiatives in the Department to (1) involve operational testers earlier in the acquisition process, (2) use models and simulations effectively, (3) combine tests where possible, and (4) combine tests and training. The goals of these initiatives included saving time and money by identifying and addressing testing issues earlier in the acquisition process; merging or closely coordinating historically distinct phases, such as developmental and operational testing to avoid duplication; and using existing technologies and training exercises to create realistic and affordable test conditions. A frequent complaint among representatives of the services’ operational testing agencies was that DOT&E frequently demanded more tests than were proposed by the operational test agencies in draft master plans or test plans. Operational test agency representatives contended that the additional testing was either unnecessary for determining the operational effectiveness or suitability of a program or unrealistic in light of the limitations in the services’ testing resources. However, our review indicated that DOT&E urged more testing to reduce the level of risk and number of unknowns prior to the decision to begin full production, while program and service officials typically sought less testing and were willing to accept greater risk when making production decisions. The additional testing DOT&E advocated, often over the objections of service testers, served to meet the underlying objectives of operational testing—to reduce the uncertainty and risk that systems entering full-rate production would not fulfill their requirements. The impact of DOT&E oversight varies with the system under development. Table 1 summarizes the types of impacts that DOT&E advocated or facilitated in operational testing among the 13 cases we studied. While the impacts vary, one consistent pattern in our case studies was a reduction in uncertainty regarding the weapon systems’ suitability or effectiveness prior to the full-rate production decision. Each of the impacts are discussed in more detail in tables 2-6 and in subsequent sections. Two of DOT&E’s typical concerns in reviewing service test plans are that the proposed test methodologies enable (1) comparisons of a system’s effectiveness through side-by-side testing between the existing and modified systems and (2) assessments of a system’s reliability through a sufficient number of test repetitions. Table 2 illustrates examples of cases where additional testing was conducted at DOT&E’s insistence or with DOT&E’s support to alleviate these and other types of effectiveness and suitability concerns. Table 3 illustrates examples where the design or conduct of operational testing was modified at DOT&E’s insistence or with DOT&E’s support to increase the realism of test conditions and reduce the uncertainty of system suitability or effectiveness. DOT&E can insist on or support changes in data analysis plans that provide more meaningful analyses for decisionmakers. Table 4 illustrates instances in which DOT&E altered the proposed data collection or analysis plans to enhance the reliability or utility of the test data. DOT&E’s independent analysis of service test data may confirm or dispute the results and conclusions reported by the service. In the cases described in table 5, DOT&E’s analysis of service operational test and evaluation data resulted in divergent, often less favorable conclusions than those reached by the service. When DOT&E concludes that a weapon system has not fully demonstrated operational suitability or effectiveness, or if new testing issues arise during initial operational test and evaluation, it may recommend that follow-on operational test and evaluation be done after the full-rate production decision. Table 6 identifies follow-on operational test and evaluation that DOT&E supported. The existence of a healthy difference of opinion between DOT&E and the acquisition community is a viable sign of robust oversight. In nearly all of the cases we reviewed, the services and DOT&E cited at least one testing controversy. For example, services differ on how they view the relationship between operational testing and their development of tactics, techniques, and procedures. In addition, DOT&E’s ability to independently view the development and testing of new systems across the services brings value to the context of testing. However, several current trends have the potential to adversely affect DOT&E’s independence and its ability to affect operational test and evaluation, including (1) service challenges to DOT&E’s authority to require and oversee follow-on operational testing and evaluation, (2) declining resources available for oversight, (3) the management of limited resources to address competing priorities, (4) DOT&E’s participation in the acquisition process as a member of the program manager’s working-level integrated product teams, and (5) greater integration of developmental and operational testing. DOT&E’s impact on operational testing is dependent upon its ability to manage these divergent forces while maintaining its independence. Although the acquisition community has three central objectives—performance, cost, and schedule—DOT&E has but one: operational testing of performance. These distinct priorities lead to testing disputes. Characteristically, the disputes for each system we reviewed revolved around questions of how, how much, and when to conduct operational testing, not whether to conduct operational testing. Conflicts encompassed issues such as (1) how many and what types of tests to conduct; (2) when testing should occur; (3) what data to collect, how to collect it, and how best to analyze it; and (4) what conclusions were supportable, given the analysis and limitations of the test program. The foundation of most disputes lay in different notions of the costs and benefits of testing and the levels of risk that were acceptable when making full-rate production decisions. DOT&E consistently urged more testing (and consequently more time, resources, and cost) to reduce the level of risk and number of unknowns before the decision to proceed to full-rate production, while the services consistently sought less testing and accepted more risk when making production decisions. Among our case studies, these divergent dispositions frequently led to healthy debates about the optimal test program, and in a small number of cases, the differences led to contentious working relations. In reviews of individual weapon systems, we have consistently found that testing and evaluation is generally viewed by the acquisition community as a requirement imposed by outsiders rather than a management tool to identify, evaluate, and reduce risks, and therefore a means to more successful programs. Developers are frustrated by the delays and expense imposed on their programs by what they perceive as overzealous testers. The program office strives to get the program into production despite uncertainties that the system will work as promised or intended. Therefore, reducing troublesome parts of the acquisition process—such as operational testing—is viewed as a means to reduce the time required to enter production. Nonetheless, the commanders and action officers within the service operational test agencies were nearly unanimous in their support for an independent test and evaluation office within OSD. For example, the Commander of the Army’s Operational Test and Evaluation Command commended the style and orientation of the current DOT&E Director and affirmed the long-term importance of the office and its independent reporting responsibilities to Congress. The Commander of the Navy’s Operational Test and Evaluation Force stated that the independence of both DOT&E and the operational test agency was an essential element in achieving their common goal of ensuring that new programs pass sufficiently rigorous and realistic operational testing prior to fielding. The Commander of the Air Force’s Operational Test and Evaluation, while critical of DOT&E oversight of several major weapon systems, said that the services were well served by DOT&E’s potential to independently report to Congress. Moreover, nearly all the operational test agency action officers we interviewed participate in the integrated product teams with the DOT&E action officers and recognized the value of the Office’s independent oversight role. The action officers within the service testing organizations also have a degree of independence that enables them to represent the future users of systems developed in the acquisition community. These action officers stated that their ability to voice positions unpopular with the acquisition community was strengthened when DOT&E separately supported their views. In discussions with over three dozen action officers and analysts responsible for the 13 cases we reviewed, the independence of DOT&E emerged as the fundamental condition to enable effective and efficient oversight.