Test and evaluation activities are an integral part of developing and producing weapon systems, as they provide knowledge of a system’s capabilities and limitations as it matures and is eventually delivered for use by the warfighter. DOD divides its testing activities into three categories: developmental, operational, and live fire. Developmental testing, which is conducted by contractors, university and government labs, and various DOD organizations, is intended to provide feedback on the progress of a system’s design process and its combat capability as it advances toward initial production or deployment. Operational test and evaluation is intended to evaluate a system’s effectiveness and suitability under realistic combat conditions before full-rate production or deployment occurs. DOD defines operational effectiveness as the overall degree of mission accomplishment of a system when used by representative personnel (e.g., warfighters) in the environment planned or expected for operational employment of the system considering organization, training, doctrine, tactics, survivability or operational security, vulnerability, and threat. Operational suitability defines the degree in which a system can be satisfactorily placed in field use, with consideration given to its reliability, transportability, interoperability, and safety, among other attributes. Under live fire test and evaluation, when applicable, survivability is a measure of a system’s vulnerability to munitions likely to be encountered in combat and lethality measures a system’s ability to combat intended targets. Operational testing is managed by the various military test organizations representing the customers, such as combat units that will use the weapons. Each of the four military services—Air Force, Army, Marine Corps, and Navy—has its own operational test agency to plan and execute testing. In 1983, Congress established DOT&E to coordinate, monitor, and evaluate operational testing of major weapon systems. As part of the Office of the Secretary of Defense, DOT&E is separate from the acquisition and test communities within the military services and other defense agencies. This enables DOT&E to provide the Secretary of Defense and Congress with an independent perspective on operational or live fire testing activities and results for DOD acquisition programs. DOT&E serves as the principal adviser on operational test and evaluation in DOD and bears several key responsibilities, which include: providing policy and guidance to DOD leadership on operational and live fire test and evaluation; monitoring and reviewing all operational and live fire test and approving test and evaluation master plans (TEMPs), operational test plans, and live fire test plans for all programs receiving oversight; reporting to the Secretary of Defense and congressional defense committees on programs generally before a full-rate production decision regarding (1) the adequacy of operational and live fire test and evaluation, and (2) operational effectiveness and suitability for combat; and reporting annually to the Secretary of Defense and Congress on all operational and live fire test and evaluation results from the preceding fiscal year. By law, DOT&E is responsible for overseeing all major defense acquisition programs, as well as any other acquisition programs it determines should be designated for oversight. The number of programs receiving operational test and evaluation oversight from DOT&E increased considerably between 2005 and 2010, from 279 to 348 programs, but since that time has declined, averaging 315 programs per year. Programs under DOT&E live fire test and evaluation oversight increased in a similar way over the past decade, averaging about 121 programs per year since 2011. DOT&E may oversee operational testing or live fire testing or both, depending on the circumstances of each program. Figure 1 shows the number of programs receiving DOT&E oversight annually over the last decade. Generally, programs are added to the oversight list when they formally enter the acquisition process, and DOT&E oversight continues through key acquisition milestones to full-rate production approval. Figure 2 illustrates the acquisition process, test phases, and DOT&E’s involvement in oversight. During their planning phase, DOD acquisition programs establish a test and evaluation working integrated product team that is made up of acquisition and test stakeholders, including DOT&E representatives. The main focus of this team is developing the TEMP, which provides a framework and plan for what developmental and operational testing will be conducted, as well as test resources needed, and how the major test events and test phases for a program link together. The TEMP also identifies criteria to be used to test and evaluate the system. The TEMP is required for key program milestone reviews, such as Milestone B and Milestone C, and as required by DOD policy, must be approved by DOT&E and several other DOD and military service organizations. During the later stages of system development (engineering and manufacturing development phase) and before a program has production units of its system available for testing, one or more operational assessments may be conducted. These assessments usually are completed by the designated operational test agency in accordance with a test plan approved by DOT&E, and the results can be used to inform the Milestone C initial production decision for a program. Following Milestone C, programs develop an operational test plan to support the initial operational test and evaluation (IOT&E) of the system. This test plan is expected to provide a detailed scope and methodology for conducting operational and live fire test events. By law, DOT&E approval of the operational test plan is required for programs on the oversight list, and the operational test agency conducts testing in accordance with the DOT&E-approved plan. Representative users (e.g., the warfighters) and production-representative units (e.g., systems from low-rate initial production) are used for IOT&E to determine if a system is operationally effective and operationally suitable for its mission. DOT&E formally monitors the IOT&E event, and reports the results of its evaluation to the Secretary of Defense and Congress, as required, in support of a full-rate production decision. Different objectives and incentives exist between the acquisition and testing communities, which can potentially fuel tension over what is needed to accomplish operational testing for programs. According to DOD officials, differences usually are resolved in a reasonable and timely manner, with modest adjustments often required in the course of developing and executing a test approach. However, sometimes acquisition and test officials have differences about operational testing requirements, methods, costs, or results that develop into significant disputes that are more difficult to resolve. Acquisition and test officials in the military services identified a small number of cases where significant disputes occurred among programs receiving DOT&E oversight during fiscal years 2010 through 2014. Of the 454 programs on DOT&E’s oversight list during that period, officials identified 42 programs—less than 10 percent—that had significant operational test disputes with DOT&E.According to military service and DOT&E officials, while these disputes can require additional time and effort to work through, they generally do get resolved. Acquisition and test officials from the military services stated that, in general, DOT&E’s execution of its oversight authorities provided valuable input and support to acquisition programs and to the entire DOD operational test enterprise. However, at times in the past, some elements of the DOD acquisition community, such as program offices, program executive offices, or senior acquisition executive offices, have expressed concerns that the test community’s approach to testing imposes undue requirements on programs. In response to concerns like these, in 2011 the Under Secretary of Defense for Acquisition, Technology, and Logistics (USD (AT&L)) chartered an independent assessment of the developmental and operational test communities’ approach to testing. The assessment found no significant evidence that the testing community typically drives unplanned requirements, cost, or schedule into The assessment, however, acknowledged that tension exists programs.between programs and the test community, but noted it can be mitigated through early and objective communication of issues, and when necessary, through involving senior DOD leadership. Tension exists between the acquisition and testing communities, in part, because they have somewhat different objectives and perspectives regarding the role of operational testing. Acquisition managers are motivated largely by their programs’ cost, schedule, and performance objectives, particularly once they have an approved program baseline that formalizes those objectives for system development. DOT&E is focused on ensuring the operational effectiveness and suitability of systems is adequately evaluated. As program managers work to preserve their program goals, they may see test reductions or delays as a reasonable option for offsetting cost and schedule growth encountered during system development. We have previously found that compromises in test approaches and resources are more readily made in deference to other priorities, such as preserving program cost goals. Postponing difficult tests or limiting open communication about test results can help a program avoid unwanted scrutiny because tests against criteria can reveal shortfalls, which may call into question whether a program should proceed as planned. DOT&E’s approval authority for core test documents—particularly TEMPs and operational test plans—also contributes to the tension. Approval can be withheld until these documents demonstrate adequate means to evaluate operational effectiveness and suitability. TEMPs must also show that sufficient resources have been dedicated to the operational test program. Costs for operational tests are predominantly borne by the programs, which creates another source of tension. Although operational testing typically represents a relatively small amount of the total program cost to develop and produce a system, this cost can be significant in the years during which operational testing events occur. In 2011, DOT&E assessed 78 recent acquisition programs and found the average marginal cost of operational test and evaluation to be about 1 percent of total program acquisition costs. and contract flexibility to accommodate discovery and respond to changes in testing needs during program execution. Aside from the cost, the fact that operational testing occurs largely in the later stages of a program—when overall research, development, test, and evaluation funds generally are more limited—may also create significant challenges if test changes are needed. This may be particularly true when production has already begun. In those cases, additional costs can stem both from increased testing and from production delays, as well as any potential retrofitting required for the systems already produced.