DOD defines software maintenance and software sustainment synonymously, to comprise any activities or actions that change the software baseline, as well as modifications or upgrades that add capability or functionality. For example, software sustainment activities involve the correction of software errors after the software is released and adaptations to enable interfacing with changing environments. The four categories of software sustainment actions are defined in figure 1 below. A software sustainment activity can be categorized in multiple areas. For example, an Army command is modifying software to incorporate Windows 10. This action may be described as corrective in that it addresses errors in previous versions of Windows; perfective in that it upgrades the software to support new capabilities and functionality provided by Windows 10; adaptive in that it can accommodate changes to firmware and hardware environments; and preventive in that it improves reliability. Sustaining software is normally different from sustaining hardware. For example, when hardware breaks, technicians can remove the broken part—such as tread on a tracked vehicle— and install a working part. In contrast, sustaining software typically requires writing, testing, and deploying lines of code. Software provides critical functionality to nearly every hardware system that DOD uses: surface (for example, mobile network systems); air (for example, secure communications arrays in aircraft); sea (for example, submarine guidance systems); missile (for example, targeting systems); ordnance (for example, Common Remotely Operated Weapon Station); and space (for example, positioning software), as shown in figure 2. Further, a weapon system may comprise numerous software systems, each supporting different components of the system. Hundreds, or even thousands, of software systems can be embedded in a single weapon system. Interoperability and integration within the weapon system as a whole constitute key software considerations for the overall weapon system’s sustainability. For example, the military departments include system-of-systems and family-of-systems considerations. These considerations are defined as a set or arrangement of systems that results when independent systems are integrated within a larger system that delivers unique capabilities. Missions are performed by a system-of- systems arrangement of the platforms and systems that deliver the mission capability. Decisions affecting the software on a weapon system are made throughout the acquisition life-cycle. The life-cycle is outlined in DOD Instruction 5000.02, Operation of the Defense Acquisition System. This instruction includes four basic and two hybrid models that serve as examples of defense program structures. The hybrid models combine models, such as a weapon system development that includes significant software development. The instruction also includes phases and milestones to oversee and manage acquisition programs, including major weapon systems. It outlines considerations affecting software sustainment for each milestone, including, for example, the following: Milestone A: The understanding of the technical, cost, and schedule risks of acquiring the materiel solution; the determination of core requirements; and the development of an intellectual property strategy, to include technical data and computer software deliverables. For example, for incrementally deployed software- intensive programs, the preliminary scope of limited deployment is determined for evaluation prior to a full deployment decision for each capability increment. Milestone B: A standard series of design reviews performed prior to converging on a final design for production. For example, for a hybrid acquisition program such as the combination of a major weapon system’s basic structural hardware development with a simultaneous software-intensive development, criteria establishing maturity for the development of software functional capability are to be identified. Milestone C: The point at which a program or increment of capability is reviewed for entrance into the production and deployment phase or for limited deployment. For example, a general criterion applied during review would be to have a mature software capability consistent with the software development schedule. Figure 3 depicts the milestones and decision points that inform a typical acquisition program. Decisions affecting the software of a weapon system are made throughout the acquisition life- cycle and involve stakeholders across a number of domains. For example, DOD officials are involved in software development, architecture and design, engineering, coding, integration and testing, cost estimation and collection, and intellectual property. Many decisions affecting software sustainment, such as software data rights decisions, typically occur in one of the phases prior to operations and support. Decisions made in the early phases may have long-term effects on a weapon system’s sustainability, especially for systems that endure beyond their originally intended design life. Software sustainment decisions are often revisited during the operations and support phase, as hardware breaks or needs to be replaced, a new capability or requirement is added, or a modification is made due to feedback received after a weapon system is fielded. DOD conducts software sustainment at a variety of depot-level maintenance locations. DOD and military policy refer to these locations variously as DOD depot-level software sustainment activities, Software Engineering Centers, Software Support Activities, and Life-Cycle Software Engineering Centers. For purposes of this report, we will refer to these facilities as DOD software centers. Section 2460 of title 10 of the United States Code defines depot-level maintenance and repair. This term includes all aspects of software maintenance classified by DOD as of July 1, 1995, as depot-level maintenance and repair—regardless of the source of funds for the maintenance or repair, or of the location at which the maintenance or repair is performed. DOD maintains many weapon systems (such as aircraft and ships) and equipment (such as radar) at the depot level because the systems are too complex to maintain exclusively at the unit, or organizational, level. Section 2464 of title10 of the United States Code requires DOD to maintain a core depot-level maintenance and repair capability that is government-owned and -operated. Maintaining this capability provides a ready and controlled source of technical competence and resources to enable effective and timely response to mobilizations, contingencies, or other emergencies. Additionally, DOD must assign these government- owned and -operated facilities (the depots) sufficient workload to ensure cost efficiency and technical competence during peacetime, while preserving the surge capacity and reconstitution capabilities necessary to fully support the strategic and contingency plans prepared by the Chairman of the Joint Chiefs of Staff. The term “data rights” in the DOD context typically refers to the license rights that the department acquires in two types of deliverables: technical data and computer software. These rights are addressed in law, in the Defense Federal Acquisition Regulation Supplement (DFARS), and in DOD guidance. These data rights are defined as follows: Technical data: recorded information, regardless of the form or method of recording, of a scientific or technical nature (including computer software documentation). Computer software: computer programs, source code, source code listings, object code listings, design details, algorithms, processes, flow charts, and related material that would enable the software to be reproduced, recreated, or recompiled. Computer software documentation: owner’s manuals, user’s manuals, installation instructions, operating instructions, and other similar items, regardless of how this documentation is stored, that will explain the capabilities of the computer software or provide instructions for using the software. DOD has policies and organizations in place within weapon system management and depot maintenance to manage the sustainment of operational system software. We found that DOD has policies for managing the life-cycle of weapon systems, including sustainment; and that DOD policy on depot maintenance and cost also considers weapon system software issues. Several organizations, including the Under Secretary of Defense for Acquisition and Sustainment and DOD software centers, play key roles in overseeing and managing software sustainment. Software sustainment activities are conducted at numerous facilities, including military department software centers, weapon system program management offices, government laboratories or software integration laboratories, and contractor facilities. Additionally, while DOD has defined software sustainment and software maintenance activities synonymously, and it defines these functions as part of depot maintenance, we determined that the Navy categorizes and reports software sustainment differently. DOD has published a directive and an instruction to guide the military departments in life-cycle management of major weapon systems, including considerations relating to software and weapon system sustainability. First, DOD’s acquisition publications provide DOD-wide policy and assign responsibilities to OSD and the military departments for executing weapon system development, production, and sustainment. For example, weapon system software considerations, including cost and access to technical data (for example, product specifications) and computer software (for example, source code), are to be included in required documentation, such as the Life-Cycle Sustainment Plan and the Systems Engineering Plan. Regulatory and reporting requirements differ depending on a system’s cost and acquisition category. These policies are in accordance with statute directing the Secretary of Defense to issue and maintain comprehensive guidance on life-cycle management. Second, DOD includes weapon system software considerations in its instruction regarding depot maintenance core capabilities. DOD-wide policy assigns responsibilities to OSD and the military departments for the performance of DOD core depot-level maintenance, including software. DOD policy states that maintenance tasks are performed to restore safety and reliability when deterioration has occurred. These tasks help to ensure military readiness, including mobilization and surge capabilities, to support national defense strategic and contingency requirements. Additionally, DOD policy states that, for inherently governmental and core capability requirements, maintenance programs are to use organic—or DOD personnel, rather than contractors—in accordance with the law. These DOD policies accord with the statute directing the Secretary of Defense to maintain a core depot-level maintenance and repair capability to ensure technical competence in peacetime while preserving the surge capacity necessary to fully support strategic and contingency needs. Third, DOD includes weapon system software considerations in its cost policy and manuals. These policies assign responsibilities for estimation of costs and collection of costs (including operations and support costs). They also prescribe cost data reporting and software resource data reporting requirements. Several DOD organizations establish policies and procedures for weapon system software sustainment. First, the Under Secretary of Defense for Research and Engineering and the Under Secretary of Defense for Acquisition and Sustainment play key roles in the establishment and maintenance of policy and procedures for software sustainment. For example: Research and Engineering: This office establishes policy and oversees research, system engineering, and developmental test processes, especially during formative stages of programs. It also supports the Joint Federated Assurance Center, a cross-DOD working group with a mission to develop, maintain, and offer software and hardware vulnerability detection, analysis, and remediation capabilities. Acquisition and Sustainment: This office establishes policy and manages acquisition and sustainment of major weapon systems. In April 2018 the Under Secretary appointed the first special assistant for software acquisition to advise and assist in addressing software challenges. According to officials, the special assistant will, among other responsibilities, oversee the development of software development policies and standards across DOD practices, and will advise leadership on best practices in software sustainment and data rights issues. Second, the Deputy Assistant Secretary of Defense for Materiel Readiness, under the Assistant Secretary of Defense (Sustainment), establishes policy for and manages DOD depot-level maintenance, including software sustainment. Third, the Office of Cost Assessment and Program Evaluation analyzes resource allocation and cost estimation, and provides independent analytic advice on, among other things, the cost-effectiveness of defense systems. Figure 4 highlights select organizations that establish and maintain software sustainment policy and procedures. Software sustainment is conducted either at DOD software centers— which include military department software centers, weapon system program management offices, government laboratories, and software integration laboratories—or at contractor facilities. The specifics of how the software sustainment is conducted vary by weapon system, in accordance with what the program manager negotiates with the DOD software center or contractor. At DOD software centers, software is developed, tested, and distributed by government staff, contractor staff, or both to maintain operational capability, correct faults, improve performance, and adapt the software to environmental changes. Activities range from small fixes for software errors to large releases that provide weapon systems with new capabilities or address cybersecurity vulnerabilities. The DOD software centers sustain a range of different systems. For example, U.S. Army Communications and Electronic Command’s Software Engineering Center sustains software for Army communications systems; and the U. S. Army Aviation and Missile Research Development and Engineering Center sustains software for missiles, space, and aviation; The Oklahoma City Air Logistics Complex’s 76th Software Maintenance Group at Tinker Air Force Base provides DOD with capabilities in operational flight programs, mission planning systems, space systems, ground-based radar, weapons support, mission support, jet engine test, training and simulation systems, and diagnostics and repair; and Space and Naval Warfare Systems Center Pacific supports and maintains Naval systems in the areas of command and control, communications, computers, and intelligence, surveillance, and reconnaissance, as well as cyber and space. This work is necessary to maintain and upgrade weapon system software and to meet immediate military operational needs. During our review, officials at DOD software centers provided additional examples of software sustainment activities they conduct on a wide variety of weapon systems. Appendix II provides these additional examples. DOD has defined software sustainment and software maintenance activities synonymously, and it defines these functions as part of depot maintenance and the core logistics process. The Departments of the Army and the Air Force categorize and report software sustainment as part of depot maintenance and the core logistics process. Specifically, the Army and the Air Force have policies that categorize and report software sustainment as part of their core logistics requirements, in accordance with DOD instruction. Contrary to DOD policy, the Department of the Navy does not categorize and report software sustainment as part of depot maintenance Specifically, Navy officials said that the Navy views software sustainment as an engineering function, not a depot maintenance function. They said that Navy policy reflects the Navy’s view of software sustainment as a continuous engineering process that occurs throughout a weapon system’s life-cycle, rather than a discrete set of activities categorized as depot maintenance. These officials stated that while the Navy believes software sustainment to be critical to maintaining its weapon systems, it also believes that managing software sustainment as part of depot maintenance is not the most effective approach for the Navy. In particular, Navy officials expressed several concerns about how reporting and categorizing software sustainment as part of depot maintenance could affect their activities. For example, Navy officials noted that this shift would require software engineering to be reported as depot maintenance, which in turn would require the Navy to carry out a greater portion of the work at Navy depots using DOD’s workforce. Navy officials stated that, in their opinion, the Navy does not have the capacity to conduct this level of effort with the current DOD workforce within the Navy depot structure, and that the Navy’s ability to develop adequate capacity in its DOD software engineering workforce in the future is uncertain. They also stated that shifting this capacity away from private industry to the DOD software engineering workforce could create instability in the management of current and future Navy programs, and would be inconsistent with the Navy’s efforts to broaden private-sector software engineering capability and capacity. We also found that the Department of the Navy does not categorize and report software sustainment as part of its core logistics requirements, in accordance with DOD policy. DOD Instruction 4151.20, Depot Maintenance Core Capabilities Determination Process, assigns responsibilities and prescribes procedures to identify required core capabilities for depot maintenance and the associated workloads needed to sustain those capabilities. It is DOD policy that the core capability requirements determination process underpins the establishment and retention of a broad set of public-sector depot maintenance capabilities necessary for DOD, and that the required core capabilities and depot maintenance workloads necessary to sustain those capabilities will be calculated by military services and then aggregated to determine the overall DOD core requirements. As such, DOD requires the military services to use a computational methodology to identify their essential core capability requirements and their planned workload to support this core maintenance capability. The Navy’s differing approach to categorizing and reporting software sustainment has created challenges for DOD-wide reporting on core logistics capabilities. DOD is required by law to submit a Biennial Core Report to Congress that identifies core logistics capabilities—and DOD has included software sustainment—at depots, and the workload required to maintain those capabilities. The Army and the Air Force included direct labor hours and estimated sustainment costs for DOD depot-level software sustainment in the 2018 DOD Biennial Core Report. However, while the Navy conducted software sustainment activities, it did not consider these activities to be part of depot maintenance or a core logistics capability, as previously discussed. As a result, the Navy reported no direct labor hours or estimated cost of sustaining its software workload for inclusion in the 2018 DOD Biennial Core Report, as shown in table 1. OSD accepted the Navy’s core report submission for the 2018 DOD Biennial Core Report. The Department of the Navy’s position that software sustainment is not part of depot maintenance is contrary to DOD Instruction 4151.20, which specifically includes software sustainment as part of depot maintenance. Without the Department of the Navy’s categorizing and reporting of its software sustainment costs, in accordance with DOD policy on the Depot Maintenance Core Capabilities Determination Process, DOD and Congress are not fully informed of the magnitude and cost of core software sustainment capability requirements for the Navy. Accordingly, DOD is impeded in its efforts to plan for a ready and controlled source of technical competence, and to budget resources in peacetime while preserving the surge capabilities necessary to fully support strategic and contingency needs. DOD’s ability to track weapon system software sustainment costs is impeded by limitations in the collection of software data by both the Office of Cost Assessment and Program Evaluation and the military departments. CAPE oversees the primary cost data collection systems: the Cost and Software Data Reporting system and the military departments’ Visibility and Management of Operating and Support Costs system. Further, CAPE has limitations in its cost and software data reporting system for data collected from DOD software centers. We also found that the military departments collect incomplete data on software sustainment costs in their VAMOSC systems. CAPE collects software sustainment cost data from contractors on certain major weapon systems through its CSDR system. According to CAPE’s CSDR manual, this system serves as the primary repository of contractor costs for use in most DOD resource analysis efforts, including cost database development, applied cost-estimating, cost research, program reviews, analysis of alternatives, and life-cycle cost estimates. Data from the two principal components of the CSDR system–contractor cost data reporting and software resources data reporting systems—can be used in managing software sustainment costs. Data in the CSDR system may also be used to prepare acquisition and life-cycle cost estimates for weapon system milestone reviews, as well as to estimate and project software sustainment costs. We identified limitations, however, in CAPE’s CSDR system. First, the system has historically not collected information from contractors for weapon system acquisition programs whose spending levels did not reach the major defense acquisition program threshold. Although collecting this information was not a requirement in the past, in 2016 Congress directed DOD to begin to collect additional information necessary to facilitate cost estimation and comparison across acquisition programs, including costs from programs with eventual total expenditures greater than $100 million. In February 2018, as part of its overall efforts to make data collection more robust, CAPE issued a memo stating that the Army, Navy, and Air Force proposed pilot programs to collect contractor cost data from 26 weapon system programs whose spending levels were below the major defense acquisition program threshold. CAPE plans to use the results of these pilot programs to inform future efforts to improve information-gathering on, and visibility into, the actual expenditures for lower-dollar programs. Additionally, CAPE plans to update its cost- collection policies and manual, if necessary, upon completion of the pilot programs. Because the department is in the midst of these pilot programs and has outlined next steps to be taken upon their completion, we are not making a recommendation about this matter at this time. Second, CAPE’s CSDR system does not collect any weapon system cost or software data from DOD software centers. Prior to 2017, CAPE required only contractors—and not DOD software centers—supporting major defense acquisition programs to report software sustainment costs into the CSDR system. However, in January 2017 CAPE recognized that the lack of cost and software data from government-executed elements of acquisition and sustainment programs was impeding accurate compilation of total program costs. Accordingly, it issued a memorandum to the military departments directing that cost and software data efforts on major defense acquisition programs should also be collected and submitted into the CSDR system by government-performed efforts, which include DOD software centers. Also, the Standards for Internal Control in the Federal Government states that management should use quality information to achieve an entity’s objectives, and that management should obtain data from reliable internal and external sources in a timely manner based on the identified information requirements for effective monitoring. According to a CAPE official, as of September 2018, CAPE had not received any inputs into the CSDR system for DOD-performed software sustainment efforts. CAPE officials told us that compliance with this requirement in the memorandum has been very low, and they attributed this to the absence of an implementation plan. The official said that CAPE is currently in the early stages of evaluating cost data systems—that is, CSDR and the military departments’ VAMOSC systems—to determine which is the more effective for use in collecting and submitting cost and software data from DOD software centers. The official acknowledged that after completing this evaluation of the systems, CAPE will develop an implementation plan. However, CAPE is still in the early stages of completing its evaluation. Having a robust implementation plan with time frames for key milestones will be important to executing and monitoring CAPE’s actions to improve the reporting of software sustainment costs. Without cost and software data from the DOD software centers, CAPE is challenged in its ability to accurately compile total program costs for program managers, cost estimators, and Congress, among other information recipients. According to the CAPE cost estimating guide, the software sustainment element excludes the costs of new development or major redesigns that provide new capabilities. However, if the costs of new development or major redesigns that provide new capabilities cannot be isolated, these costs will be considered as part of software sustainment and should be so noted in the estimate documentation. defines cost elements that cover the range of weapon system operating and support costs, including software sustainment. CAPE’s cost guide defines the software sustainment cost element as the labor, material, and overhead costs incurred after deployment to maintain, modify, and integrate software. According to the CAPE cost estimating guide, the software sustainment element excludes the costs of new development or major redesigns that provide new capabilities. However, if the costs of new development or major redesigns that provide new capabilities cannot be isolated, these costs will be considered as part of software sustainment and should be so noted in the estimate documentation. major commands. Therefore, in order to include software sustainment costs for all shipboard systems in the VAMOSC system, Navy officials must manually collect these cost data. This official explained that since the Navy collects these costs manually, officials focus their efforts on the most expensive and most populous shipboard systems. According to the official, they intend to address the Navy VAMOSC system’s incomplete software sustainment data issue by expanding their manual data collection efforts to include additional Navy systems. According to DOD policy, CAPE’s executive oversight responsibilities include annually reviewing the services’ VAMOSC systems to address data accessibility, completeness, timeliness, accuracy, and compliance with CAPE guidance. CAPE formed a VAMOSC task force in partnership with the service cost-analysis agencies and the Product Support Division in the office of the Assistant Secretary of Defense for Sustainment. The task force is aware of gaps in the military departments’ reporting of software sustainment costs within their VAMOSC systems, particularly within the Army and the Navy, and it has included data completeness in the scope of its efforts. However, closing data gaps is not one of the specific purposes of the task force; these purposes include (1) discussing integration of operating and support cost collection across the department and (2) clearly defining the technical differences across the military services’ VAMOSC systems. The task force is concerned with multiple cost-reporting issues. We recognize that the task force can enable DOD to improve the completeness of its software sustainment cost reporting. Further, systematic and institutionalized cost data collection by each military department is important to support credible cost estimates of current and future programs. However, without CAPE taking steps to prioritize obtaining complete information on operating and support costs for software sustainment, it cannot provide reliable life-cycle cost estimates to DOD acquisition or maintenance officials—or Congress— to assist with current and future years funding decisions. DOD continuously makes decisions about securing data rights, both early and throughout the life-cycle of a weapon system (see sidebar). DOD may obtain data rights, including access to technical data and computer software related to weapon systems, for a variety of reasons. For example, as we have previously reported, data rights may be obtained to help control costs and maintain flexibility in future acquisition and sustainment of systems and subsystems, including maintenance and upgrade of weapon system software. DOD officials we spoke with emphasized that there is no one-size-fits-all approach. Further, obtaining data rights for software sustainment constitutes only one of many competing priorities that must be considered along with cost, schedule, and performance in the acquisition of weapon systems. technical data or computer software, to be delivered under a contract. DOD officials told us that this was due to cost and proprietary reasons— that is, the contractor retains ownership of the intellectual property, such as the source code. DOD strives to balance the cost of purchasing the rights against the extent of data rights it expects it will need to maintain and support the system for years into the future. For example, DOD obtains data rights for the following reasons: To support its ability to evaluate weapon system design in order to sustain weapon system software. Computer software: computer programs, source code, source code listings, object code listings, design details, algorithms, processes, flow charts, and related material that would enable the software to be reproduced, recreated, or recompiled. owner's manuals, user's manuals, installation instructions, operating instructions, and other similar items, regardless of how this documentation is stored, that will explain the capabilities of the computer software or provide instructions for using the software. upgrades and sustainment activities to achieve cost savings. Re- competing requires complete technical data packages that enable the manufacture of data equipment from specification. During the operating and support phase of a weapon system, DOD may need to reconsider its previous decisions about the extent of data rights it previously acquired. DOD officials we spoke with emphasized that there are situations in which the data rights needed may not be known until years into sustainment. A senior-level DOD official told us that it would be useful if data rights could have a pre-negotiated price and be an option as part of the initial contract. Such an option would give the government the right, but not the obligation, to purchase the data rights at the pre- negotiated price if needed, in the future. DOD has faced challenges in securing the necessary data rights to sustain weapon system software. Specifically, having either partial or incomplete data, or unclear data rights, or both can impede the government’s ability to support the weapon system as intended. For example, our recent work on the F-35 Joint Strike Fighter Program found that DOD has not defined all of the technical data it needs from the prime contractor, and at what cost, to enable competition of future sustainment contracts. Officials at DOD software centers told us that they take steps to mitigate challenges posed by having either partial or incomplete data, or unclear data rights, or both for decades-old weapon systems and new acquisitions. For decades-old weapon systems, officials at some DOD software centers stated that they use public-private partnerships to bridge gaps for systems that lack access to the necessary data rights. For example, an Air Force official at Robins Air Force Base told us that the C- 5 software sustainment workload has been successful due to a public- private partnership involving the C-5 System Program Office, the 402nd Software Maintenance Group, and the contractor. As part of this partnership, a C-5 software integrated laboratory was established at Robins Air Force Base for DOD personnel to perform software sustainment activities, including deficiency report investigations and testing. In doing so, the 402nd Software Maintenance Group supports $8.4 million in annual C-5 software sustainment requirements. Officials at DOD software centers further explained that they have the expertise to optimize software that is transferred from a contractor to a DOD software center or to reverse-engineer software for weapon systems, if needed. In some cases, for example, a contractor may decide that it is no longer profitable or advantageous to continue performing the software sustainment; the activities can then be transferred to a DOD software center. Air Force officials at the 402nd Software Maintenance Group stated that on many occasions they have worked to take over software from a contractor without any transition period. In 2013 this DOD software center assumed sustainment responsibility from a contractor without any transition period for a radar system on the F-15 aircraft in order to maintain and upgrade its software. After assuming sustainment responsibility, according to an Air Force official, this DOD software center corrected latent defects and added new capabilities to adapt the radar to a changing threat environment. According to the official, this occurred because the contractor shifted focus to newer radar systems. Further, the contractor priced the support for the older radar system above what the Air Combat Command had budgeted for the updates. Officials at some DOD software centers told us that if they have the source code but do not have the computer software documentation— such as manuals or instructions—they may need to reverse-engineer the software. For example, engineers at U.S. Army Research, Development and Engineering Command, Armament Research, Development, and Engineering Center (ARDEC) reverse-engineered a key software function, as shown in figure 5 below. For newer acquisitions, DOD has increased the consideration it affords to the potential needs for access to and delivery of data. For example, Air Force officials said that because of past issues with data rights on legacy systems, they had launched an initiative to ensure that program offices use standardized contract clauses (for example, DFARS software data rights) and contract delivery requirements (for example, models, drawings, associated lists, and specifications) for data rights. To illustrate this, an Air Force official told us that the HH60W Combat Rescue Helicopter program committed early in the life-cycle to securing the necessary data rights for a DOD software center in the 402nd Software Maintenance Group to perform the software sustainment activities. The official told us that the Statement of Work requests that the contractor provide the DOD software center with the source code and full technical data package, to include a complete software-supporting documentation package. Provisions in the fiscal years 2016 and 2018 National Defense Authorization acts (NDAA) directed the Secretary of Defense to commission studies related to DOD intellectual property, establish an intellectual property policy, and establish a cadre of intellectual property experts. In response, DOD is in the early stages of developing intellectual property policy and establishing a cadre of intellectual property experts. Also, DOD has commissioned studies to review its access to intellectual property for DOD weapon systems, including necessary data rights. However, the department has missed some required reporting time frames, and it has not yet reported to congressional defense committees on the studies’ findings and recommendations. In the fiscal year 2018 NDAA, Congress directed the Secretary of Defense, through the Under Secretary of Defense for Acquisition and Sustainment, to (1) develop policy on the acquisition or licensing of intellectual property; and (2) establish a cadre of intellectual property experts to help support the acquisition workforce on intellectual property matters, including acquiring or licensing intellectual property. The law did not include a time frame for completion. The department is in the early stages of addressing these statutory provisions. According to the law, the policy is intended to enable DOD-wide coordination and consistency in strategies for acquiring or licensing intellectual property; to help ensure that program managers are aware of DOD’s rights and consider and use best practices early in the acquisition process; and to encourage customized intellectual property strategies based on the unique characteristics for each system. The cadre of experts is intended to ensure a consistent, strategic, and knowledgeable approach to acquiring or licensing intellectual property by providing expert advice, assistance, and resources to the acquisition workforce on intellectual property matters. While the department is in the early stages of addressing these statutory provisions, senior-level DOD officials have acknowledged a delay in these efforts, primarily due to the department’s recent reorganization. DOD officials stated that the details concerning organizational structure, roles, responsibilities, and realignment of resources had to be finalized in order for the newly formed organizations to implement these provisions. Regarding the intellectual property policy, a senior-level DOD official told us that the Office of Strategy and Design, within the Office of the Secretary of Defense, will facilitate the collaboration of stakeholders to assist in developing the intellectual policy, which the Assistant Secretary of Defense (Acquisition) will then issue and oversee. Senior-level DOD officials spoke with us regarding the complexity of developing this intellectual property policy, as it spans the weapon system life-cycle, including research, development, acquisition, and operating and support considerations. Regarding the intellectual property cadre, a senior-level DOD official told us that the Assistant Secretary of Defense (Acquisition) may house the cadre. As of August 2018 the department had not yet specified details on the potential size or scope of the intellectual property cadre, nor a time frame to guide implementation. Although not required by law, development of a robust implementation plan with time frames for key milestones could help DOD to execute and monitor its actions. In the fiscal year 2016 NDAA Congress directed DOD to establish a Government-Industry Advisory Panel to review technical data rights, and to submit its final report and recommendations to the Secretary of Defense not later than September 30, 2016. The panel, comprising members from both the public and private sectors, was to review defense regulations on technical data and proprietary restrictions to ensure, among other things, that DOD does not pay more than once for the same work, and that contractors are appropriately recompensed for innovation and invention, among several other considerations. The law also directs that the Secretary of Defense submit comments or recommendations to congressional defense committees not later than 60 days after receiving the report. DOD established the panel, as legislatively required. As of November 2018 the panel had submitted its report to DOD but not to Congress. Panel members acknowledged that the panel is late in reporting to the congressional defense committees, and they attributed the lateness to the complexity of the task. Panel members told us that obtaining consensus between DOD and industry has been difficult, in part because of competing interests. For example, panel members discussed balancing DOD’s needed ability to upgrade and support weapon systems—which is difficult to forecast 30 to 40 years into the future—with industry’s need for a fair return on its intellectual property investments. In November 2018 the panel submitted the report to the Under Secretary of Defense for Acquisition and Sustainment. The report includes 19 recommendations for legislative, regulatory, and policy changes that, according to the panel chairman, recognize and seek to balance the equities of both government and industry. As of November 21, 2018, the panel had not yet transmitted the report to Congress, but the panel Chairman stated that it planned to do so before the end of the month. In the fiscal year 2016 NDAA, Congress directed DOD to contract with an independent entity to review DOD regulations, practices, and sustainment requirements related to government access to and use of intellectual property rights of private-sector firms. The law also directs the Secretary of Defense to submit a report to the congressional defense committees on the findings of the independent entity, along with a description of any actions the Secretary proposed in order to revise and clarify laws, or actions the Secretary may take to revise or clarify regulations, related to intellectual property rights. In response, DOD contracted with the Institute for Defense Analyses to review the intellectual property for weapon system sustainment. In May 2017 the Institute released its report on access to intellectual property for weapon system sustainment. The report made six recommendations, including that DOD establish or expand existing organizational capabilities within the DOD components (with OSD support) to provide expertise in the acquisition of intellectual property data and rights to program managers throughout their programs’ life-cycles, as well as to other staff involved in weapon system acquisition. However, DOD has not yet submitted its report to the congressional defense committees on the study’s findings and recommendations, though it was required to do so by March 1, 2016. OSD officials acknowledged that they are late in reporting to congressional defense committees on the study’s findings and recommendations. They attributed the delay to their intent of awaiting the findings and recommendations on technical data rights, if any, of the Government-Industry Advisory Panel, as discussed above. DOD informed the congressional defense committees twice—most recently in January 2018—that the department would consider the recommendations of the Institute for Defense Analyses and those of the Panel collectively, and would provide its recommendations in a single report after receiving the Panel’s report. In this January 2018 update, DOD noted that the Panel expected to complete its report by March 2018. However, the Panel did not complete its report—for which DOD was waiting before responding to the Institute’s study—until November 2018. DOD’s report to Congress on any actions it might take in response to the study’s findings and recommendations could provide insight into whether laws or regulations related to intellectual property rights need to be revised or clarified. Software is essential to the capabilities and operations of a vast range of military systems, including tactical and combat vehicles, aircraft, ships, submarines, and strategic missiles. DOD has policies and organizations within weapon system management and depot maintenance to manage operational system software sustainment. DOD has defined software sustainment and software maintenance activities synonymously, and the department includes software maintenance as part of depot maintenance core capabilities. However, the Department of the Navy does not categorize or report software sustainment as part of depot maintenance. Without the Department of the Navy’s categorizing and reporting of its software sustainment costs, in accordance with DOD policy on the Depot Maintenance Core Capabilities Determination Process, DOD and Congress are not fully informed of the magnitude and cost of core software sustainment capability requirements. As such, DOD is impeded in its efforts to plan for a ready and controlled source of technical competence and to budget resources in peacetime while preserving the surge capabilities necessary to fully support strategic and contingency needs. Limitations exist in DOD’s cost and software data reporting system with regard to its obtaining cost data from DOD software centers, as well as in the military departments’ operating and support cost systems. These limitations impede DOD’s tracking of weapon system software sustainment costs. Without cost and software data from the DOD software centers as well as complete information on the military departments’ operating and support costs for software sustainment, CAPE is challenged in its ability to accurately compile total program costs for program managers, cost estimators, and Congress, among other information recipients. Lastly, while DOD makes decisions about securing data rights both early and throughout the life-cycle of a weapon system, the department faces challenges in balancing the cost of purchasing the rights against the extent of data rights it expects it will need over the life of the system. DOD has begun taking actions to address these challenges. For example, DOD has commissioned several studies, at congressional direction, to examine DOD’s access to and use of intellectual property, including technical data rights and proprietary restrictions. However, Congress has yet to receive two of those studies. Reporting on the findings and recommendations, as well as on any actions DOD may take in response to both studies, would provide insight and would highlight timely issues with technical data rights to keep Congress and DOD informed of government and industry concerns and enable them to use that knowledge in their decision making on weapon systems that may be in operation for decades to come. We are making five recommendations to the Department of Defense— one to the Secretary of the Navy and four to the Secretary of Defense. We recommend that the Secretary of the Navy categorize and report the Navy’s software sustainment costs, in accordance with DOD policy on the Depot Maintenance Core Capabilities Determination Process. We recommend that the Secretary of Defense ensure that the Director for Cost Assessment and Program Evaluation complete its evaluation and select the most effective system to obtain cost and software data from DOD software centers, and develop an implementation plan that includes time frames for key milestones to execute and monitor the centers’ submission of required data. We recommend that the Secretary of Defense ensure that the Director for Cost Assessment and Program Evaluation takes steps to prioritize the respective military departments’ obtaining and reporting of complete operating and support costs for software sustainment through its VAMOSC systems. We recommend that the Secretary of Defense develop an implementation plan with time frames for key milestones for establishing a cadre of intellectual property experts. We recommend that the Secretary of Defense submit a report, as required by law, to Congress about the study on access to intellectual property for weapon system sustainment conducted by the Institute for Defense Analyses, along with a description of any actions that the Secretary proposes, or may take, to revise or clarify regulations related to intellectual property rights. We provided a draft of this report to the Department of Defense for review and comment. DOD provided written comments, which are reprinted in appendix III. In its comments, DOD concurred with our recommendations and stated it has actions underway or plans to take actions in response to all five of our recommendations. We are sending copies of this report to the appropriate congressional committees and the Acting Secretary of Defense. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9627 or maurerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. This report examines the extent to which (1) DOD has policies and organizations in place to manage the sustainment of operational system software for weapon systems; (2) DOD and the military departments track costs to sustain weapon system software; and (3) DOD has addressed challenges securing necessary data rights to sustain weapon system software. Our scope included software sustainment of operational weapon systems. For objective one, we reviewed DOD policies and organizations in place to manage the sustainment of operational system software for weapon systems. This included DOD Directive 5000.01 and DOD Instruction 5000.02, which establish acquisition program policies; and DOD Directive 4151.18 and DOD Instruction 4151.20, which outline requirements for DOD materiel maintenance and DOD programs’ core capabilities. We reviewed statutory requirements, including 10 United States Code § 2337, which requires the Secretary of Defense to issue and maintain comprehensive guidance on life-cycle management and the development and implementation of product support strategies for major weapon systems. We compared the processes used by DOD and the military departments against those outlined in DOD policy and statute, and against software sustainment activities performed at several DOD software centers. We identified the roles and responsibilities for conducting software sustainment activities among personnel at each level of DOD bureaucracy. We also interviewed officials from the Office of the Secretary of Defense (OSD) and the military departments regarding the department’s guidance and the processes used to collect the data for DOD’s Biennial Core Report. As in our previous reviews of DOD’s biennial core reports, we did not assess the reliability of the underlying data provided by the military services for the 2018 DOD Biennial Core Report. However, we determined that the data were sufficiently reliable for the purpose of determining whether the military services had reported costs of workloads in 2012—2018. We interviewed officials from the Office of the Secretary of Defense (OSD), including within the Office of the Under Secretary of Defense for Research and Engineering and the Office of the Under Secretary of Defense for Acquisition and Sustainment. Using a semi-structured questionnaire, we also interviewed officials from each of the military department headquarters—U.S. Army G4, Air Force Acquisition office, and the Assistant Secretary of the Navy for Research, Development, and Acquisition—to understand policies and organizations in place to manage the sustainment of operational system software for major weapon systems. We also interviewed industry officials, such as from the Center for Strategic and Budgetary Assessments and the Software Engineering Institute at Carnegie Mellon University. We conducted interviews using a semi-structured questionnaire with officials at select DOD depot-level software sustainment activities, also referred to as DOD software centers for the purposes of this report. We used DOD’s Fiscal Year 2016 Maintenance Fact Book to select 11 of 20 DOD depot-level software sustainment activities based on several criteria, including (1) military department, (2) weapon system type, (3) geographical location, and (4) random selection. Although this sample is not generalizable to the population of DOD depot-level software centers, the use of a random sample of software centers helped mitigate any potential selection bias, and the interviews provided valuable information on those sites selected. The officials we interviewed at DOD software centers included a variety of engineers and others who perform software sustainment activities for weapon system software on several DOD weapon systems, including air and sea platforms, targeting systems, and communications systems, among others. We interviewed these officials to gain an understanding of policies and procedures they follow to guide their software sustainment activities, how they are organized, and the activities they undertake to sustain the software. For objective two, we reviewed DOD policy and military department guidance regarding software sustainment cost reporting requirements, including Department of Defense Manual 5000.04, Cost and Software Data Reporting Manual, and applicable financial management regulations. We reviewed the Office of Cost Assessment and Program Evaluation (CAPE) Reports to Congress for Fiscal Years 2016 and 2017 to learn about steps that CAPE is taking to address challenges. We interviewed officials at the DOD software centers responsible for weapon system software on several DOD weapon systems to gain an understanding of how they track cost data. We also interviewed officials from OSD, including officials from CAPE, and officials from the three cost analysis agencies responsible for collecting operating and support costs for the military departments’ Visibility and Management of Operating and Support Costs (VAMOSC) data collection systems. These agencies include Office of the Assistant Secretary of the Army for Cost and Economics, the Air Force Cost Analysis Agency, and the Naval Center for Cost Analysis. For objective three, we reviewed statutes governing DOD intellectual property, including technical data rights, computer software, and computer software documentation. These statutes included, for example, 10 U.S.C. §2320, “Rights in Technical Data,” and 10 U.S.C. §2321, “Validation of Proprietary Data Restrictions.” Both of these statutes are implemented, in part, by the Federal Acquisition Regulation and the Defense Federal Acquisition Regulation Supplement (DFARS), which we also reviewed. Specifically, we reviewed DFARS Subpart 227.71, “Rights in Technical Data,” and DFARS Subpart 227.72, “Rights in Computer Software and Computer Software Documentation.” Both include sections that address DOD definitions of technical data; computer software; and computer software documentation, policy, acquisition, licensure, and delivery rights, among other items. We also reviewed DOD policy and guidance, including DOD 5010.12-M, Procedures for the Acquisition and Management of Technical Data. We reviewed the Defense Acquisition Guidebook, which addresses the acquisition and maintenance of technical data rights to sustain and upgrade software on major weapon systems. We also reviewed guidance put forth on intellectual property strategies, including a checklist arranged by contract phase for key intellectual property management activities and considerations. We interviewed officials from OSD, including from the Office of General Counsel and the Office of Strategic Design, as well as officials from the military department headquarters, to gain an understanding of the necessary technical rights to sustain weapon system software, the reasons that technical data rights are needed, and challenges faced by the department. We interviewed officials at the DOD software centers covering a variety of DOD weapon systems to gain an understanding of what technical data rights they need for their respective weapon systems, and the ways in which they manage issues they may encounter in which contractors own the technical data. We analyzed select weapon systems for which DOD had complete data and rights, as well as weapon systems for which DOD had partial or incomplete data rights, and the actions DOD took for sustainment, such as public-private partnerships. We also interviewed members of the Government-Industry Panel examining technical data rights and proprietary data restrictions to gain an understanding of necessary data rights for sustaining weapon systems coupled with proprietary concerns from industry. Finally, we reviewed statutory provisions in the fiscal years 2016 and 2018 National Defense Authorization acts, which directed the Secretary of Defense to commission studies related to DOD intellectual property, and we interviewed officials to understand DOD’s status on the provisions. Table 2 lists the offices that we visited or contacted during our review. In addition to the contact listed above, Sally Newman (Assistant Director), Laura Czohara (Analyst-in-Charge), Steven Bagley, Steven Boyles, Vincent Buquicchio, Amie Lesser, Janine Prybyla, Andrew Stavisky, and Cheryl Weissman made key contributions to this report. GAO- F-35 Joint Strike Fighter: Development is Nearly Complete, but Deficiencies Found in Testing Need to Be Resolved, GAO-18-321 (Washington, D.C.: June 5, 2018). GAO- F-35 Aircraft Sustainment: DOD Needs to Address Challenges Affecting Readiness and Cost Transparency, GAO-18-75 (Washington, D.C.: Oct. 26, 2017). GAO, Military Acquisitions: DOD Is Taking Steps to Address Challenges Faced by Certain Companies, GAO-17-644 (Washington, D.C.: July 2017). GAO- F-35 Sustainment: DOD Needs a Plan to Address Risks Related to Its Central Logistics System, GAO-16-439, (Washington, D.C., Apr. 14, 2016). GAO- F-35 Joint Strike Fighter: Preliminary Observations on Program Progress, GAO-16-489T (Washington, D.C.: Mar. 23, 2016). GAO, Defense Contracting: Early Attention in the Acquisition Process Needed to Enhance Competition, GAO-14-395 (Washington, D.C.: May 5, 2014). GAO- F-35 Joint Strike Fighter: Problems Completing Software Testing May Hinder Delivery of Expected Warfighting Capabilities, GAO-14-322 (Washington, D.C.: Mar. 24, 2014). GAO- F-35 Joint Strike Fighter: Program Has Improved in Some Areas, but Affordability Challenges and Other Risks Remain, GAO-13-500T (Washington, D.C.: Apr. 17, 2013). GAO, Defense Acquisition: DOD Should Clarify Requirements for Assessing and Documenting Technical-Data Needs, GAO-11-469 (Washington, D.C.: May 11, 2011). GAO, Federal Contracting: Opportunities Exist to Increase Competition and Assess Reasons When Only One Offer Is Received, GAO-10-833 (Washington, D.C.: July 26, 2010). GAO, Weapons Acquisition: DOD Should Strengthen Policies for Assessing Technical Data Needs to Support Weapon Systems, GAO-06-839 (Washington, D.C.: July 14, 2006). GAO, Defense Management: Opportunities to Enhance the Implementation of Performance-Based Logistics, GAO-04-715 (Washington, D.C.: Aug. 16, 2004). GAO, Defense Logistics: Opportunities to Improve the Army’s and the Navy’s Decision-making Process for Weapons Systems Support, GAO-02-306 (Washington, D.C.: Feb. 28, 2002). GAO, Defense Logistics: Air Force Lacks Data to Assess Contractor Logistics Support Approaches, GAO-01-618 (Washington, D.C.: Sept. 7, 2001). GAO, Test and Evaluation: DOD Has Been Slow in Improving Testing of Software Intensive Systems, GAO/ NSIAD-93-198 (Washington, D.C.: September 1993). GAO, Mission Critical Systems, Defense Attempting to Address Major Software Challenges, GAO/IMTEC-93-13 (Washington, D.C.: December 1992). GAO, Risk and Control of the Software Maintenance Process (Washington, D.C.: January 1987). GAO, Federal Agencies’ Maintenance of Computer Programs: Expensive and Undermanaged, AFMD-81-25 (Washington, D.C., Feb. 26, 1981).

Summary: Software is integral to the operation and functionality of DOD equipment, platforms, and weapon systems, including tactical and combat vehicles, aircraft, ships, submarines, and strategic missiles. DOD estimates that software sustainment funding will total at least $15 billion over the next 5 fiscal years. DOD carries out software sustainment at various locations, where DOD uses its maintenance capabilities to maintain, overhaul, and repair its military weapon systems. GAO was asked to review several issues relating to the sustainment of operational system software for DOD weapon systems. This report examines, among other things, the extent to which (1) DOD has policies and organizations in place to manage the sustainment of operational system software for weapon systems; and (2) DOD and the military departments track costs to sustain weapon system software. GAO reviewed DOD policies and procedures and interviewed cognizant officials from select DOD software centers, among others, who perform weapon system software sustainment activities. The Department of Defense (DOD) has policies and organizations to manage the sustainment of operational system software. DOD policy defines software sustainment and software maintenance activities synonymously, to comprise any activities or actions that change the software baseline, as well as modifications or upgrades that add capability or functionality. One example of such an action is the Air Force's modifying the security software on the B-52 bomber to better protect against attempted system penetration. The figure below defines the four categories of software sustainment actions. DOD policies on life-cycle management of weapon systems address software sustainment, and several DOD organizations—including DOD software centers—play key roles in overseeing and managing software sustainment. DOD policy includes software maintenance as part of core logistics, and it requires the military departments to report biennially to Congress on their estimated workloads to sustain core logistics capabilities, including estimated costs of these workloads. However, while the Army and Air Force categorize and report software sustainment as part of core logistics, the Navy does not. Without the Navy's categorizing and reporting its software sustainment costs, DOD and Congress are not fully informed of the magnitude and cost of core software sustainment capability requirements. This impedes DOD's efforts to plan for a ready and controlled source of technical competence, and to budget resources in peacetime while preserving necessary surge capabilities. DOD's ability to track weapon system software sustainment costs is impeded by limitations in its collection of software cost data. First, GAO found that the Office of Cost Assessment and Program Evaluation's (CAPE) Cost and Software Data Reporting system did not collect weapon system cost data from DOD software centers. Recognizing this, CAPE directed in January 2017 that cost and software data efforts on major acquisition programs should begin to be collected from government organizations, including DOD software centers. However, CAPE acknowledges that it lacks an implementation plan to execute and monitor the requirement for these centers to submit cost and software data. Second, GAO also found that the military departments' operating and support cost systems have incomplete software sustainment cost data. DOD policy requires the military departments to collect and maintain actual operating and support costs, including software sustainment costs. Without CAPE's taking steps to prioritize obtaining complete information on operating and support costs for software sustainment, CAPE is challenged in its ability to accurately compile total program costs or provide reliable life-cycle cost estimates to DOD and Congress. GAO is making five recommendations, including that (1) the Navy categorize and report its software sustainment costs in accordance with DOD policy; and (2) CAPE improve the collection of weapon system software cost data. DOD concurred with GAO's recommendations.