DOD’s organizational structure includes the Office of the Secretary of Defense, the Joint Chiefs of Staff, the military services, numerous defense agencies and field activities, and various unified combatant commands that contribute to the oversight of DOD’s MAIS programs. Figure 1 provides a simplified depiction of DOD’s organizational structure. The Under Secretary of Defense for Acquisition, Technology, and Logistics (USD (AT&L)) serves as the Defense Acquisition Executive and is the official responsible for supervising the acquisition and oversight of MAIS programs. This official is the principal acquisition official of the department and is the acquisition advisor to the Secretary of Defense. The USD (AT&L) has policy and procedural authority for the defense acquisition system, which establishes the steps that DOD programs generally take as the department plans, designs, acquires, deploys, operates, and maintains its IT systems (discussed in more detail following this section). The USD (AT&L)’s authority also includes directing the military services and defense agencies on acquisition matters and— unless the authority is delegated to the service or agency—making milestone decisions for MAIS programs. The Department of Defense Instruction 5000.02 is the policy that establishes guidelines for the management of all acquisition programs. The department’s policy guidance includes six defense acquisition models based on the type of product being acquired (e.g., software- intensive programs and hardware-intensive programs). A generic acquisition model that shows all of the program life-cycle phases and key decision points is shown in figure 2 and described below. Materiel solution analysis: Refine the initial system solution (concept) and create a strategy for acquiring the solution. A decision—referred to as Milestone A, the risk reduction decision point—is made at the end of this phase to authorize entry into the technology maturation and risk reduction phase. Technology maturation and risk reduction: Determine the preferred technology solution and validate that it is affordable, satisfies program requirements, and has acceptable technical risk. A decision—referred to as Milestone B, the program initiation decision point—is made at the end of this phase to authorize entry of the program into the engineering and manufacturing development phase and award development contracts. An acquisition program baseline is first established at the Milestone B decision point. A program’s first acquisition program baseline contains the original life-cycle cost estimate (acquisition and operations and maintenance costs), the schedule estimate (major milestones and decision points), and performance parameters that were approved for that program by the Milestone Decision Authority. The first acquisition program baseline is established after the program has refined user requirements and identified the most appropriate technology solution that demonstrates that it can meet users’ needs. Engineering and manufacturing development: Develop a system and demonstrate through testing that the system meets all program requirements. A decision—referred to as Milestone C, the deployment decision point—is made during this phase to authorize entry of the system into the production and deployment phase or into limited deployment in support of operational testing. Production and deployment: Achieve an operational capability that meets program requirements, as verified through independent operational tests and evaluation, and implement the system at all applicable locations. Operations and support: Operationally sustain the system in the most cost-effective manner over its life cycle. Entities such as the Project Management Institute, the Software Engineering Institute at Carnegie Mellon University, IEEE, and GAO have developed and identified leading practices to help guide organizations to effectively plan and manage the acquisition of major IT systems, such as the MAIS programs. Our work has shown that proper implementation of such practices can significantly increase the likelihood of delivering promised system capabilities on time and within budget. These practices include, but are not limited to, Risk management: A process for anticipating problems and taking appropriate steps to mitigate risks and minimize their impact on program commitments. It involves identifying and documenting risks, categorizing them based on their estimated impact, prioritizing them, developing risk mitigation strategies, and tracking progress in executing the strategies. Requirements management: Requirements establish what the system is to do, how well it is to do it, and how it is to interact with other systems. Appropriate requirements development involves eliciting and developing customer and stakeholder requirements, and analyzing them to ensure that they will meet users’ needs and expectations. It also consists of validating requirements as the system is being developed to ensure that the final systems to be deployed will perform as intended in an operational environment. Systems testing and integration management: A rigorous and efficient test and evaluation program provides early knowledge of developmental and operational issues. Correcting these issues early can mitigate risks of cost overruns and schedule slippages, and can ultimately contribute to delivery of effective and suitable systems in a timely manner. System integration ensures that all the “pieces” of the system can work together to realize the program’s goals. It involves the collaborative planning and execution of test phases and events to provide shared data in support of independent analysis, evaluation, and reporting by all stakeholders. The majority of the 18 selected MAIS programs that we reviewed had experienced changes in their planned cost estimates and in their planned schedule estimates when comparing the first acquisition program baseline to the most recent acquisition program baseline estimate. These cost estimate changes ranged from a decrease of $1,466.2 million (-39 percent) to an increase of $1,625.7 million (469 percent), and schedule estimate changes ranged from a slippage of 13 years and 9 months to no change. Further, nine of the programs had met all of their technical performance targets, while four of the programs had partially met the performance targets. The remaining five programs had not yet conducted testing activities. As a result, it is too early to report on the status of technical performance targets. Table 2 shows the extent of changes in planned cost and schedule estimates for each of the selected MAIS programs we reviewed since the first program baseline estimate and the number of technical performance targets met. See appendix II for the detailed profiles of each program. Of the 18 MAIS programs we reviewed, 16 had experienced changes in their planned cost estimates. Specifically, five of the 16 programs had cost decreases that ranged from $58.5 million (-1.5 percent) for the Army’s Global Combat Support System, Increment 1 (GCSS-A Inc 1) program to $1,466.2 million (-39 percent) for the Air Force’s Base Information Transport Infrastructure Wired (BITI Wired) program. Officials reported various reasons for the decreases in planned cost estimates. For example, the Defense Health Agency’s integrated Electronic Health Record Increment 1 (iEHR Inc 1) program reported that the 70 percent cost decrease was due to a reduction in the program’s scope. Officials for the Air Force’s BITI Wired program reported that the 39 percent cost decrease was due to early assumptions about risks not materializing. For example, when the program had solicited bids for the contract, they included risk factors that never materialized. In addition, 11 of the 16 programs had experienced cost increases. These increases ranged from $2.6 million (1 percent) for the Defense Logistics Agency’s Defense Agencies Initiative Increment 2 (DAI Inc 2) program to $1,625.7 million (469 percent) for the Navy’s Common Aviation Command and Control System Increment 1 (CAC2S Inc 1) program. Among the 11 programs that reported an increase, the average increase was $457.2 million. Program officials reported a variety of reasons for the increases in planned cost estimates. For example, officials for the Army’s Tactical Mission Command (TMC) program attributed its 159 percent increase to, in part, higher than expected costs for research and development testing. In addition, officials for the Air Force’s Defense Enterprise Accounting and Management System Increment 1 (DEAMS Inc 1) program attributed a 9 percent cost increase to the program’s growth in scope and the addition of software upgrade enhancements. Further, officials for the Navy’s CAC2S system attributed cost increases of 469 percent to the program’s early developmental challenges, which included program scope growth and restructuring. Two programs had not experienced any changes in their cost estimates: These include the Defense Health Agency’s Department of Defense Healthcare Management System (DHMSM) program and the Air Force’s Integrated Strategic Planning and Analysis Network Increment 4 (ISPAN Inc 4) program. Of the 18 selected MAIS programs we reviewed, 14 programs had experienced changes in their planned schedule estimates. These changes consisted of schedule slippages that ranged from 2 months for the Air Force’s BITI Wired program to 13 years and 9 months for the Navy’s CAC2S program. Program officials reported various reasons for the changes in their planned schedule estimates. For example, officials for the Navy’s CAC2S program attributed the schedule slippages to the addition of new requirements and program restructuring. Officials for the Air Force’s AOC- WS Inc 10.2 program reported that the 3 years of schedule slippages were due to a combination of test problems and system maturity issues. In addition, officials for the Air Force’s JMS Inc 2 system cited a larger than expected number of discrepancies in testing that had caused schedule delays totaling 2 years and 10 months. In turn, the program delayed completion of its Milestone C and Full Deployment Decision by more than 1 year beyond the original estimate. Four MAIS programs experienced no change to their planned schedule estimates—the DAI Inc 2, DHMSM, IPPS-A Inc 2, and the ISPAN Inc 4 programs. Among other information, DOD uses key performance parameters as a metric to report on each program’s progress toward meeting the system technical performance targets. This information includes a description of the performance characteristics, the objective and threshold value for each target and, importantly, whether the target has been met in demonstrating performance. For example, one program identified “network readiness and exchange of information” as a technical performance target. The program also established an objective value for this target as successfully processing 98 percent of the information into the system, and established an threshold value for this target as successfully processing 95 percent of the information into the system. Among the 18 selected MAIS programs we reviewed that had established technical performance targets, nine had met all of their defined targets. As an example of technical performance targets that had been met, the Army’s GCSS Inc 1 program reported in the 2016 Major Automated Information System Annual Report that both of its technical performance targets had been met. These targets related to the readiness of the system to fully support operational activities and information exchanges in the architecture (referred to as Net Ready) and compliance with the operational activities that represent the system’s mission critical functions (referred to as Mission Critical Functions). As another example, the Defense Health Agency’s iEHR Inc 1 program reported that all 3 performance targets had been met. These targets related to ensuring that end user devices display the same information, context, and display of the patient, expediting user access to the applications, and ability of users to move from one device to another. Further, of the 18 MAIS programs, four programs—the Air Force’s BITI- Wired and DEAMS Inc 1 programs, Navy’s CANES and Defense Information Systems Agency’s Teleport Gen 3 programs—had partially met their technical performance targets. For example, BITI-Wired had partially implemented all 4 of its performance targets—related to readiness of the network, availability of redundant features such as dual power supplies and spares, reliability of systems, and the installation of support equipment at all 178 sites. According to program officials, support equipment had been installed at 167 of the 178 total sites and the program expects all 4 performance targets to be met in September 2017, the planned full deployment date. As another example, the Air Force’s DEAMS Inc 1 program officials reported that 6 of 9 targets had been met. For the 3 performance targets that had not been met, officials reported that processing issues had prevented them from meeting these targets, which included establishing accurate fund balances, reporting in a timely manner, and ensuring readiness of the network. As of June 2016, program officials reported they were working to address these issues. Finally, five programs—IPPS-A Inc 2, AOC-WS Inc 10.2, ISPAN Inc 4, JMS Inc 2, and DHMSM—were in the early stages of the system development and had not begun testing. As a result, it is too early to determine the status of performance targets because information on the status of technical performance is not yet available. According to the Software Engineering Institute’s Capability Maturity Model® Integration for Acquisition (CMMI- ACQ), leading practices for managing an IT program include risk management, and requirements management. Effective requirements management includes establishing an agreed-upon set of requirements and ensuring traceability between requirements and work products. Further, changes to requirements should be managed in collaboration with stakeholders for the acquisition of systems within their established cost, schedule, and delivery of the expected product. Likewise, an effective risk management process includes identifying problems before they occur, so that risk-handling activities may be planned and invoked, as needed, across the life of the project in order to mitigate the potential for adverse impacts. Further, regarding systems testing and integration, other leading practices from the Institute of Electrical and Electronics Engineers, Incorporated (IEEE) and DOD policy guidance (DOD Instruction 5000.02, Defense Acquisition Guidebook, and Test & Evaluation Management Guide) state that roles and responsibilities should be established and test-related plans, schedules, and reports should be developed to better manage test and integration activities. All five MAIS programs that we selected for an in-depth review had fully or partially implemented leading practices for managing risk, requirements, and systems testing and integration. Table 3 shows the extent to which these programs had implemented the practices for requirements management, risk management, and systems testing and integration management. We assessed one program as “not applicable” because it was initiated after a limited production decision point (post- Milestone C) and, therefore, had not implemented certain practices. Leading requirements management practices include establishing an agreed-upon set of requirements, ensuring traceability between requirements and work products, and managing any changes to the requirements in collaboration with stakeholders. Such leading practices help organizations to better manage the design, development, and delivery of systems within established cost and schedule time frames. These practices include developing an understanding with the requirements providers on the meaning of the requirements, obtaining commitment to requirements from project participants, managing changes to requirements as they evolve during the project, maintaining bidirectional traceability among requirements and work, ensuring that project plans and work products remain aligned with requirements. Three of the five programs had fully implemented the requirements management practices, while the other two had partially implemented some practices. Air Force— Air and Space Operations Center-Weapon System Increment 10.2 (AOC-WS Inc 10.2) The Air Force had fully implemented four, and partially implemented one, of the five requirements management practices for the AOC WS Inc 10.2 program. For example, the program developed an understanding with requirements providers on the meaning of the requirements and established a requirements working group that oversees the development and management of program requirements. In addition, the AOC-WS Inc 10.2 program obtained commitment to requirements from project participants through the DOD configuration control board process, in which requirements were reviewed and, upon acceptance, approved, documented, and tracked. The program also managed changes to requirements as they evolved during the project; however, although changes to requirements were documented and maintained, the program did not provide evidence of its rationale for the changes. The AOC-WS Inc 10.2 program has since completed its acquisition phase and, therefore, providing a rationale for changes made during the requirements development phase would not impact the program moving forward. Air Force— Base Information Transport Infrastructure Wired (BITI Wired) The Air Force had fully implemented the two applicable requirements management practices for the BITI Wired program. Specifically, the program had developed an understanding with requirements providers on the meaning of the requirements and had managed changes to requirements as they evolved during the project. The BITI Wired requirements management plan established the criteria for the evaluation and acceptance of the requirements. On the other hand, three of the five requirements—obtain commitment to requirements from project participants; maintain bidirectional traceability among requirements and work products; and ensure that project plans and work products remain aligned with requirements—were not applicable for the BITI Wired Program. This was due to the Air Force beginning the program after Milestone C. Air Force— Joint Space Operations Center, Mission System Increment 2 (JMS Inc 2) The Air Force had fully implemented the five requirements management practices for the JMS Inc 2 program. Specifically, the program had developed an understanding with requirements providers on the meaning of the requirements. For example, the program’s Requirements Engineering and Management Plan established criteria for distinguishing appropriate requirements and for evaluating and accepting requirements. In addition, changes to requirements were managed as they evolved during the project. For example, requirements change requests had been sent through the Lead Command, which validated them and sent them to the Program Management Office for approval. Lastly, the program maintained requirements traceability from source requirements to lower- level requirements to ensure that all source requirements had been completely addressed. Specifically, the program had established processes for managing the program’s Requirements Traceability Matrix database, which included activities to enter approved requirements, sequence traceability changes, update development, test status, and report data. Army— Global Combat Support System-Army Increment 1 (GCSS-A Inc 1) The Army had partially implemented all five requirements management practices for the GCSS-A Inc 1 program. Officials reported that, although the program had progressed to the sustainment phase, and its requirements activities had been completed since 2014, the Configuration Steering Board review process had been used to manage, oversee, and approve requirements as they evolved during the project. The program had developed a Software Development Overview Schedule that provided a general overview of their software development life cycle process. Nevertheless, an approved and signed Requirements Management Plan at program initiation had not been implemented. Such a plan would have been important to ensuring an understanding with the requirements providers of the meaning of the requirements. Further, the program’s Release Management Plan included a section on the requirements process and references to a central information repository in the Requirements Traceability database. However, the requirements traceability matrix provided did not contain all relevant information such as “priority.” These important management tools could have provided the program with the discipline, structure, and oversight typically associated with successful organizations that apply this practice. However, since the GCSS-A Inc 1 program has completed its acquisition phase, fully implementing these practices now would not impact the program moving forward. Army—Integrated Personnel and Pay System—Army Increment 2 (IPPS-A Inc 2) The Army had fully implemented the five requirements management practices for the IPPS-A Inc 2 program. Specifically, the program had developed an understanding with requirements providers on the meaning of the requirements and had managed changes to requirements as they evolved during the project. For example, the program’s Requirements Management Plan described the criteria and process through which requirements are evaluated and accepted. In addition, the IPPS-A Requirements/Configuration Control Board Charter requires that all changes be vetted through senior governance boards, such as the Council of Colonels and General Officer’s Steering Committee. Further, the program maintained bidirectional traceability of requirements. Specifically, a history for all requirements changes was maintained within a repository for requirements that were managed by the Application Lifecycle Management Tool. According to leading practices, an effective risk management process identifies potential problems before they occur, so that risk-handling activities may be planned and invoked, as needed, across the life of the project in order to mitigate the potential for adverse impacts. These practices include determining risk sources and categories; defining parameters used to analyze and categorize risks and to control the risk management effort; establishing and maintaining the strategy to be used for risk identifying and documenting risks; evaluating and categorizing each identified risk using defined risk categories and parameters, and determining its relative priority; developing a risk mitigation plan in accordance with the risk monitoring the status of each risk periodically and implementing the risk mitigation plan as appropriate. Three of the five programs had fully implemented the risk management practices, while two had partially implemented some practices. The Air Force had fully implemented four, and partially implemented three, of the seven risk management practices for the AOC WS Inc 10.2 program. For example, the program had determined risk sources and categories used to examine and oversee changes that impact the project, as described in the program’s Risk Management Plan. The program also had defined parameters used to analyze and categorize risks and to control the risk management effort. Specifically, the Risk Management Plan evaluated and assigned values to each risk by calculating the probability and consequence of a risk, as well as its likelihood. The AOC-WS Inc 10.2 program had partially implemented the key management practice to identify risks that could negatively affect work efforts. For example, although the program identified risk in specific risk categories, it did not identify risks that were outside the normal scope of the project. In addition, while the program had developed criteria for evaluating and qualifying risk likelihood and severity levels, it did not develop an overall risk mitigation plan in accordance with the risk management strategy. This occurred because the program underwent a critical change and, according to program officials, this critical change altered the management of the program due to its complexity. Until an overall risk mitigation plan is developed, the program will not be able to effectively manage risk mitigation and contingency planning activities. The Air Force had fully implemented all seven of the risk management practices for the BITI Wired program. Specifically, the program had defined parameters used to analyze and categorize risks and to control the risk management effort. For example, the program evaluated and assigned values to each risk by calculating the probability and consequence of a risk as well as its likelihood and consequence. The program had also identified and documented risks that could negatively affect work efforts. For example, the Risk Management Plan established risk conditions, thresholds, and probabilities of occurrence. In addition, the BITI Wired program monitored the status of each risk periodically and implemented risk mitigation plans as appropriate through the use of a risk management tool and risk board. The Air Force had fully implemented all seven of the risk management practices for the JMS Inc 2 program. Specifically, the program had defined parameters used to analyze and categorize risks and to control the risk management effort. For example, as detailed in the program’s Risk and Opportunity Management Plan, risk ratings, including likelihood and consequence of occurrence, were established through the risk matrix; this provided a consistent, standardized method for risk analysis. In addition, defined thresholds were used to prioritize risks and trigger management action. Further, the program had developed risk management plans establishing a strategy used for risk management. Specifically, risks were monitored and risk status was reviewed in the issue database as well as through meetings used to report on programmatic metrics. Standard operating procedures had also been developed to manage risk, including one to manage risk identification, analysis, mitigation, and tracking. The Army had fully implemented five, and partially implemented two, of the risk management practices for the GCSS-A Inc 1 program. Specifically, the program had a Risk and Opportunity Management Plan that identified risk sources and categories. Further, in order to control the risk management effort, risks were categorized and grouped according to defined risk categories in the program’s Risk Management Plan. Further, identified risks were evaluated using defined risk parameters. For example, the Risk Register identified risk consequence and probability. On the other hand, the program did not define thresholds or bounds for mitigation or management action. According to leading practices, the risk management effort can be prioritized, controlled, and managed better by defining thresholds and categorizing risks for mitigation/management action. Although program officials stated that they were in the process of continuing to optimize their risk management efforts, and were updating their standard operating procedures, this key leading practice has not yet been implemented. Until the program has, and is guided by, updated procedures, it will not be positioned to ensure that risks can be effectively prioritized, controlled, and managed. The Army had fully implemented all seven of the risk management practices for the IPPS-A Inc 2 program. Specifically, the program’s Risk Management Plan provided guidelines that determined risk sources and categories. These risks were categorized in the Risk Management Tool based on the “Risk Taxonomy” found in the Risk Management Plan. Specifically, the risks were assigned to three main categories—technical, programmatic, and external risks. The program also ensured potential risks were identified and documented. For example, the Defense Information Systems Agency, which hosts the two IPPS-A Defense Enterprise Computing Centers (in St. Louis, MO, and Ogden, UT), conducted assessments to ensure the sites were in compliance with environmental controls. Further, the program developed risk mitigation plans that identified the person or group responsible for addressing each risk through the Risk Register. According to leading practices for managing systems testing and integration activities, roles and responsibilities should be established and test-related plans, schedules, and reports should be developed. Doing so provides a baseline by which to measure progress so that appropriate corrective actions can be taken when the program’s performance deviates from the plans. These practices include establishing roles and responsibilities to manage testing and developing test-related plans, schedules, and reporting. Four of the five programs had fully implemented practices for systems testing and integration. The Air Force had fully implemented both systems testing and integration practices for the AOC-WS Inc 10.2 program. The program established roles and responsibilities to manage testing/integration activities and formulated test related plans, schedules, and reports. Specifically, the program’s Integrated Test Team Charter and the Test and Evaluation Master Plan (TEMP) established organizational and individual roles and responsibilities for testing and integration. Further, the program developed a test plan that included a test and evaluation strategy and schedule. The Air Force had fully implemented both systems testing and integration practices for the BITI Wired program. In this regard, the program had established testing roles and responsibilities. The program also formulated all applicable test related plans, schedules, and reports. For example, the BITI Wired Program developed testing plans and schedules for each base installation site. The Air Force had fully implemented one, and partially implemented one, of the systems testing and integration practices for the JMS Inc 2 program. For example, the program had developed a Test and Evaluation Master Plan that provided the framework and strategy for planning and executing its testing and evaluation activities. Although the program had established roles and responsibilities to manage testing and integration activities, a chief developmental tester to oversee testing activities had not been identified. Program officials stated that it had been difficult to fill this position due to the lack of qualified personnel. Until this position is filled, it may be difficult to manage and mitigate risks during development, verify that products are compliant with operational requirements, and inform decision makers throughout the program’s life cycle. The Army had fully implemented both systems testing and integration practices for the GCSS-A Inc 1 program. In this regard, the program had established and documented roles and responsibilities for all testing and evaluation activities, including identifying a chief developmental tester. In addition, a GCSS-A Inc 1 Test and Evaluation Master Plan had been developed, which described the total testing and evaluation planning from component development through operational testing and evaluation. The program also had developed test plans, such as the Capstone Test Plan and Systems Engineering Plan, which further detailed the program’s testing and evaluation strategy. The Army had fully implemented both systems testing and integration practices for the IPPS-A Inc 2 program. In this regard, the program had established testing roles and responsibilities and formulated test plans, schedules, and reports, where applicable. Specifically, the Increment II Test and Evaluation Working Integrated Product Team Charter was established to provide guidance in overseeing testing activities. In addition, a chief developmental tester had been appointed to serve as the focal point for all testing and evaluation efforts. Further, the IPPS-A Inc 2 program developed a Test and Evaluation Master Plan, which described the overall testing and evaluation plans from system development through operational testing and evaluation for all releases. MAIS programs are important for sustaining DOD’s key operations in communications, business, and command and control, and provide the department with access to information to organize, plan, direct, and monitor mission operations. Therefore, it is vital that these programs follow leading practices to ensure success. While a number of leading practices for risk, requirements, and systems testing and integration had been fully or partially implemented by five programs that we reviewed in- depth—the Air Force’s AOC WS Inc 10.2, BITI Wired and JMS Inc 2 programs; and the Army’s GCSS-A Inc 1 and IPPS-A Inc 2 programs— three programs lacked practices to better manage the development of systems. The program office for GCSS-A Inc 1 did not have standard operating procedures for defining thresholds or bounds to manage risks, thus putting at risk the program’s ability to effectively prioritize, control, and manage for mitigation and management actions. In addition, the program office for AOC-WS Inc 10.2 did not have an overall Risk Mitigation Plan to guide the implementation of individual risk mitigation and contingency planning activities. An overall risk plan is necessary to help ensure that individual efforts are managed in a cohesive and repeatable way. Further, the program office for JMS Inc 2 had not assigned a chief developmental tester, an important role to ensure that risks are managed and mitigated effectively throughout testing and integration. Unless this position is filled, it will be difficult for the program to effectively manage risks and verify compliance with certain operational requirements throughout the acquisition. To help improve the management of DOD’s MAIS programs, we are recommending that the Secretary of Defense take the following three actions: direct the Secretary of the Army to direct the program manager for GCSS-A Inc 1 to establish standard operating procedures for managing risks that include guidance for establishing thresholds and bounds for key risk areas. direct the Secretary of the Air Force to direct the program manager for AOC-WS Inc 10.2 to develop an overall risk mitigation plan to guide the implementation of individual risk mitigation and contingency plan activities. direct the Secretary of the Air Force to direct the program manager for JMS to appoint a chief developmental tester to oversee systems testing and integration activities. DOD provided written comments on a draft of this report, which are reproduced in appendix III. In its comments, the department agreed with all three of our recommendations. Further, the department provided evidence that it had fully addressed one of the recommendations and it described plans and actions for addressing the remaining two recommendations. Regarding our recommendation that the GCSS-A Inc 1 program establish standard operating procedures for managing risks, DOD provided evidence that the Army had updated its risk management plan. Specifically, the Army’s GCSS-A Inc 1 program manager amended the Risk Management Plan for Project Manager Army Enterprise Systems Integration Program on March 1, 2017. The plan now describes how risk prioritization occurs; it requires that risk owners work with the risk management officer to define appropriate trigger conditions in order to ensure a mitigation plan can be initiated at the appropriate point. By taking these actions to implement this recommendation, the program should be in a better position to more effectively manage risks. DOD also concurred with our recommendation to develop an overall risk mitigation plan to guide the implementation of individual risk mitigation and contingency planning activities on the AOC WS Inc 10.2 program. Toward this end, the Air Force reported that it plans to restore the program’s risk management practices to include weekly meetings to review program risks and publish a risk management plan in April 2017. Once implemented, these actions should help the Air Force to better ensure that individual risk mitigation and contingency plan activities are conducted in a consistent and repeatable way. Further, DOD concurred with our recommendation that JMS appoint a chief developmental tester to oversee systems testing and integration activities. In this regard, the Air Force reported that it had identified a qualified candidate to fill the Chief Developmental Tester position. According to the Air Force, the program expects the new test chief to be in place in July 2017. Once the position is filled, the program should be better able to oversee important systems testing and integration activities. We are sending copies of this report to the appropriate congressional committees; the Secretary of Defense; the Secretary of the Army; the Secretary of the Air Force; and other interested parties. This report also is available at no charge on the GAO website at http://www.gao.gov. Should you or your staffs have any questions on information discussed in this report, please contact me at (202) 512-4456 or harrisc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV. The National Defense Authorization Act for Fiscal Year 2012 mandated that we select, assess, and report on Department of Defense’s (DOD) major automated information system (MAIS) programs annually through March 2018. This report is the fifth in our series of annual assessments. Our objectives were to: (1) describe the extent to which selected MAIS programs have changed their planned cost and schedule estimates and met technical performance targets and (2) assess the extent to which selected MAIS programs have used leading IT acquisition practices, including requirements and risk management, and systems testing and integration. To address the first objective, we used DOD’s official list of 35 MAIS programs as of May 1, 2016, to establish a basis for selecting programs. We selected 18 of 35 MAIS programs based on our criteria that programs must be unclassified and have a first acquisition program baseline that could be used as a reference point for evaluating cost, schedule, and technical performance characteristics. For each selected program, we asked relevant DOD officials to complete a data collection instrument with questions on estimated cost, schedule, and technical performance goals, including the latest program status in meeting those estimated goals. Using the responses from the data collection instrument, we then compared each program’s first acquisition program baseline cost estimate to the latest estimate to determine the extent to which planned program costs had changed. Similarly, to determine the extent to which these programs had changed their planned schedule estimates, we compared each program’s first acquisition program baseline schedule estimate to the latest schedule. To determine whether the selected programs met their performance targets, we reviewed the most recent program management reports and examined each program’s self-identified system performance targets and the status of their progress. We also reviewed additional information on each program’s cost, schedule, and performance, including program documentation, such as DOD’s MAIS annual and quarterly reports, acquisition program baselines, system test reports, and our prior reports. We then aggregated and summarized the results of these analyses across the programs, and developed individual profiles for each program. To address the second objective, we started with the list of 18 MAIS programs that were selected specifically for the first objective. From this list, we identified programs for a more in-depth review based on two criteria: the programs selected must represent at least two military services, and the programs must have not been evaluated by us within the last year. Using this criteria, we identified 5 programs from two military services: the Army’s Global Combat Supply System Increment 1 (GCSS-A Inc 1) and Integrated Personnel and Pay System Increment 2 (IPPS-A Inc 2); and the Air Force’s Air and Space Operations Center– Weapon System Increment 10.2 (AOC-WS Inc 10.2), Base Information Transport Infrastructure Wired (BITI Wired), and Joint Space Operations Center, Mission System Increment 2 (JMS Inc 2). Based on prior work evaluating MAIS programs, we identified three management practice areas—requirements management, risk management, and systems testing and integration that are applicable to these programs. For evaluating how the selected five programs applied practices in the risk and requirements management areas, we identified key practices related to these areas from the Software Engineering Institute’s Capability Maturity Model® Integration for Acquisition (CMMI- ACQ). Specifically, for requirements management, we evaluated each of the five programs against the following 5 practices: develop an understanding with the requirements providers of the meaning of the requirements, obtain commitment to requirements from project participants, manage changes to requirements as they evolve during the project, maintain bidirectional traceability among requirements and work, and ensure project plans and work products remain aligned with requirements. In doing so, we analyzed program requirements documentation against the practices. This documentation included requirements management plans, traceability matrices, requirements change forms, technical performance assessments, and requirements board meeting minutes. We also interviewed program officials to obtain additional information about their requirements management practices. For the risk management area, we evaluated each of the five programs against the following 7 practices: determine risk sources and categories; define parameters used to analyze and categorize risks and to control the risk management effort; establish and maintain the strategy to be used for risk management; identify and document risks; evaluate and categorize each identified risk using defined risk categories and parameters, and determine its relative priority; develop a risk mitigation plan in accordance with the risk management monitor the status of each risk periodically and implement the risk mitigation plan as appropriate. In doing so, we analyzed each program’s key documents, such as risk management plans, risk charts, and other artifacts, to determine the extent to which these practices had been applied. We also interviewed program officials to obtain additional information about their risk management practices. For the systems testing and integration area, we evaluated each of the five programs against the following 2 practices: establish roles and responsibilities to manage testing and integration develop test-related plans, schedules, and reporting. In this regard, we evaluated the test manager’s roles and responsibilities, as well as test plans, schedules, and test reports to determine the extent that these practices had been applied. We also conducted interviews with program officials to obtain additional information about their systems test and integration practices. Further, in determining the extent to which each of the five MAIS programs had implemented the three practice areas—requirements management, risk management, and systems testing and integration—we rated a practice as “fully implemented” when all of the practices for the area had been implemented. Similarly, when a program had implemented at least one of the practices within the area, we rated the practice as “partially implemented” to indicate the practice had been partially implemented. For a MAIS program that did not conduct activities for any of the practices because the program had been initiated after a limited production decision point (post-Milestone C), we assessed the practice as “not applicable” because such programs do not apply practices to develop, manage requirements, or test the systems. To assess the reliability of the data that we used to support the findings in this report, we corroborated program office responses with relevant program documentation and interviews with agency officials. We found no data reliability issues and determined that the data used in this report were sufficiently reliable for our reporting purposes. We have also made appropriate attribution indicating the sources of the data. We conducted this performance audit from April 2016 to March 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. This section contains profiles of the 18 selected MAIS programs that we analyzed to determine whether they had changed their planned cost and schedule estimates and met performance measures. Each profile presents data on the program’s purpose and status, its latest cost and schedule estimates compared to the first acquisition program baseline and system performance data. The first page of each two-page profile provides (1) a description of the program’s purpose, (2) essential details, such as the responsible DOD component, program owner, prime contractor, the total number of active contractors supporting the program; and (3) program costs (in then-year dollars), comparing the program’s latest life-cycle cost estimate (separated into acquisition and operations and maintenance costs) to its first acquisition program baseline (subsequent acquisition program baselines that may have been established are not identified). The profile also provides information on the program’s status, costs, schedule, and performance. In the program status section, we discuss recent milestones achieved and acquisition events. In the cost section, we identify the extent to which the program’s life- cycle cost estimate has changed from its first acquisition program baseline, as well as any notable causes for any changes. In the schedule section, we discuss the extent to which the program’s schedule has changed from its first acquisition program baseline and causes for significant changes. In the performance section, we identify the extent to which each program has met its established measures. These performance ratings represent point-in-time assessments as reported by the program. System performance targets are rated as “met” when the program fully met all of its key performance parameters. System performance was rated as “did not fully meet” when a program did not pass system testing. System performance was rated as “not yet applicable” when the system had not yet been tested because the program was not yet at the appropriate acquisition phase. The AOC-WS Inc 10.2 program is expected to provide system hardware, software, technical documents, and technology refresh to ensure it remains a viable weapons system. The program will enable personnel at selected air and space operations centers to better plan, execute, and assess theater-wide air and space operations. It will replace the currently fielded AOC 10.1 system and provide additional capabilities, such as dynamic planning and execution; data management; information assurance; predictive battlespace awareness; and airspace management. Program essentials (as of October 2016) DOD component: Air Force Program owner: Air Combat Command Prime contractor: Northrop Grumman, Mission Systems Total number of contractors (as of May 2016): 10 As of June 2016, the AOC-WS Inc 10.2 program was in the engineering and manufacturing development phase of acquisition in the Defense Acquisition Management System. In March 2016, the program declared a critical change because of significant cost and schedule growth due to schedule and technical difficulties in reaching Milestone C, which is the key juncture for approval to move into limited production. According to the program office, major factors contributing to the cost and schedule delays were the prime contractor (1) underestimating the software development effort; (2) insufficiently allocating manpower; and (3) underestimating the end-to-end testing capability. As of June 2016, the AOC-WS Inc 10.2 program’s total life-cycle cost was $5,642.0, an increase of $56.8 million (1 percent) from the original estimate of $5,585.2. The estimated total acquisition cost is $706.0 million, an increase of $243.3 million (53 percent) from the original estimate of $462.7 million. The AOC-WS Inc 10.2 program experienced significant schedule delays since establishing the original schedule in 2007. Specifically, Milestone C was estimated to be completed on July 2015, but, as of September 2016, the program had not reached the milestone. According to program officials, the program was undergoing a critical change that will result in a 3-year schedule delay. The AOC-WS Inc 10.2 program was still early in development and system performance data was not available at the time of our review. The BITI Wired program is to upgrade the enterprise network and provide the core cyber infrastructure. The upgrade includes wired and wireless systems, and network cables and servers for more than 150 active duty, Air Force Reserve, and Air National Guard bases. The program is expected to update the fixed local area network and all necessary information transport infrastructure to support current and future communications needs of the Air Force and Joint Command warfighter. This includes enabling base level data, voice, video, imagery, sensor, and telemetry requirements. Program essentials (as of October 2016) DOD component: Air Force Program owner: Air Force Space Command (AFSPC) Prime contractors: General Dynamics, Harris Corp, Lockheed Martin, Telos Corp, Three Wire Systems, and Red River for the procurement contracts and Affigent and ThunderCat for the operations and support contracts Total number of contractors (as of May 2016): 8 The BITI-Wired program achieved a full deployment decision on February 18, 2014, and expects to reach full deployment in September 2017. The program completed 167 out of 178 turnkey network upgrades and expects to remain within the cost, schedule, and performance thresholds. As of July 2016, the BITI Wired program’s total life-cycle cost estimate was $2,323.7 million, a decrease of $1,466.2 million (-39 percent) from the original estimate of $3,789.9 million. The decrease was due to more accurate cost estimates being provided by vendors. Specifically, initial estimates included large risk factors because, at the time, the government could not provide vendors a definitive list of items. As a result, vendors’ earlier estimates reflected the potential for risks in their estimates. The new list of refined equipment resulted in reduced costs. In addition, while officials cited insufficient human resources as a potential risk to cost as the program moves forward, they fully expect to stay within the cost estimates. The BITI Wired program’s full deployment decision was 2 months’ behind schedule. However, program officials report they are on track to meet full deployment in September 2017. Did not fully meet all of the system performance targets The BITI-Wired program had partially implemented all 4 of performance targets—net ready, availability, support, and reliability. According to program officials, installation of equipment had been installed at 167 of the 178 sites. The program expects to report all 4 performance targets as fully met in September 2017, the full deployment date, when the equipment has been installed at all sites. The CANES program is expected to consolidate and standardize the Navy’s existing network infrastructures and services by reducing and eliminating existing standalone afloat (i.e., surface ships and submarines) networks, providing a technology platform that can rapidly adjust to changing warfighting requirements, and reducing the hardware footprint on 259 afloat and maritime operations center platforms. Program essentials (as of October 2016) DOD component: Navy Program owner: C4I Prime contractors: Northrop Grumman, Serco, BAE Systems, DRS Laurel Technologies, General Dynamics, CGI Federal, Global Technical Systems Total number of contractors (as of May 2016): 82 The CANES program had moved past the full deployment milestone. The program is expected to have all systems fully fielded by September 2024. According to program officials, risks remain that could result in delaying the fielding of CANES, thereby increasing risk to the program’s ability to achieve full deployment on schedule. As of July 2016, the CANES program’s total life-cycle cost estimate was $11,910.6 million, a decrease of $830.3 million (-7 percent) from the original estimate of $12,740.9 million. In February 2016, in response to a full deployment decision, the life-cycle cost estimate was updated to reflect reductions in procurement costs, the program’s technology refresh schedule, and manpower costs. The CANES program experienced a significant delay in reaching a full deployment decision. In June 2014, the program slipped the objective date for full deployment decision 2 years, from December 2013 to December 2015. A full deployment decision was achieved in October 2015, and full deployment is estimated for September 2024. According to program officials, an unstable funding environment is the largest risk to achieving full deployment. Did not fully meet all of the system performance targets The CANES program had not met all 9 of the performance targets. Specifically, of the 9 performance targets, 8 have been met, while 1 performance target was partially met—the net ready performance target. According to the program office, this performance target was only partially met because testing for afloat core services has been deferred. The CAC2S Inc 1 program is expected to provide an integrated and coordinated modernization effort for the equipment of the Marine Air Command and Control System. The system is expected to provide enhanced capability for three air defense centers to support aviation employment in joint, combined, and coalition operations. It will provide tactical situational display, information management, sensor and data link interface, and operational facilities for planning and execution of Marine Aviation missions. Program essentials (as of October 2016) DOD component: Navy Program owner: USMC, Program Executive Officer, Land Systems Prime contractors: General Dynamics, C4 Systems, Inc. Total number of contractors (as of May 2016): 5 As of March 2016, Phase 1 of the CAC2S Inc 1 program was in the operations and sustainment acquisition phase and Phase 2 was in the production and deployment phase. The program has been rebaselined once following a critical change that occurred in 2009 due to the program’s inability to meet the initial operational capability date. As of March 2016, the CAC2S Inc 1 program’s total estimated life-cycle estimate was $1,972.7 million, an increase of $1,625.7 million (469 percent) from the original life-cycle estimate of $347 million. Similarly, the program’s current total acquisition estimate is $882 million, an increase of $708.7 million (409 percent) from the original total acquisition estimate of $173.4 million. As we reported in March 2016, factors that contributed to the cost increase were early challenges in estimating costs due to program scope growth and need for restructuring. Program officials noted funding instability as a potential area of risk in meeting cost estimates. As of October 2015, the CAC2S Inc 1 program had a 13-year, 9-month slippage in their full deployment date due to a major change in project scope and restructuring of the project. Met all of the system performance targets CAC2S Inc. 1 had met both of the technical performance targets which were (1) net ready, and (2) data fusion. The DAI Inc 2 program’s mission is expected to deliver auditable, accurate, timely, authoritative financial data to support DOD’s goal of standardizing financial management practices, improving financial decision support, and supporting audit readiness. Defense agencies use more than ten different non-compliant financial management systems supporting diverse operational functions and the warfighter in decision making and financial reporting. Program essentials (as of October 2016) DOD component: Defense Logistics Agency Program owner: OSD (C) Prime contractor: Credence Management Solutions, LLC; TeraThink Corp; IBM; CACI, Inc; Northrop Grumman Systems Corp; CACI ISS, Inc; Amyx, Inc Total number of contractors (as of May 2016): 25 As of March 2016, the DAI Inc 2 program was in the limited deployment phase of the business capability life cycle. According to a 2016 MAIS annual report, the program was on track to remain within the schedule, cost, and performance thresholds identified in the original estimate. According to program officials, the program was rebaselined to adjust release 2 objective and threshold dates for limited fielding to accommodate the time frame for resolving a bid protest. Notwithstanding this additional time for the bid protest, the program achieved the original acquisition program baseline objective date for this event. As of March 2016, the DAI Inc 2 program’s total life-cycle estimate was $868.8 million, an increase of $2.6 million (<1 percent) from the original life-cycle cost estimate of $866.2 million. Similarly, the program’s current total acquisition estimate was $184.9 million, an increase of $2.6 million (1 percent) from the original acquisition estimate of $182.3 million. However, these cost increases were within the planned cost objective estimates. Officials cited funding instability as a potential risk to cost as the program moves forward. The DAI Inc 2 program had met the first acquisition program baseline objective schedule estimate for limited fielding in September 2015. Officials noted they were on schedule to stay within the threshold date of April 2017 for a full deployment decision. Did not fully meet all of the system performance targets According to the 2016 MAIS annual report, the DAI Inc 2 program had not met any of the five of the technical performance targets which are (1) financial system performance, (2) business enterprise architecture compliance, (3) support net centric DOD military operations, (4) managed in network, and (5) effectively exchange information. The DHMSM Program is expected to provide a configurable and scalable modernized electronic Health Record System. It will replace DOD’s legacy healthcare systems including the Armed Force’s Health Longitudinal Technology Application, Composite Health Care System (inpatient), and most components of the Theater Medical Information Program-Joint program, with a commercial Off-The-Shelf electronic health record system. It is expected to address the current state of the Military Health System, where multiple legacy systems and data stores, developed over decades, are in need of modernization. Program essentials (as of October 2016) DOD component: Defense Health Agency Program owner: PEO Defense Healthcare Management Systems Prime contractor: Leidos Total number of contractors (as of May 2016): 35 Cost, schedule, and performance summary Not applicable: The program obtained its first baseline on May 3, 2016. Due to the recent establishment of the program’s first baseline, a history of cost, schedule, and technical performance has not been established. Therefore, the cost, schedule, and technical performance is not discussed. The DEAMS Inc 1 program is expected to provide the Air Force with an entire spectrum of financial management capabilities, including collections; commitments and obligations; cost accounting; general ledger; funds control; receipts and acceptance; accounts payable and disbursement; billing; and financial reporting. The DEAMS program is also intended to be a key component of DOD’s solution for achieving fully-auditable financial statements by September 30, 2017, as required by the National Defense Authorization Act for Fiscal Year 2010. Program essentials (as of October 2016) DOD component: Air Force Program owner: Air Force Financial Management Prime contractors: Accenture and Kearney Total number of contractors (as of May 2016): 29 As of June 2016, the DEAMS Inc 1 program was in the limited deployment phase. In March 2016, the program was approved for limited deployments at smaller major commands. Since then, the program has undergone a Critical Change due to schedule slippage of the threshold full deployment date of more than a year. However, the Critical Change report was not completed at the time of our review. Specifically, the DEAMS Inc 1 program was not demonstrating enough progress towards operational effectiveness, suitability, and survivability to reach the full deployment date within one year of the milestone objective. In addition, the program office identified requirements management and technical risks as areas that pose risks to successfully meeting schedule milestones. As of July 2016, the DEAMS Inc 1 program’s total life-cycle estimate was $1,563.7, an increase of $124.9 million (9 percent) from the original life- cycle estimate of $1,434.3 million. In addition, the program’s cost estimate decreased by $49.0 million (-8 percent) when compared to the original total acquisition estimate of $641.1. The DEAMS Inc 1 program was undergoing a critical change due to schedule slippages. The full deployment date has been exceeded by more than a year. However, a Critical Change report had not yet been issued at the time of our review. Did not fully meet all of the system performance targets The DEAMS Inc 1 program had partially met the system performance targets. Specifically, 6 of the 9 performance targets have been met, while 3 have not. The GCSS-A Inc 1 program is a commercial off-the-shelf enterprise resources planning system that is expected to provide users with supply, maintenance, property, integrated materiel management, and management functionality and support. It uses non-developmental items made up of software and server hardware and has two components: GCSS-Army, which provides logistics and financial operations, and the Army Enterprise System Integration Program, which is expected to provide enterprise hub services, master data management, and business intelligence. Program essentials (as of October 2016) DOD component: Army Program owner: Assistant Secretary of the Army, Acquisition, Logistics and Technology Prime contractor: Northrop Grumman Total number of contractors (as of May 2016): Northrop Grumman As of March 2016, the GCSS-A Inc 1 program was in the production and deployment phase of the Defense Acquisition Management System. According to the 2016 Annual Report, the program was on track to remain within schedule, cost, and performance thresholds. However, the GCSS- A Inc 1 program realized a decrease of $16 million in funding that could potentially cause, in part, a six-month slip in fielding with a completion date of March 2018. As of May 2016, the GCSS-A Inc 1 program’s total life-cycle estimate was $3,910.4 million, a decrease of $58.5 million (-1.5 percent) from the original life-cycle estimate of $3,968.9 million. The current total acquisition estimate was $1,932.7 million, an increase of $100.8 million (5.5 percent) from the original total acquisition estimate of $1,831.9 million. The GCSS-A Inc 1 program reached the full deployment decision in December 2012, 10 months after the objective date of February 2012, and 4 months after the threshold date of August 2012. This represents a significant change from the original estimate of February 2012. The program office cited the potential for a 6-month slip in schedule to meet the full deployment date of September 2017. According to a MAIS summary report, the program conducted a trade-off analysis to minimize a 6-month slip in fielding, as a result of a $16 million decrement in 2016 funding. Other challenges in meeting schedule include requirements management, insufficient human resources, and the possibility for Army schedule changes. However, the program reports they are on schedule for full deployment prior to the threshold schedule date of March 2018. Met all of the system performance targets The GCSS-A Inc 1 program had met both of the systems performance targets which were (1) net ready, and (2) mission critical functions. The GCSS-MC LCM Inc 1 program is a portfolio of systems that is expected to support logistics elements of command and control, joint logistics interoperability, and secure access to and visibility of logistics data. As a program within GCSS-MC, Increment 1 is based on the implementation of Oracle e-Business Suite 11i as the core software package. It is expected to provide the foundation for all future Marine Corps logistics systems modernization. Program essentials (as of October 2016) DOD components: Department of the Navy, United States Marine Corps Program owner: Deputy Commandant Installations and Logistics Headquarters Marine Corp Prime contractor: N/A, managed by the federal government Total number of contractors (as of May 2016): Not applicable As of March 2016, the GCSS-MC LCM Inc 1 program had been fully implemented and reported as a closed acquisition program. As of March 2016, the GCSS-MC LCM Inc 1 program’s total life cycle cost estimate was $1,821.70 million, an increase of $1,360.3 million (295 percent) from the original total life cycle cost estimate of $461.4 million. The GCSS-MC LCM Inc 1 program’s current total acquisition cost estimate was $556.7 million, an increase of $362.3 million (186 percent) from the original estimate of $194.4 million. The GCSS-MC LCM Inc 1 program had a significant change in reaching full deployment. The program reached the full deployment date in December 2015. When compared to the original date of May 2010 for full operational capability (equivalent is full deployment), this resulted in a 5 year and 7 month delay. The program is fully implemented. The iEHR Inc 1 program is expected to deliver two user capabilities—a medical single sign on that streamlines the login process to allow users to sign in once and leverage securely stored credentials to automatically access the other available applications, and a context management capability that will automatically present a patient’s data within all applications. Other deliverables include enhanced infrastructure services, a development test center/environment configuration, and clinical data repository upgrades. Program essentials (as of October 2016) DOD component: Defense Health Agency Program owner: PEO DHMS Prime contractor: General Dynamics Information Technology Total number of contractors (as of May 2016): Not applicable As of March 2016, the iEHR Inc 1 program was in the sustainment phase and, according to a 2016 MAIS Annual Report, the program is closed out as an acquisition program. The iEHR Inc 1 program’s March 2015 estimate for total life-cycle cost was $307.6 million, a decrease of $718.3 million (-70 percent) from the original total life-cycle estimate of $1,025.9 million. Likewise, the program’s total estimated acquisition cost was $249.3 million, a decrease of $117 million (-32 percent) from the original total acquisition cost estimate of $366.3 million. The reduction in estimated cost was primarily because of reductions in the program’s scope. The iEHR Inc 1 program’s original objective schedule estimate for full deployment decision was December 2013. When compared to the actual full deployment decision date of November 2014 with the original date, the program’s schedule slipped 11 months. The program achieved full deployment in April 2015, 1 month ahead of schedule. The IPPS-A Inc 1 program is expected to provide a 24-hour, web-based, integrated human resources system to soldiers, human resource professionals, combatant commanders, personnel and pay managers, and other authorized users. Specifically, the system is expected to provide a single, multi-component, trusted database with a single record for all Army soldiers, and serve as a trusted data source for personnel and human resources data for the entire Army. Soldiers will have web- based access to their personnel data, enabling them to better manage their careers and ensure accuracy of information. The system will allow for interface communications and new multi-component reports that include a Soldier Record Brief for all Army components (Active, National Guard and Reserve). Program essentials (as of October 2016) DOD component: Department of the Army Program owner: Department of the Army, Deputy Chief of Staff, G-1 Prime contractors: EDC Consulting, LLC - System Integration, Booz Allen Hamilton-Support Services Total number of contractors (as of May 2016): N/A: Program is in sustainment As of March 2016, the IPPS-A Inc 1 program had achieved full deployment and was in the sustainment phase. The program was re- baselined after implementing strategic recommendations by an independent review team established in June 2013 that determined the system design was sufficient to meet Increment I requirements. The re- baselined acquisition program baseline was signed by the Milestone Decision Authority in February 2015. As of January 2015, the IPPS-A Inc 1 program’s total estimated life-cycle cost was $368.9 million, an increase of $10.5 million (3 percent) from the original total estimated life-cycle cost of $358.4 million. The total estimated acquisition cost was $191.4 million (as of February 2015), an increase of $34.4 million (22 percent) from the original total acquisition cost of $157 million. According to program officials, the increase in cost was a direct result of deferring continuity of operations requirements and capturing the testing environment requirement under Increment 2. The IPPS-A Inc 1 program achieved Milestone C and Full Deployment Decision about 6 months after the original threshold dates in August 2013 and October 2013, respectively. During the acquisition, the delay associated with the slip for a Milestone C decision triggered a significant change to the program’s schedule. The program has achieved full deployment and is now in the sustainment phase. The IPPS-A Inc 2 program is expected to provide a 24-hour, web-based, integrated Human Resources system for Army personnel. It is to deliver an integrated personnel and pay services for all Army components (Active, National Guard, and Reserve), building on a trusted database delivered by the IPPS-A Inc 1 program. The IPPS-A Inc 2 program is planned to improve the Army’s financial management processes in support of military personnel and payroll by linking the personnel and payroll functions for all Army personnel, eliminating duplicate data entry, reducing complex system maintenance, and minimizing pay discrepancies. The IPPS-A Inc 2 program is expected to account for status changes between Active, Reserve, and National Guard components to ensure accurate service time, minimize impact on individual pay, credit for service, and other benefits. Program essentials (as of October 2016) DOD component: Department of the Army Program owner: Department of the Army, Deputy Chief of Staff, G-1 Prime contractor: CACI-System Integration, Booz Allen Hamilton-Support Services Total number of contractors (as of May 2016): 38 As of March 2016, the IPPS-A Inc 2 program is in the development and deployment phase of the defense acquisition life cycle. A Milestone B decision was achieved in December 2014. In fiscal year 2016, the program completed several reviews—system requirements, system functional, and integrated baseline. The program indicated requirements growth as an area of potential risk. As of March 2016, the IPPS-A Inc 2 program’s total life-cycle cost estimate was $2,081.9 million, an increase of $330.9 million (19 percent) from the original estimated life-cycle cost of $1,751.0 million. The current total estimated acquisition cost estimate was $820.2 million, an increase of $94.3 million (13 percent) from the original estimated acquisition cost of $725.8 million. The IPPS-A Inc 2 program achieved Milestone B on 19 December 2014, within the planned threshold schedule. The IPPS-A Inc 2 program was still early in development and system performance data was not available. The ISPAN Inc 4 program is expected to be a system-of-systems that spans multiple security enclaves for strategic and operational level planning and leadership decision making. The system is composed of two primary components: (1) a global adaptive planning collaborative information environment that manages strategy-to-execution planning across all United States Strategic Command (USSTRATCOM) mission areas and (2) a mission planning and analysis system that supports developing nuclear and conventional force application plans. Both components establish a framework to support the USSTRATCOM’s effects-based planning and analysis activities. Program essentials (as of October 2016) DOD component: Air Force Program owner: USSTRATCOM Prime contractors: BAE, Leidos, Lockheed Martin, and Northrop Grumman Total number of contractors (as of May 2016): 19 As of June 2016, the ISPAN Inc 4 program was in the engineering and manufacturing development phase of the Defense Acquisition Management System. According to officials, the program was on track to remain within the schedule, cost, and performance thresholds identified in the original estimate, with no major risks. The program has not been rebaselined. The ISPAN Inc 4 program’s current cost estimates have not changed since their first estimate. The ISPAN Inc 4 program’s current schedule estimates have not changed since their first estimate The ISPAN Inc 4 program was still early in development and system performance data was not yet available. The JMS Inc 2 Program is expected to replace the legacy Space Defense Operations Center hardware and services with extensible and sustainable infrastructure. The effort is planned to integrate components of the Space Situational Awareness mission applications and Command and Control capabilities into the Joint Space Operations Center to create timely, actionable knowledge necessary for maintaining space superiority and exercising command and control capabilities of space forces. Program essentials (as of October 2016) DOD component: Air Force Program owner: Air Force Space Command Prime contractor: N/A, managed by federal government Total number of contractors (as of May 2016): 7 As of March 2016, the JMS Inc 2 program was in the engineering and manufacturing development acquisition phase. Due to issues associated with schedule delays of more than 12 months and acquisition development costs increasing more than 25 percent, the program underwent a critical change. A critical change report was delivered to Congress in September 2016 due to the delayed completion of the program’s Milestone C and full deployment decision by more than 12 months. According to the report, three factors that caused a critical change were (1) the program’s original schedule was aggressive and contained a significant level of concurrency, (2) funding and manpower issues, and (3) contracting issues. As of March 2016, the JMS Inc 2 program’s total life-cycle cost estimate was $965.9 million, a decrease of $134.6 million (-12 percent) from the original total estimate of $1,100.5 million. The latest estimate for total estimated acquisition cost was $462.8 million, an increase of $150.1 million (48 percent) from the original acquisition cost estimate of $312.7 million. The decrease for the total life-cycle cost estimate was because the original estimate for operations and sustainment costs was 20 years after fielding while the new estimate is for 10 years after fielding. Due to significant delays in the JMS Inc 2 program’s aggressive schedule, the program underwent a rebaseline effort and issued a critical change report in September 2016. The new estimate for a full deployment decision is May 2019. With an original schedule date of July 2016, this represents a 2-year, 10-month slippage. The JMS Inc 2 program was still early in development and system performance data was not yet available. The LMP Inc 2 program is expected to deliver an enterprise solution that builds, sustains, and generates warfighting capabilities using a fully- integrated supply chain and maintenance, repair, and overhaul system. The LMP Inc 2 program’s support is critical to the Army’s ability to achieve an integrated system that enables materiel readiness and asset management and accountability, architecture, and acquisition compliancy. When deployed, it is expected to deliver capabilities that address critical Army Materiel Command requirements for automation of the industrial base shop floor operations, standardization of ammunition automatic identification technology, non-Army managed items management, Army prepositioned stock planning, national maintenance program, and the expansion and refinement of the integration between Army enterprise resource planning systems. Program essentials (as of October 2016) DOD component: Army Program owner: PEO EIS Prime contractor: CSRA, LLC Total number of contractors (as of May 2016): 188 As of June 2016, the LMP Inc 2 program was in the operation and support phase of the acquisition life cycle. The program went to full deployment in September 2016. The program will continue to provide national level logistics and financial operations to Army arsenals and depots. As of July 2016, the LMP Inc 2 program’s total estimated life-cycle cost estimate was $736 million, an increase of $6.9 million (1 percent) from the original total life-cycle cost estimate of $729.9 million. Likewise, the total current estimated acquisition cost estimate was $419.2 million, an increase of $14.1 million (3 percent) from the original estimate of $405.1 million. However, these increases were within the threshold estimate established for the program. The LMP Inc 2 program had a schedule slip of 5 months when comparing the original objective full deployment decision date of October 2015 to the current estimate of July 2016. However, the schedule slip was within the established threshold date of April 2016. Met all of the system performance targets The LMP Inc 2 program had met all seven of the system performance targets which were (1) support net-centric military operations, (2) enter and be managed in the network, (3) net readiness exchange information, (4) survivability system information assurance, (5) survivability information assurance system disaster recovery, (6) sustainment operational availability, and (7) training proficiency level. The Teleport Gen 3 program is expected to be a satellite communications gateway that will enable the transportation of voice, video, and data information to deployed forces, and links the deployed warfighter to the sustaining base. The system is expected to provide expanded system capacity, throughput, and functional capabilities to greatly enhance support to tactical and deployed warfighters worldwide. Program essentials (as of October 2016) DOD component: Defense Information Systems Agency Program owner: DISA Component Acquisition Executive Prime contractors: Booz Allen Hamilton, Femme Comp Inc., and Itility Total number of contractors (as of May 2016): 5 As of July 2016, the Teleport Gen 3 program was in the production and deployment acquisition phase. As of July 2016, the Teleport Gen 3 program’s current total life-cycle cost estimate was $575.3 million, an increase of $11.6 million (2 percent) from the original estimate of $563.7 million. The Teleport Gen 3 program had a schedule slip of 3 years and 2 months, from January 2013 to March 2016 when comparing the original baseline Milestone C (Gen 3, Phase 3) objective schedule estimate. Did not fully meet all of the system performance targets The Teleport Gen 3 program’s Phase 1 performance target had been met. However, performance targets for Phases 2 and 3 had not been met. According to the program office, all developmental and operational testing had been completed for Teleport Gen 3 Phases 1 and 2. Additional testing is planned for Phase 3. The TMC program is a suite of systems that is expected to provide mission command computing capabilities to Army commanders and their staffs, consisting of a user-customizable common operating picture enabled with real-time collaboration. These capabilities, together with voice-over-internet-protocol, will provide real-time situational awareness and decision support across the formation (battalion thru Army Service Component Command). A component of the mission command capability is the ability to layer information from multiple functions on the same map display and share among command posts, allowing all parties to have a common view. Program essentials (as of October 2016) DOD component: Army Program owner: Mission Command Prime contractor: General Dynamics (previous); currently a government led effort by Army Research and Development Engineering Center Total number of contractors (as of May 2016): N/A. Army is the lead As of March 2016, the TMC program is currently post Milestone C. The program has been rebaselined two times. A critical change report was issued in response to a cost breach caused by a growth in the program’s scope growth that was considered a “fact of life” change. As of March 2016, program officials reported they are on track to remain within the schedule, cost, and performance thresholds identified in the original estimate. As of March 2016, the TMC program’s total life cycle cost estimate was $2,315.50 million, an increase of $1,421.7 million (159 percent) from the original total life cycle cost estimate of $893.80 million. The TMC program’s latest total acquisition cost estimate was $2,192.1 million, an increase of $1,371.7 million (167 percent) from the original acquisition estimate of $820.40 million in March 2016. We reported in March 2016 that the Army’s TMC program estimated program development costs had increased by 45 percent over the original estimate due to program scope changes derived from the realignment of Command Post of the Future as a foundation for Mission Command Collapse, the integration of Personalized Assistant that Learns, and the incorporation of future force requirements. According to a 2016 MAIS Annual Report, the program was on track to remain within the schedule, cost, and performance thresholds identified in the original estimate. As we reported in our March 2016 report, the TMC program experienced a slight slippage in schedule (3 months), however, it was within the established threshold date. Program officials report they are still on track to reach full deployment by December 2018. Met all of the system performance targets As of March 2016, the TMC program had met all three of the performance targets which were (1) net ready, (2) display the common operational picture, and (3) disseminate (create and exchange) orders. The TMIP-J Inc 2 program is expected to integrate components of the Military Health System sustaining base systems and the Services´ medical information systems to ensure timely interoperable medical support for mobilization, deployment, and sustainment of all theater and deployed forces. The system is expected to enhance clinical care and information capture at all levels of care in theater, transmits critical information to the theater commander; the evacuation chain for combat and non-combat casualties; and forges the theater links of the health record to the sustaining base and the Department of Veterans Affairs. The TMIP-J program is the medical component of the Global Combat Support System. Program essentials (as of October 2016) DOD components: DOD, Joint Participants: U.S. Army; U.S. Navy and Marine Corps; U.S. Air Force Program owner: PEO Defense Healthcare Management Systems Prime contractor: KRATOS Inc. Total number of contractors (as of April 2016): 6 Program status As of July 2016, the TMIP-J Inc 2 program was reported as fully deployed and was in the sustainment phase. The system consists of two increments—Increment 1 was fielded in 2003, and development of Increment 2 was being done in three releases. Release 1 was fielded in 2009 and Release 2 was fielded in 2013. Release 3, is the last major release. In February 2016, the Under Secretary of Defense for Acquisition, Technology, and Logistics declared that the TMIP-J Inc 2 program had met its full deployment decision action items. He delegated Milestone Decision Authority to the Program Executive Officer, Defense Healthcare Management Systems, and authorized the Services to declare full deployment in accordance with their Service acquisition strategies. Exceeded planned cost estimate As of July 2016, the TMIP-J Inc 2 program’s total life cycle cost estimate was $1,056.4 million, an increase of $380.3 million (56 percent) from the original estimate of $676.1 million. The total acquisition cost estimate was $608.2 million, an increase of $374.9 million (161 percent) from the original estimate of $233.3 million. Exceeded planned schedule estimate The TMIP-J Inc 2 program had a 5-year, 6-month change to the full deployment decision date when a breach occurred. Program officials stated the schedule change was due to a mandate to increase the scope of the program. In turn, a revised acquisition program baseline was approved in January 2012. The objective date for full deployment changed from June 2008 to December 2013, and a threshold date changed from December 2008 to June 2014. The TMIP-J Inc 2 program reached full deployment in May 2016 and has transitioned into the sustainment phase. Met all of the system performance targets The TMIP-J Inc 2 program had met all ten of the performance targets which were (1) net ready, (2) data transfer data availability, (3) data availability, currency, and responsiveness—medical logistics visibility, (4) data availability, currency and responsiveness—patient visibility, (5) data availability, currency and responsiveness—medical infrastructure readiness, (6) data availability, currency and responsiveness—individual medical readiness, (7) data availability, currency and responsiveness— illness and injury rates, (8) data availability, currency and responsiveness—reportable medical events, (9) standards compliance, and (10) system operational availability and responsiveness. In addition to the contact name above, the following staff also made key contributions to this report: Eric Winter (Assistant Director), John Ortiz (Analyst in Charge), Neha Bhatt, Nancy Glover, Kate Nielsen, and Teresa Smith.

Summary: DOD's MAIS programs include systems that are intended to help the department sustain its key operations. The National Defense Authorization Act for Fiscal Year 2012 includes a provision for GAO to select, assess, and report on the department's MAIS programs annually through March 2018. This is GAO's fifth report and (1) describes the extent to which selected MAIS programs have changed their planned cost and schedule estimates and met technical performance targets and (2) assesses the extent to which selected MAIS programs have used leading IT acquisition practices, including risk management. GAO selected and reviewed cost, schedule, and performance data for 18 of DOD's MAIS programs that were non-classified and had an acquisition performance baseline. In addition, GAO performed an in-depth review of 5 of the programs, comparing selected IT management practices used by them to leading practices for requirements and risk management and systems testing and integration. The five selected programs were from at least two military services and had not been assessed by GAO in the past year. GAO also interviewed relevant program officials. Most of the 18 selected Department of Defense (DOD) major automated information system (MAIS) programs that GAO reviewed had experienced changes in their planned cost and schedule estimates and half of the programs had met their technical performance targets. Specifically, 16 programs experienced changes in their cost estimates ranging from a 39 percent decrease ($1.47 billion) to a 469 percent increase ($1.63 billion). The average cost increase was $457.2 million among the 11 programs reporting an increase. Fourteen programs experienced schedule delays, which ranged from 2 months to over 13 years. Finally, half of the MAIS programs fully met all of their technical performance targets. Of the remaining nine programs, 4 four had partially met their target because each was still conducting tests. The other five programs were in the early stages of system development and had not begun testing. In addition, for the five MAIS programs GAO selected for in-depth review, all had either fully or partially applied leading practices for managing requirements, risks, and for conducting systems testing and integration. Managing requirements. Three of the five programs had fully implemented the practices for managing requirements, while the other two had partially implemented some practices. Leading practices in this area include establishing requirements and ensuring traceability between requirements and work products. Managing risks. Three of the five programs had fully implemented the risk management practices, while two had partially implemented some practices. An effective risk management process identifies potential problems before they occur. For example, one Army program did not have standard operating procedures for defining thresholds or bounds to manage risk. Unless such procedures are defined, the program will not have the tools needed to define risk management activities, including whether and how certain risks are prioritized. Further, programs should include practices to identify potential problems so that risk-handling activities may be planned and invoked across the project to mitigate the potential for adverse impacts. However, one Air Force program did not develop an overall risk mitigation plan to guide the implementation of individual risk mitigation activities. Without an overall risk plan to guide individual development efforts, those efforts cannot be managed cohesively. Testing and integration. Four of the five programs had fully implemented practices for systems testing and integration. Programs should, among other activities, establish roles and responsibilities to manage testing and integration activities, including a chief developmental tester to oversee testing activities. However, one Air Force program reported difficulty in hiring a qualified individual to perform these duties. Until this position is filled, the program may not effectively manage risks and verify compliance with system acquisition and operational requirements. GAO recommends that DOD improve the management of specific MAIS programs, including establishing procedures for defining risk thresholds, developing an overall risk mitigation plan, and filling a key test management position. DOD concurred with all of the recommendations.