The MHS is a complex organization that provides health services to almost 10 million beneficiaries across a range of care venues, including its own direct care system in hospitals and clinics, through its purchased care network of private providers, and on the battlefield in support of contingency operations. Responsibility for the delivery of care is shared among the Office of the Assistant Secretary of Defense (Health Affairs), the military services, and the Defense Health Agency. The Office of the Assistant Secretary of Defense (Health Affairs) reports to the Under Secretary of Defense for Personnel and Readiness, who in turn reports to the Secretary of Defense. The Army and Navy medical commands and the Air Force medical agency report through their service chiefs to their respective military department Secretaries and then to the Secretary of Defense. The Office of the Assistant Secretary of Defense (Health Affairs) manages the Defense Health Program appropriation, which funds the service medical departments. However, the military treatment facilities, including hospitals and clinics, are under the direction and control of the services, which maintain the responsibility to staff, train, and equip those commands to meet mission requirements. The MHS collaboratively develops strategy to meet policy directives and targets. The service components and the Defense Health Agency are responsible for execution. Figure 1 depicts the MHS organizational structure. The Study was chartered by the Assistant Secretary of Defense (Health Affairs) and led by a team including representatives from that office, as well as from the Defense Health Agency and from the medical commands and agencies of the Army, Navy, and Air Force. Officials were organized into four working groups that collectively made up the Study team. Study officials told us that the majority of the Study’s work was completed in early 2013, but DOD did not finalize the Study until early 2015, and it did not submit the Study to the congressional defense committees until February 2016. The Study included eight recommendations for further study and analysis in a wide range of areas, such as defining medical force readiness. The areas addressed in the Study included the following: Medical Readiness Personnel Requirements: The Study team sought to determine the number of active-duty uniformed medical personnel needed to support military operations. Health Care Demand: The Study team analyzed the demand for health care in geographical medical markets both within DOD’s direct care system of hospitals and clinics and from its purchased care network of private providers. The Study developed goals for the transfer of health care from the private sector to DOD’s own facilities both to increase the efficiency of its spending by making better use of active-duty providers and to provide sufficient workload for them to maintain their skills. Provider Currency Requirements: The Study team sought to develop a metric to assess the currency of medical personnel—that is, an assessment of the maintenance of clinical skills. Currency, in turn, supports the readiness of medical personnel, meaning their ability to provide medical support during operations. However, the Study team was unable to agree upon a clinical currency measure, and instead adopted a productivity metric as a proxy for currency. The productivity metric aims to measure active-duty providers’ workload level measured in Relative Value Units, a measure of health care production. The accompanying goal is for active-duty specialist providers to achieve 40 percent of the median productivity of specialist private-sector providers as defined by a major medical survey. Provider Allocation Model: The Study team developed a model that matched uniformed active-duty medical providers with demand for health care within geographic medical markets, with the assumption that the aforementioned goals for the transfer of health care from the purchased network of private providers to DOD’s facilities and increasing the productivity of active-duty providers will be achieved. This process was intended to demonstrate over- or undersupply of providers in certain specialties. The analysis found that for some specialties, there was insufficient demand or other constraining factors that prevented the matching of all providers to markets. For example, based on the Study’s assumptions regarding the transfer of health care to DOD facilities and increasing active-duty provider productivity, 32 orthopedic surgeons could not be matched to medical markets, based on 2013 data. Primary-Care Analysis: The Study examined the ratio of enrolled beneficiaries to primary-care providers, and found that the MHS was close to its goal of 1,100 enrollees per primary-care provider. Overseas Hospital Analysis: Due to a variety of cultural, logistical, and readiness concerns, the Study recommended no change to overseas military hospitals. Officials told us that this was the primary issue of contention within DOD due to concerns surrounding the potential need for these facilities in support of operations. The Study also included a facility analysis of 24 small hospitals within the MHS to determine their future medical capabilities, such as whether they should continue to have inpatient facilities or birthing centers. The Study recommended changes for 10 of the 24 hospitals, including closing inpatient services at 8 of them. The Under Secretary of Defense for Personnel and Readiness approved proposed changes for 8 of the 10 facilities. The Study estimated annual net cost savings of $366 million in fiscal year 2012 dollars for the recommended changes to the 10 affected hospitals and for achieving its goals for transferring health care to DOD facilities and increasing active-duty provider productivity. This estimated cost savings included about $132 million of savings associated with reductions in civilian Full Time Equivalents (FTE), $333 million of savings associated with decreased purchased care costs, and $98 million in increased nonlabor costs. Prior to the release of DOD’s report to Congress, the Navy completed changes to two of the hospitals and retained, though reduced, inpatient services at the third. The Army also approved selected changes at three of its facilities. In the Report on Military Health System Modernization (the Study), DOD included information that addressed four of the elements listed in section 713 of the NDAA for Fiscal Year 2015, partially addressed four elements, and did not address one element. For example, the Study addressed the element concerning the extent to which DOD consulted with community hospitals to determine their ability to absorb additional patients by discussing with local TRICARE experts. Similarly, DOD addressed the element concerning the recommendations of the Secretary to restructure or realign military medical treatment facilities by including a list of recommended changes to 10 such facilities. However, the Study partially addressed four elements by not addressing every item of the relevant element or including incomplete information. For example, the Study partially addressed the element concerning the impact of recommended changes to military treatment facilities on the access of eligible beneficiaries to high-quality health care, and satisfaction with such care. While the Study listed the criteria by which DOD evaluated the impact of recommended changes on beneficiaries’ access to high-quality health care and satisfaction with such care, it did not provide further details on the results of its evaluation. Similarly, DOD partially addressed the element that listed 14 items concerning hospitals considered under the Study from 2006 to 2012. While the Study provided information addressing 11 of these items, it did not provide full information for the remaining 3, such as the omission of budgeted funding levels for Navy facilities from 2006 through 2009. The Study did not address the element concerning the impact that proposed changes to military medical treatment facilities would have on timely access by local civilian populations to health care services. While the Study included data on civilian hospital capabilities, it did not explain what impact, if any, the proposed changes would have on access to such facilities. Appendix II provides further information on our analysis of the extent to which the Study addressed the nine elements listed in Section 713 of the NDAA for Fiscal Year 2015. Six of the nine elements required a single response, such as an explanation or analysis. The remaining three elements listed multiple items, including two elements that required information concerning all of the hospitals considered under the Study. Although DOD assessed 24 hospitals in its Study, it chose to include information in these two elements that pertained only to the 8 hospitals for which the Secretary of Defense approved the recommended changes. Neither the Study nor the appendix provided a full explanation as to why not every aspect of the required items was included. The Study appendix noted in one instance, for the item concerning mental-health availability at training centers, that there were limited data available. According to senior Study team officials, DOD compiled the data required by Section 713 after the Study was finalized in May 2015. Further, officials stated that, in general, the compilation of these data did not inform the Study’s analysis, findings, or recommendations, because DOD had initiated the study on its own initiative prior to the enactment of the NDAA for Fiscal Year 2015. Although Study team officials stated that they independently analyzed some of the same data elements as those listed in the NDAA, they did not examine all such elements. However, for the data elements they did examine, they may not have examined the exact attributes specified in the NDAA. Our analysis of the Study found that it followed some, but not all, of the relevant generally accepted research standards for research design and execution. Such standards contribute to a sound and complete study. According to these standards, a study’s design should include, for example, establishing the objectives, scope, and methodology, and identifying study assumptions. Successful execution involves ensuring that the methodology is carried out as planned and ensuring that data used in the study are sufficiently valid and reliable for the study’s purposes. (See app. I for further information on the generally accepted research standards that we used to conduct our assessment.) Specifically, we found the following: The Study’s objectives, scope, and methodology were generally consistent with relevant accepted research standards. The Study did not sufficiently mitigate limitations concerning its analysis of the required number of active-duty medical and civilian medical personnel. The Study included a limited assessment of clinical currency—the requirements necessary to maintain the skills of active-duty medical providers. The Study established goals for the transfer of health care from DOD’s purchased care network into its own network of hospitals and clinics and for increasing the productivity of active-duty medical providers without a strategy explaining how these goals would be achieved. The Study’s provider allocation model did not reflect differences among the military services. The Study did not describe steps taken to ensure the reliability of the data supporting its analyses and recommendations. In general, Study officials, including those from the various working groups, stated that they were limited in their analysis by the time available to them. Study officials told us that the majority of the Study’s work was completed in early 2013, but DOD did not finalize the Study until early 2015, and did not submit it to the congressional defense committees until February 2016. The Study’s recommendations for future work and analysis position DOD to continue important efforts to modernize the MHS. However, without consistently following generally accepted research standards, DOD’s work on the Study’s recommendations may not be fully sound and complete. By addressing the areas that we identified in our review, DOD can strengthen its future efforts. The Study’s objectives, scope, and methodology were generally consistent with relevant accepted research standards for the design of a study. Generally accepted research standards state that objectives outline what a study is intended to accomplish; a study’s scope defines the subject matter that will be assessed and reported on, and is directly tied to the objectives; and a study’s methodology should be consistent with and address the objectives. The Study had five objectives that defined the subject matter to be addressed, and all of these related to DOD’s high-level goals for the Study of ensuring the readiness of the medical force and creating a more efficient MHS. The Study’s objectives generally corresponded to the goals in the Study’s Charter document. However, two goals stated in the Study’s Charter were not addressed in these objectives or in the Study’s execution. Specifically, the Study did not address the goals to finalize development of an updated joint casualty model, which would provide a cross-service assessment of the need for medical support in support of operations, and to develop recommended civilian medical force levels. (The efforts by the Study team to address the Study’s objectives and Study’s Charter goals are discussed later in this report.) Although the Study did not address all of its Charter’s goals, it provided new information in some areas, such as currency of active-duty providers, which DOD can use to develop future analysis. The Study’s scope and methodology were linked with the objectives, consistent with generally accepted research standards. The Study’s scope included all medical personnel and facilities that deliver direct patient care in the Defense Health Program–funded treatment facilities. The Study’s methodology involved various analyses by the working groups that were structured around each of the five objectives. The Study’s work was carried out by corresponding working groups, overseen by an overarching group. The Study also provides a brief discussion of “risks and mitigation strategies” comprising a selected list of six changes in the environment or underlying assumptions that could affect the Study’s recommendations and an accompanying potential alternative mitigation strategy to address each change. The Study presented data on the size and composition of the wartime active-duty medical force, also known as the “operational force.” However, the data were largely drawn from service models with known problems and not based on new analysis conducted by the Study team. To estimate the size of the operational force, the Study presented the sum total of the services’ individual estimates of their needs. These estimates are produced by service-specific models and planning tools that examine a common set of defense planning scenarios. However, the Study highlighted differences in approach between the services. For example, the Army chose to use its own estimating tool because the Joint Medical Planning Tool was not accredited at the time of the Study. In contrast, the Navy chose to use an early, unaccredited version of the Joint Medical Planning Tool. Similarly, while the Army used the Joint Integrated Contingency Model to develop its casualty estimates, which are an input into the various planning tools, the Navy used four separate models, including two prototypes. The Air Force does not use casualty estimates, but rather an estimate of the number of likely evacuees from theater. The Study noted that a separate independent analysis conducted by the Joint Staff correlated with the services’ estimates for the operational force. In addition, Study team officials stated that the services’ processes for developing their “generating” and “institutional” forces, which include personnel responsible for issues such as graduate medical education and headquarters staff functions, were not shared with the Study team. Previous reviews of the necessary size of the medical force have highlighted the differences between analyses based on service- generated estimates and analyses based on DOD-wide estimates. Specifically, during the 1990s, two analyses based on DOD-wide estimates found the services had excess medical personnel relative to their needs, while one analysis based on service-generated estimates found the services needed nearly all of their medical personnel to meet their needs. Generally accepted research standards state that limitations and constraints, which are generally found in studies, should be identified and reasonably mitigated. While the Study clearly outlined the limitations of its service-oriented approach, the Study did not fully mitigate them, such as by explaining how these limitations affected the results of the analysis. Officials with the working group that addressed readiness requirements stated that their methodology was limited by the time available for analysis. Specifically, the officials stated that they knew they would be unable, in the time allowed, to conduct a full, independent analysis of the number of active-duty medical personnel necessary to support contingency operations, and that they therefore relied upon estimates from the military services. Officials stated that it would be feasible to conduct an independent analysis of the required number of active-duty medical personnel if there were greater information sharing from the services and more time to complete such an analysis. While we recognize the time constraints faced by the Study team, addressing these issues would provide more meaningful results as DOD considers further changes to the MHS. Absent such efforts to address these limitations, DOD will not have a full assessment of its medical workforce needs. In addition, the Study’s Charter included a goal to determine civilian personnel levels, but the Study did not include such an assessment in its scope. Civilians represent a significant proportion of the MHS workforce. For example, in 2015, civilians represented 61 percent of the Army’s medical positions. Civilian personnel levels also have a direct effect on active-duty personnel levels, and DOD policy requires the services and defense agencies to determine the appropriate mix of manpower (military and civilian) and private-sector support. DOD’s Medical Readiness Review, an outgrowth of the 2006 Quadrennial Defense Review, found that efficiencies could be made by converting military billets to civilian positions, although a prohibition on such conversions was later put in place. Without addressing the role of civilians in the MHS, the Study’s analysis does not present a complete picture of DOD’s needs. The Study included a limited assessment of clinical currency that did not sufficiently identify or mitigate several limitations and constraints. As noted earlier, the Study team was unable to agree upon a measure of the clinical currency of active-duty medical personnel—that is, an assessment of the maintenance of skills. Currency, in turn, supports the readiness of medical personnel, meaning their ability to provide medical support during operations. The Study stated that the Study team attempted to identify an industry standard for clinical currency, but none existed. In addition, a Study official stated that the working group had a limited time frame to complete its work. The Study team instead adopted a productivity metric as a proxy for currency, along with an accompanying goal for active-duty specialist providers to achieve 40 percent of the median productivity of private-sector specialist providers as defined by a major medical survey. However, some Study team officials expressed doubts concerning the accuracy of provider productivity data used to develop the proxy measure for clinical currency. Specifically, officials stated that they believe that the MHS as a whole does not accurately record Relative Value Units, meaning that the MHS does not accurately record providers’ workload. Further, a senior Study team official stated the military services and individual military treatment facilities differ in how they record this information, and, due to these differences, the Study team adjusted the Relative Value Unit figures for a number of hospitals, sometimes with a margin of error of up to 50 percent. Generally accepted research standards state that limitations and constraints should be identified and reasonably mitigated. However, the limitations in the accuracy of information concerning providers’ workload were not identified or mitigated in the Study. Having accurate workload information is important to establishing a sound standard for maintaining the clinical skills of medical providers. Further, in comparing the productivity of active-duty medical providers with civilian medical providers, the Study reduced the amount of time active-duty providers are available to practice medicine because of the “readiness cost” of time that they must devote to military-specific duties and requirements, such as physical training, military training, and federal holidays. The Study team’s approach relied on subject-matter experts to estimate the time active-duty providers had available to practice medicine, and a Study official said there were no reliable data concerning the average number of clinical hours worked by active-duty providers. Other approaches, such as time studies of active-duty providers, are more rigorous, but the Study official stated that the working group’s approach was limited by time constraints. The Study contained a recommendation to further study the costs of medical force readiness, and DOD is currently working to establish a metric for medical force readiness. However, without the benefit of more rigorous analysis concerning active-duty providers’ workload and the time they must devote to military-specific duties and responsibilities, DOD will not have a sound and accurate standard for maintaining the clinical skills of medical providers. The Study, in developing its findings and recommendations, assumed that DOD would be successful in its goals for transferring health care from patients being treated in DOD’s purchased private-sector network into its own military treatment facilities, as well as in achieving related goals for improving provider productivity. The Study recommended the establishment of associated goals, and DOD adopted such goals in March 2014. However, this approach was not consistent with the generally accepted research standard that assumptions should be reasonable. Specifically, the Study’s assumptions regarding its goals for transfer of health care to DOD’s facilities and increasing active-duty provider productivity are unsupported because DOD has not developed an accompanying strategy describing how these goals can be achieved. Our prior work has identified six selected leading practices of effective federal strategic planning, which we derived in part from the Government Performance and Results Act (GPRA), as updated by the GPRA Modernization Act of 2010, associated guidance, and our prior work. Specifically, these leading practices are to: (1) define the mission and goals, (2) define strategies that address management challenges and identify resources needed to achieve goals, (3) ensure leadership involvement and accountability, (4) involve stakeholders, (5) coordinate with other federal agencies, and (6) develop and use performance measures. Study team officials told us that, notwithstanding the absence of a strategy, transfer of health care to DOD facilities is a priority and that such goals were modest and achievable. However, the Study set out goals for transferring health care into its own facilities for large groups of beneficiaries, such as TRICARE for Life beneficiaries, for whom, unlike active-duty military and their families, the MHS has few tools to encourage their use of military treatment facilities. In addition, transfer of health care into DOD facilities has been an MHS goal for several years, but limited progress has been made. For example, the Congressional Budget Office reported in 2014 that most of the growth in outpatient utilization of health care services within the MHS has occurred within the purchased care system, while military hospitals and clinics have maintained a relatively constant workload since 2005. Similarly, it is uncertain that DOD can achieve its related productivity goals without an accurate means for measuring providers’ workload, as discussed earlier. Without a strategy that incorporates leading practices, it remains unclear whether or how DOD can achieve its goals to transfer health care from the purchased care network into its own network of facilities. The Study included a provider allocation model that identified the medical specialties needed with anticipated health care demand within specific geographic medical markets. The model’s results highlighted specialties for which the MHS has too many medical specialists relative to anticipated market demand for various health care services and other specialties for which the MHS has too few physicians to accommodate market demand. Generally accepted research standards state that models should, among other things, represent a reasonably complete range of conditions and be appropriate for the intended purpose. However, an important aspect of the provider model did not reflect the standards in that it unreasonably assumed that medical providers from the different services were interchangeable. Specifically, the Study’s model for allocation of specialist physicians based on anticipated health care demand in geographic markets did not account for the military- service ownership of the relevant hospitals and the physicians’ military service. Instead, it matched the number of physicians across all services within a given specialty, such as orthopedic surgery, to anticipated health care demand in geographic markets, regardless of the service ownership of the medical treatment facility in that market. While such arrangements do exist in “multiservice markets,” such as the National Capital Region and San Antonio, they are less common elsewhere. The model could potentially provide results that do not reflect MHS business practices in the real world. For example, it could suggest that there are sufficient physicians to provide a given type of specialty care to a particular geographic medical market, when in reality there are not sufficient physicians from that medical market’s primary military service to assign. Currently, active-duty providers generally practice within their own services’ hospitals in a team context unique to their military service. The Study emphasized the view that it is important for physicians to practice medicine within this team context of other specialists and nurses from the same service, which can be seen within the services’ different configurations of medical personnel in a deployed setting. CNA independently assessed one part of DOD’s model. However, its assessment included only the mechanics of the model and not an examination of the supporting data used in the model. The Study stated that the Study team relied on military-service representatives to verify these data. In addition, CNA noted that alternative approaches could yield different results depending on assumptions concerning transfer of health care to DOD facilities and increased active-duty provider productivity. The Study asserted that its modeling results were reasonable given the time constraints of the Study. The Study recommended further work to refine and formalize the model and use it to inform decisions concerning the delivery of health care services. However, unless DOD addresses differences between the services and the effects of the model’s practical application, the model’s usefulness will be limited and could lead to impractical decisions. The Study and the data appendix did not describe steps that the Study team took to ensure that the data used to support the Study and data appendix were sufficiently reliable for the Study’s purposes. Generally accepted research standards state that the data used in a study should be valid for the intended research purpose; in other words it represents what it purports to represent. In addition, documenting a study’s analytical and decision-making process helps to ensure the accuracy, completeness, and credibility of study results. However, we found that the study presented limited information on its efforts to assess data reliability. For example, the Study’s data appendix did not document the source of its data in several instances. Specifically, the data source is provided for 9 of 15 data metrics, but not for the other 6 metrics. Because the Study presented limited information on the Study team’s efforts to assess the reliability of the data used, we provided each working group’s lead official with a standard set of questions concerning their efforts to ensure data reliability. Specifically, we asked the working group leads which data sources they used in their respective analyses. Their responses indicated that they used five databases and one data set that were not mentioned in the main body of the Study or the data appendix. We questioned the working group leads concerning steps taken to ensure data reliability, such as whether they consulted with the source or owner of the data to determine what methods were used to ensure reliability or whether they took other steps to ensure reliability, such as conducting their own assessment or reliance on existing assessments. Working group officials stated that they generally relied on the reliability processes of the owners of the data to ensure the data provided were reliable. For example, two of the working groups stated that they used databases that are commonly used within the MHS, and that these databases are subject to oversight reviews and periodic validity checks. An official from one working group stated that the group also conducted reviews of relevant data to check for completeness of records related to inpatient care and found missing data. The working group developed adjustment factors and applied them by market and provider specialty to impute missing data where possible. An official from another working group stated that the group used data provided by contractors, and that DOD’s subject-matter experts routinely discuss report content with the contractors as a way to verify the validity and reliability of the data provided as part of the performance assessment of the contract. In addition, we interviewed service officials regarding the accuracy and reliability of the data used in the Study, and officials from the Navy disputed the accuracy of the Study’s reported physician productivity data. Army officials stated that they had some differences in data interpretation based on their own analysis, which led them to some different recommendations. When asked if there were any limitations on the use of specific data sources, working group officials reported to us some limitations that were not specifically described in the main body of the Study or data appendix. For example, one working group official mentioned that military treatment facility bed counts and ambulatory surgery counts were not consistently reported, though officials stated that these data did not appear to have a significant effect on the group’s findings. Another limitation concerned the consistency of data from the Medical Expense and Performance Reporting System affecting the assessment of a primary-care provider’s availability. An official noted that as a result of this limitation, the final analysis of primary-care capacity was curtailed and specific recommendations were limited. A systematic review of over 300 supporting work papers provided to us by Study officials found multiple examples of known data or analysis limitations that, if the Study was adhering to generally accepted research standards, would have been conveyed in the Study. For example, one of the work papers noted data reliability issues with certain overseas-related data which were not noted in the report. One work paper noted some limitations in a data source used in the study that were not discussed in the report, and another work paper mentioned having to manually adjust some data metrics, which was not noted in the Study. Conveying data or analysis limitations is important so that decision makers relying on the Study can fully understand the quality and soundness of the research. Without sufficiently demonstrating the accuracy, reliability, and limitations of the data upon which DOD makes policy changes, decision makers do not have assurance that the Study’s findings and recommendations are based on sound information. DOD’s estimated cost savings for the recommended changes to 10 small hospitals and the related achievement of goals related to transfer of health care to DOD facilities and increased active-duty provider productivity included a net estimate of savings and costs, but did not fully reflect key practices for cost and cost savings estimates. For example, DOD included estimated savings from three categories—reductions in civilian personnel, net decreases in purchased care, and net increases in nonlabor costs—and listed general sources of potential savings. However, the Study did not fully reflect key practices we derived from cost-estimating, budgeting, and related guidance. These key practices emphasize the importance of an appropriate level of detailed documentation, the inclusion of all significant costs, and an assessment of the reliability of the data used to develop the estimate, among others. First, the Study did not provide an appropriate level of detail concerning the calculation of the estimated cost savings or the assumptions upon which they were based. For example, the Study’s cost savings estimate did not include details concerning the number of military personnel who would be moved from the 10 affected facilities to achieve the savings associated from transfer of health care to DOD facilities and increased active-duty provider productivity. In addition, of the total estimated savings, $333 million was associated with changes that were based on an assumption concerning DOD’s ability to achieve its goals related to transfer of health care to DOD facilities and increased active-duty provider productivity without stating a strategy to reach those targets. Absent a clear and likely achievable strategy, the predicted cost savings associated with transfer of health care to DOD facilities and increasing active-duty provider productivity are questionable. Second, the Study’s predicted net savings estimate did not include all potentially significant costs. Specifically, its cost savings estimate did not address the effect of the following costs: Costs associated with a reduction-in-force of civilian FTEs: The Study estimated savings of $132 million associated with the elimination of 1,324 civilian FTEs across the MHS due to the aforementioned assumption of greater productivity of active-duty providers. However, we have previously reported that DOD’s efforts to reduce civilian FTEs do not always result in planned decreased costs of a proportionate magnitude to the decrease in FTEs. For example, in 2016, we found that across DOD, from fiscal years 2012 through 2016, civilian FTEs declined by 3.3 percent, but civilian personnel costs declined by only 0.9 percent, adjusted for inflation. Onetime costs to realign active-duty providers to other hospitals: The Study recommended changes that would realign active-duty providers to other facilities. However, movement of these providers, the associated active-duty medical professionals with whom they work in a team setting, and their families will incur costs such as permanent change of station expenditures. Study officials told us that they did not include these costs because permanent change of station costs vary widely and final decisions concerning where personnel would be relocated rest with the military services. We previously reported on the costs of permanent change of station moves and found that these moves, on average, cost about $6,700 per servicemember in 2014. Construction costs, if applicable, at facilities losing or gaining active- duty providers: The Study recommended that a number of inpatient facilities be closed or converted to ambulatory surgery centers, but our analysis found that the Study did not identify estimated costs associated with these changes. Construction costs also may be applicable at facilities gaining active-duty providers, but Study officials stated that execution of such changes would be the responsibility of the services. However, without specifically addressing these costs or justifying their exclusion, the Study’s cost savings estimate leaves open the question of whether potential savings could be reduced by such costs. Third, DOD’s savings estimate did not describe the steps taken by the Study team to assess the reliability of cost data used to develop the cost savings estimate. As discussed earlier in this report, the Study did not cite the data sources from which the cost estimates were drawn, and did not discuss what limitations, if any, such data might have had for the purposes to which they were applied. As a result of these weaknesses, DOD may not have accurately estimated the net savings that might result from its proposed hospital changes. As DOD considers additional changes to its network of military treatment facilities, decision makers do not have a full and accurate assessment of all relevant costs and savings. Without a full and accurate assessment, DOD is not in a position to determine whether its efforts to respond to the Study’s recommendations will result in savings or, alternatively, in additional costs. In initiating the Study, DOD began the process of examining difficult and complex problems affecting the MHS. The Study’s recommendations for further analysis and work in a number of areas position DOD, over time, to explore these areas and take actions to improve the effectiveness and efficiency of the MHS. However, our analysis of the Study highlighted shortcomings that merit review and mitigation before DOD takes definitive action on these recommendations. Specifically, without addressing the issues discussed in our report, DOD’s work to address the Study’s recommendations will continue to reflect the various unsupported assumptions, unmitigated limitations and constraints, and questions concerning the reliability of data used to develop the Study’s recommendations that we have highlighted. Similarly, as DOD considers further changes to its network of facilities and the medical services they offer, the analysis supporting these decisions will be incomplete without a full and accurate picture of the true cost of such changes. To fully assess the size and composition of the medical force, we recommend that the Secretary of Defense direct the Assistant Secretary of Defense (Health Affairs) to conduct a new analysis of the required number of active-duty and civilian medical personnel that mitigates known limitations. To strengthen ongoing efforts to analyze the costs of medical force readiness and establish clinical currency standards, we recommend that the Secretary of Defense direct the Assistant Secretary of Defense (Health Affairs) to take steps to identify and mitigate limitations regarding the standard for maintaining providers’ clinical skills, including improving the accuracy of information concerning providers’ workload and conducting an analytically rigorous calculation of active-duty providers’ time devoted to military-specific responsibilities. To help achieve DOD’s goals for transferring health care into its own facilities and increasing the productivity of active-duty medical providers, we recommend that the Secretary of Defense direct the Assistant Secretary of Defense (Health Affairs) to develop a strategy for achieving these goals that reflects the leading practices of effective federal strategic planning. To strengthen ongoing efforts within DOD to address the Study’s recommendations to use the provider model outputs to inform execution of health care delivery and to refine the model for future use, we recommend that the Secretary of Defense direct the Assistant Secretary of Defense (Health Affairs) to modify DOD’s model to reflect the military service of the physicians and military treatment facilities included in the model. To strengthen any future assessments of additional changes to DOD’s network of military treatment facilities, we recommend that the Secretary of Defense direct the Assistant Secretary of Defense (Health Affairs) to take the following two actions: describe steps taken to assess the reliability of data supporting the assessment, including, at a minimum, the sources of data, data limitations, and efforts to test data reliability; and include in any accompanying cost estimates an appropriate level of detail, all significant costs, and an assessment of the reliability of the data supporting the cost estimate. In written comments on a draft of this report, DOD concurred with our six recommendations to improve DOD’s analysis of the size, readiness, and efficiency of the medical force. DOD’s comments are reprinted in appendix III. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Deputy Secretary of Defense, the Under Secretary of Defense for Personnel and Readiness, the Assistant Secretary of Defense (Health Affairs), the Defense Health Agency Director, the Surgeon General of the Air Force, the Surgeon General of the Army, and the Surgeon General of the Navy. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions regarding this report, please contact me at (202) 512-3604 or farrellb@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. To guide our review of the Department of Defense’s (DOD) Report on Military Health System Modernization (the Study), we identified relevant generally accepted research standards for the design, execution, and presentation of findings that define a sound and complete study. We developed these standards by reviewing research literature and DOD guidance and identifying frequently occurring, generally accepted research standards that are relevant for defense studies. We initially used these standards in a prior GAO review of a DOD study on mobility capabilities. That report includes a list of 36 questions to ask in assessing the application of generally accepted research standards. To develop these standards, we reviewed research literature and DOD guidance and identified frequently occurring, generally accepted research standards that are relevant for defense studies that define a high-quality or methodologically sound and complete study. The following were our sources for these standards: GAO, Government Auditing Standards: 2011 Revision, GAO-12-331G (Washington, D.C.: December 2011); GAO, Designing Evaluations, GAO-12-208G (Washington, D.C.: January 2012); GAO, Dimensions of Quality, GAO/QTM-94-1 (Washington, D.C.: February 2004); RAND Corporation, RAND Standards for High-Quality Research and Analysis, CP-413 (5/15) (Santa Monica, CA.: May 2015); Air Force Office of Aerospace Studies, Analysts Handbook: On Understanding the Nature of Analysis (January 2000); Air Force, Office of Aerospace Studies, Air Force Analysis Handbook, A Guide for Performing Analysis Studies: For Analysis of Alternatives or Functional Solution Analysis (July 2004); Department of Defense, DOD Modeling and Simulation (M&S) Verification, Validation, Accreditation (VV&A), Instruction 5000.61 (Washington, D.C.: Dec. 9, 2009); Department of Defense, Support for Strategic Analysis, Directive 8260.05 (Washington, D.C.: July 7, 2011); Department of Defense, Implementation of Data Collection, Development, and Management for Strategic Analyses, Instruction 8260.2 (Washington, D.C.: Jan. 21, 2003). We determined that these standards are still current and relevant for the purposes of this report. We confined our assessment to the standards related to design (11 of 14 standards) and execution (10 of 15 standards) because the National Defense Authorization Act for Fiscal Year 2015 mandated we assess the methodology and data used in the Study. Consequently, we did not assess DOD’s report against the generally accepted research standards for presentation of results (7 standards). Of the applicable standards related to design and execution, we combined and slightly revised the standards to match the particular analyses used in the study. Some of the design and execution standards were not relevant to our assessment. For example, the standards concerning analysis of scenarios and threats are specific to certain types of defense studies and did not relate to the study that was the subject of our review. We compared DOD’s approach to the Study with these selected generally accepted research standards. Accordingly, we technically reviewed the content of DOD’s report against the following 18 standards that were relevant to our review: a. Objectives, Scope and Methodology: Objectives outline what a study is intended to accomplish. A study’s scope defines the subject matter that will be assessed and reported on, and the scope is directly tied to the audit objectives. A study’s methodology should be consistent with and address a study’s objectives. Is the study’s design clear? Is the study’s objective clearly stated? Is the study’s scope clearly defined? Do the study scope, methodology, and objectives fully address the study charter and associated guidance? Was the study plan followed? Were deviations from the study plan explained and documented? b. Assumptions: Assumptions should be reasonable, consistent, appropriately varied, and explicitly identified in order to understand the context of the study’s results. Assumptions define the parameters for what will be included and how the study is executed. Are the assumptions explicitly identified? Are the assumptions reasonable and consistent? Are the assumptions varied to allow for sensitivity analyses? c. Limitations and Constraints: Limitations and constraints, which are generally found in studies, should be identified and reasonably mitigated. Research standards do not call for the elimination of all limitations, but do call for them to be identified, assessed, mitigated, and explained. Are the limitations explicitly identified? Are the major constraints identified and discussed? a. Models: Models should be reasonable ones to consider, represent a reasonably complete range of conditions, as well as be appropriate for the intended purpose. Are the scenarios that were modeled reasonable ones to consider? Do the scenarios represent a reasonably complete range of conditions? Were the models used to support the analyses appropriate for their intended purpose? Have they been described and documented adequately? Were any modeling limitations identified, explained, and justified? b. Data reliability: The data used in a study should be valid for the intended research purpose; in other words, they represent what they purport to represent. Documenting a study’s analytical and decision-making process helps to ensure the accuracy, completeness, and credibility of study results. Were the data used valid for the study’s purposes? Were the data used sufficiently reliable for the study’s purposes? Were any data limitations identified and was the effect of the limitations adequately explained? Table 1 summarizes our assessment of the extent to which the Department of Defense (DOD) in its Report on Military Health System Modernization (the Study) addressed each of the required nine elements listed in Section 713 of the National Defense Authorization Act (NDAA) for Fiscal Year 2015. Some of the elements required that multiple items be addressed. We considered an element “addressed” if it included all of the items listed in the NDAA; “partially addressed” if it included some, but not all, of the items; and “not addressed” if it did not include any of the items. In addition to the contact named above, Lori Atkinson, Assistant Director; Rebekah Boone; Patricia Donahue; Mae Jones; Linda Keefer; Carol Petersen; Adam Smith; Sabrina Streagle; and Adriel Taslim made key contributions to this report. Defense Health Care Reform: Actions Needed to Help Ensure Defense Health Agency Maintains Implementation Progress. GAO-15-759. Washington, D.C.: September 10, 2015. Defense Health Care Reform: Actions Needed to Help Realize Potential Cost Savings from Medical Education and Training. GAO-14-630. Washington, D.C.: July 31, 2014. Military Health System: Sustained Senior Leadership Needed to Fully Develop Plans for Achieving Cost Savings. GAO-14-396T. Washington, D.C.: February 26, 2014. Defense Health Care Reform: Additional Implementation Details Would Increase Transparency of DOD’s Plans and Enhance Accountability. GAO-14-49. Washington, D.C.: November 6, 2013. Defense Health Care: Additional Analysis of Costs and Benefits of Potential Governance Structures Is Needed. GAO-12-911. Washington, D.C.: September 26, 2012. Defense Health Care: Applying Key Management Practices Should Help Achieve Efficiencies within the Military Health System. GAO-12-224. Washington, D.C.: April 12, 2012. Military Personnel: Enhanced Collaboration and Process Improvements Needed for Determining Military Treatment Facility Medical Personnel Requirements. GAO-10-696. Washington, D.C.: July 29, 2010.

Summary: DOD initiated the Study to address perceived weaknesses within the Military Health System and to leverage advances in civilian business practices. The National Defense Authorization Act for Fiscal Year 2015 included a provision for DOD to submit the Study to the congressional defense committees and for GAO to review the Study. DOD submitted its study in February 2016. This report assesses, among other things, the extent to which the Study followed an approach that is consistent with relevant generally accepted research standards and utilized key practices for estimating cost savings. GAO compared the Study with generally accepted research standards that were developed by reviewing research literature and DOD guidance and with key practices derived from cost-estimating guidance. The Department of Defense's (DOD) approach in its Report on Military Health System Modernization (the Study) did not consistently follow relevant generally accepted research standards for research design and execution. While the Study's recommendations position DOD, over time, to take actions to improve the effectiveness and efficiency of the Military Health System, GAO found a number of shortcomings, including the following: The Study did not fully mitigate limitations identified in its analysis of the required number of active-duty and civilian medical personnel. For example, the Study did not explain how known issues with the military services' workforce models affected the results of its requirements analysis. Without addressing such limitations, DOD will not have a full assessment of its medical workforce needs. The Study did not sufficiently identify or mitigate limitations concerning its assessment of the requirements necessary to maintain the skills of active-duty medical providers. For example, although there were limitations concerning the accuracy of information on medical providers' workload, the Study did not identify or mitigate these limitations. Having accurate workload information is important to establishing a sound standard for maintaining the clinical skills of medical providers. The Study established goals for transferring health care from DOD's purchased care network into its own network of hospitals and clinics and for increasing the productivity of active-duty medical providers, but did not develop a strategy explaining how these goals would be achieved. Without such a strategy it remains unclear whether DOD can achieve its goals to transfer health care from the purchased care network into its own network. DOD's estimated cost savings did not fully utilize key practices for developing such estimates. DOD estimated net annual savings of $366 million from changes to 10 small hospitals and achievement of its goals for recapturing health care and increasing the productivity of active-duty health care providers. However, DOD did not include in its estimate an appropriate level of detail concerning the calculation of estimated savings, all potentially significant costs, or a description of the steps taken by the Study team to assess the reliability of cost data used to develop the estimate. For example, the Study recommended that a number of inpatient facilities be closed, but GAO's analysis found that the Study did not identify estimated costs associated with these changes. As a result, DOD's cost savings estimate did not present a full and accurate picture of possible costs and savings. GAO is making six recommendations, including that DOD conduct a new analysis of the required number of active-duty and civilian medical personnel that mitigates known limitations; identify and mitigate limitations regarding the standard for maintaining providers' clinical skills; develop a strategy for achieving its goals for transferring health care to DOD facilities and increasing the productivity of active-duty providers; and, when considering proposed changes to facilities, include in any accompanying cost estimates an appropriate level of detail. DOD concurred with each of GAO's recommendations.