Joint education is provided to some extent at all levels of officer and enlisted professional military education. The professional development and career progression for both officers and enlisted personnel through DOD’s professional military education is a service responsibility; embedded within the professional military education systems is a program of JPME that is overseen by the Joint Staff. For officers, this system is designed to fulfill the educational requirements for joint officer management as mandated by law. The joint education program is intended to prepare aspiring military leaders for both conducting operations in a coherently joint force and thinking their way through uncertainty. Officer JPME program courses are taught at multiple sites across the country, including the service Staff and War Colleges and the National Defense University. See appendix II for a list and map of DOD’s academic institutions where officer JPME courses are taught. According to the Officer Professional Military Education Policy (OPMEP), the JPME program for officers comprises curriculum components in the five levels of the officer professional military education system and includes three-statutorily mandated levels of JPME designed to progressively develop the knowledge, analytical skills, perspectives, and values that are essential for U.S. officers to function effectively in joint, interagency, intergovernmental, and multinational operations. For each of those levels, the emphasis changes to provide instruction in target areas to enhance leader attributes of attending service members at their specific rank and tenure. Within officer PME, the first level of JPME—precommissioning—provides officer candidate and officer training school, Reserve Officer Training Corps, and Military Service Academy students with a basic foundation in defense structure, roles and missions of other military services, the combatant command structure, and the nature of American military power and joint warfare. The second level of JPME—primary—provides junior officers at the O-1 through O-3 ranks with primary education on the tactical level of war. This level of JPME helps to foster an understanding of the Joint Task Force combatant command structure and how national and joint systems support tactical-level operations, among other subjects. The third level begins the first statutorily-mandated level of JPME requirements—JPME Phase I. This phase generally focuses on tactical and operational levels of war, and is typically attended by intermediate- level officers at the O-4 rank, as well as some officers at the O-5 and O-6 ranks. JPME Phase I is incorporated into the curricula of the intermediate- and senior-level service colleges, as well as other appropriate educational programs that meet JPME criteria and are accredited by the Chairman of the Joint Chiefs of Staff. This level of education provides officers the opportunity to gain better understanding from a service component perspective of joint force employment at the operational and tactical levels of war. The subject matter to be covered by JPME Phase I instruction must, by law, include at the least (1) national military strategy; (2) joint planning at all levels of war; (3) joint doctrine; (4) joint command and control; and (5) joint force and joint requirements development. The fourth level of JPME provides Phase II of the statutorily-directed JPME requirements. JPME Phase II is a follow-on for selected graduates of service schools and other appropriate education programs that complements and enhances the JPME Phase I instruction. Phase II is taught at National Defense University’s National War College and Dwight D. Eisenhower School for National Security and Resource Management, the Joint Forces Staff College’s Joint and Combined Warfighting School and Joint Advanced Warfighting School to both intermediate- and senior-level (O-5 and O-6) students, and at senior-level service colleges to senior-level students, and consists of courses on the operational and strategic levels of war. Phase II helps prepare officers for high-level policy and command and staff responsibilities, with a focus on areas such as national security strategy and joint strategic leadership. In addition to the subjects specified in JPME Phase I, the curriculum for JPME Phase II must, by law, include: (1) national security strategy; (2) theater strategy and campaigning; (3) joint planning processes and systems; and (4) joint, interagency, intergovernmental, and multinational capabilities and the integration of those capabilities. This phase completes the educational requirement for joint officer management. The fifth and final level of instruction is the CAPSTONE course of JPME for general/flag officers and focuses on the operational and strategic levels of war for high-level joint, interagency, intergovernmental, and multinational responsibilities. Specifically, the CAPSTONE course focuses on (1) the fundamentals of joint doctrine; (2) integrating elements of national power across military operations to accomplish security and military strategies; and (3) how joint, interagency, intergovernmental, and multinational operations support strategic goals and objectives. This level of JPME is tiered to ensure the progressive and continuous development of executive level officers. Figure 1 summarizes the five levels of JPME. In addition to meeting legislative requirements to provide education on JPME matters, DOD’s colleges and universities that provide academic year-long programs for officers are Master’s Degree-granting institutions. According to the Middle States Commission for Higher Education, the body that accredits National Defense University, among others, accreditation is the means of self-regulation and peer review adopted by the educational community. The accrediting process is intended to strengthen and sustain the quality and integrity of higher education, making it worthy of public confidence and minimizing the scope of external control. Accreditation by the Commission is based on the results of institutional reviews by peers and colleagues and attests to the judgment that the institution has met the following criteria: that it has a mission appropriate to higher education; that it is guided by well-defined and appropriate goals, including goals for student learning; that it has established conditions and procedures under which its mission and goals can be realized; that it assesses both institutional effectiveness and student learning outcomes, and uses the results for improvement; that it is substantially accomplishing its mission and goals; that it is organized, staffed, and supported so that it can be expected to continue to accomplish its mission and goals; and that it meets the Requirements of Affiliation and accreditation standards of the Middle States Commission on Higher Education. DOD also conducts periodic assessments of the three statutorily- mandated levels of officer JPME to ensure that the curricula meet the prescribed joint educational requirements at each level, and uses the results of these assessments to update educational policy as appropriate. For enlisted personnel, the Enlisted Professional Military Education Policy circulates the policies, procedures, objectives, and responsibilities for enlisted professional military education and enlisted JPME. According to that policy, enlisted professional development and progression through the enlisted military education continuum is a service responsibility. The initial focus of enlisted professional military education is military occupational specialty training and education required to produce enlisted personnel capable of performing assigned tasks. Beyond occupational specialty training, the Enlisted Professional Military Education Policy states that all enlisted personnel also should be exposed to enlisted JPME as they progress through their respective services’ enlisted professional military education system. In addition, the policy states that, for some enlisted personnel, more comprehensive joint education is required to prepare those servicemembers for specific joint assignments. Enlisted JPME includes programs that span an enlisted member’s career and apply to all enlisted personnel. Basic Enlisted JPME addresses educational guidelines that should be completed by pay grade E-6, while Career Enlisted JPME addresses educational guidelines for senior enlisted personnel in grades E-6 or E-7 and above. Beyond these programs, Senior Enlisted JPME is a stand-alone Web-based course designed specifically for senior enlisted personnel in pay grades E-6 through E-9 who are serving in or are slated to serve in joint organizations. DOD’s KEYSTONE course exists to prepare command senior enlisted leaders for service in flag level joint headquarters or joint task force organizations. This course is designed for personnel who are serving in pay grade E-9. The focus of this course is to enable command senior enlisted leaders to think intuitively from a joint perspective while serving in their capacity in a general/flag officer joint organization. DOD also developed a program that, since 2003, has provided JPME specifically to reserve officers. In response to legislative direction, the Joint Forces Staff College established a 40-week, blended learning Advanced Joint Professional Military Education program (AJPME) that consists of two distance learning periods and two face-to-face periods. Similar to the active component courses, the Officer Professional Military Education Policy outlines the requirements for AJPME. The program’s mission is to educate reserve component officers to plan and execute joint, interagency, intergovernmental, and multinational operations to instill a primary commitment to teamwork, attitudes, and perspectives. AJPME builds on the foundation established by the institutions teaching JPME Phase I and prepares reserve component officers (O-4 to O-6) for joint duty assignments. DOD’s JPME programs are approved by the Chairman of the Joint Chiefs of Staff. The Director for Joint Force Development retains responsibility to support the Chairman of the Joint Chiefs of Staff and the joint warfighter through joint force development, in order to advance the operational effectiveness of the current and future joint force. The Joint Staff’s Directorate for Joint Force Development (J7) oversees general policy for the JPME programs. The MECC, which was tasked with conducting the most recent DOD-initiated study on JPME, serves as an advisory body to the Director of the Joint Staff on joint education issues. The purpose of the MECC is to address key educational issues of interest to the joint education community, promote cooperation and collaboration among the MECC member institutions, and coordinate joint education initiatives. Its membership consists of the MECC principals and a supporting MECC Working Group. These principals include the Director for Joint Force Development; the Deputy Director of the Joint Staff for Military Education; the presidents, commandants, and directors of the joint professional military education colleges and service universities; and the heads of any other JPME-accredited institutions. Additional representatives from other commands and organizations may be invited to participate as is appropriate. The MECC Working Group is comprised of dean’s level/ O-6 representatives of the MECC principals. The Deputy Director of the Joint Education and Doctrine Division chairs the MECC Working Group. Additionally, the Secretary of Defense, service chiefs, and combatant commanders are invited to send participants to the MECC meetings to provide feedback to improve the educational process. The purpose of DOD’s issued study of the JPME program was to identify desired leader attributes by defining what is needed from the JPME career-long learning experience to support DOD’s strategic vision, as well as any gaps in the current educational program to facilitate the development of the leaders needed to achieve that vision. The original purpose and objectives for the study were laid out in a July 16, 2012, white paper from the Chairman of the Joint Chiefs of Staff and a Joint Staff memorandum issued with the white paper. In the white paper, the Chairman stated that the study’s purpose was to support the development of the department’s leaders by fostering the values, strategic vision, and critical thinking skills needed to lead and support the development of Joint Force 2020. This white paper also identified four attributes that the department’s education programs should develop in its leaders. Those attributes include the following: the ability to understand the security environment and the contributions of all elements of national power; the ability to deal with surprise and uncertainty; the ability to anticipate and recognize change and lead transitions; and the ability to operate on intent through trust, empowerment, and understanding. The Chairman’s white paper also stated that other attributes for leader development would evolve and would need to be aligned with future operations and incorporated into curricula to help ensure that gaps are identified and eliminated. The memorandum, signed by the Director for Joint Force Development, identified three questions meant to shape the review. The first question focused on the value of joint education, why it is needed, and how the military will train and educate its leaders to meet the requirements of Joint Force 2020. The second question focused on the educational outcomes sought by the department, specifically with regard to training leaders for carrying out DOD’s strategic vision, in order to help ensure alignment of desired leader attributes with the curriculum. The third question built upon the first two by focusing on any changes needed to strengthen and achieve the value and outcomes DOD seeks in building leaders for Joint Force 2020. The memorandum also instructed DOD’s MECC to conduct the study. Subsequent to the issuance of the white paper and memorandum, the MECC met on August 23, 2012, to refine the study’s specific objectives. According to officials, at the direction of the Director for Joint Force Development, the MECC removed the requirement to address the value of joint education and instead focused its efforts on identifying the intended outcomes of the educational program and actions needed to achieve the outcomes. The MECC’s discussions also established two broad objectives for the study—documented in meeting minutes and an enclosure—based on the direction in the memorandum and further direction received by the Chairman. The two broad objectives the MECC established were to: 1. define the Joint Education Continuum needed to meet the joint leader development goals for Joint Force 2020, and 2. determine the gaps that need to be addressed for the Joint Education enterprise to progress from the current joint leader development outcomes to producing the future desired joint leader attributes for Joint Force 2020. The evolution of the study’s objectives is described below in figure 2. The MECC’s first objective focused on defining and improving upon both the officer and enlisted joint education continuums to ensure that they are in line with the development goals for DOD’s strategic vision. The MECC identified a need to retain many aspects of the current officer joint education continuum, as well as a need to incorporate new elements into the proposed officer joint education continuum that would help to meet the changing needs of future leaders of Joint Force 2020. For example, the proposed officer continuum retains the comprehensive approach of embedding elements of JPME at each level of service-delivered education with an emphasis on the statutory requirements for JPME Phases I and II, as well as key milestones from the precommissioning level of JPME to the general/flag officer rank. In addition to retaining the existing institutional structure for providing JPME education, however, in its report the MECC also emphasized the importance of self-directed, career-long learning and development. This facet of officer development is intended to convey an expectation that an individual is responsible for his or her own education and development, thereby inculcating a culture of lifelong learning. The MECC’s study concluded, on the other hand, that changes to the enlisted JPME continuum—specifically, between the Senior Enlisted JPME and KEYSTONE courses— are needed to improve the depiction of the Chairman’s intent for enlisted professional military education. Currently, enlisted personnel who complete the Senior Enlisted JPME course at pay grade E-6 or E-7 are not eligible to attend the KEYSTONE course until they attain pay grade E-9, potentially leaving a significant interval in time between participation in the two courses. Accordingly, the MECC recommended dividing the existing Senior Enlisted JPME course into two parts. Both parts would be offered prior to the KEYSTONE course and reduce the interval of time that senior enlisted servicemembers must wait between phases of enlisted joint education. The second objective expanded upon the first, and focused on identifying and closing gaps in the current JPME curriculum to produce joint leaders with attributes that are in line with Joint Force 2020. Specifically, the MECC refined the four attributes proposed in the Chairman’s white paper, added two additional attributes, and identified a set of subattributes associated with each of the six desired leader attributes, that the MECC used to analyze the current capabilities of the department’s joint education programs. The refined list of desired leader attributes consists of the abilities to: 1. Understand the security environment and contributions of all instruments of national power, 2. Anticipate and respond to surprise and uncertainty, 3. Anticipate and recognize change and lead transitions, 4. Operate on intent through trust, empowerment, and understanding, 5. Make ethical decisions based on shared values of the profession of 6. Think critically and strategically in applying joint warfighting principles and concepts to joint operations. The MECC then conducted a gap analysis to identify and crosswalk the desired leader attributes for Joint Force 2020 with the current officer and enlisted personnel JPME education continuums. For officers, the MECC’s analysis showed that, while the JPME program currently addresses the desired leader attribute outcomes at some level and also meets the intent of the existing requirements identified by the Skelton Panel, gaps exist where changes are needed to meet the challenges of Joint Force 2020. These include changes to teaching methodologies, assessment mechanisms, and other areas in support of the newly-identified desired leader attributes. Table 1 summarizes where DOD has identified gaps in its officer JPME curricula associated with the identified list of desired leader attributes and subattributes. For enlisted personnel, the MECC determined that a separate and distinct set of desired leader attributes should be developed for senior enlisted leaders that will require additional study. Officials from the Joint Staff initiated this work on April 8, 2013 and assigned this task to DOD’s Enlisted Military Education Review Council (EMERC). The EMERC took up this effort at its August 22, 2013 meeting and proposed a list of six desired leader attributes for enlisted personnel. DOD officials provided us with the proposed list of enlisted desired leader attributes on September 16, 2013. The MECC made a total of 21 recommendations, which collectively address the study’s objectives and span four categories, including: (1) desired leader attributes, subattributes, or educational outcomes; (2) joint education continuums; (3) lifelong learning and advancements in learning technologies; and (4) faculty quality. See appendix III for a list of the 21 recommendations. Several of these recommendations call for additional study by the National Defense University and the services to support the achievement of leader attributes. Other recommendations emphasize the importance of strengthening the educational outcomes at the primary level of joint education, making education programs more accessible, and using prior learning assessments to tailor education opportunities to students’ needs, among other things. One recommendation, in particular, calls for undertaking an approach to develop a separate and distinct set of desired leader attributes to guide enlisted joint education. Additionally, the MECC considered whether to make any legislative proposals, but did not identify the need for any legislative changes to the joint education enterprise based on the results of the review. According to officials, the results of the study will be used ultimately to inform updates and revisions to the Officer Professional Military Education Policy and the Enlisted Professional Military Education Policy, which are the primary instructions that distribute the policies, procedures, objectives, and responsibilities for professional military education and JPME. Our next section discusses DOD’s plans in this area. The methodology DOD used for its study of JPME generally included leading practices for evaluating strategic training and other programs, such as reviewing existing literature, using available data, and assessing skills gaps. However, the department has not yet fully planned for follow- on actions or fully engaged all stakeholders to help ensure that they are held accountable for making progress in implementing the study’s recommendations. Specifically, the department has not taken steps to ensure that the results of the study will be implemented using an approach agreed upon by all stakeholders, or that the results will be implemented in a timely manner. Further, the department has not yet fully evaluated the potential costs associated with the implementation of the MECC’s recommendations to provide decision makers with more complete information and assurance that the recommendations will be cost-effective. According to our 2012 report, Designing Evaluations, a key first step in designing a program evaluation is to conduct a literature review in order to understand the program’s history, related policies, and knowledge base. A review of the relevant policy literature can help focus evaluation questions on knowledge gaps, identify design and data collection options used in the past, and provide important context for the requester’s questions. Further, according to the RAND Corporation’s Standards for High-Quality Research and Analysis, a study team should demonstrate an understanding of other related studies, which should be evident in, for example, how a problem is formulated and approached. The team should also take particular care to explain the ways in which its study agrees, disagrees, or otherwise differs importantly from previous studies. The MECC’s report states that the initial step in its review was to determine the JPME enterprise’s effectiveness in meeting its current requirements. According to an official, the MECC reviewed many documents as part of a literature review, but derived its findings from four specific sources—all issued within the last 5 years—that it determined were most relevant to the review. Specifically, these sources were: The 2010 House Armed Services Committee (HASC), Subcommittee on Oversight and Investigations (O&I), study which provided a detailed assessment of the state of JPME; A Decade of War, conducted by the Joint and Coalition Operational Analysis Division within the Joint Staff (J7); The Ingenuity Gap: Officer Management for the 21st Century; and Keeping the Edge: Revitalizing America’s Military Officer Corps. According to the MECC’s report, these documents collectively provide a broad look across joint education. The MECC found that the 2010 HASC report, for example, provided a detailed assessment of the state of JPME, and concluded that while the overall PME system was basically sound, some areas needed improvement, such as an increased emphasis on joint, interagency, intergovernmental, and multinational operations. Consistent with this finding, the MECC made a recommendation that the Joint Professional Military Education Division review specific subject areas for increased emphasis within joint education as part of the process of revising the officer and enlisted policies. These subject areas include cyber warfare, interagency and intergovernmental operations, and information and economic instruments of national power, among others. Further, these sources suggested that responsibilities for junior officers will increase and that joint education should be expanded at lower levels. Similarly, the MECC found that officers are receiving joint education exposure earlier in their careers, and also recommended that the services strengthen the instruction of the desired leader attributes at the primary level of joint education for junior officers. Additionally, these sources recognized the need for increased emphasis on handling uncertainty and critical thinking skills. DOD’s study also focused on the ability to anticipate and respond to surprise and uncertainty, and critical thinking in joint operations. RAND’s Standards report further indicates that the data and information used for a study should be the best available and that a research team should indicate limitations in the quality of the available data. As part of its second objective, the MECC analyzed data from joint accreditation visits, that provided broad areas of assessment, and institutional survey and outcome assessments, that provided more detailed observations related to curricula. More specifically, the MECC conducted a review of the Process for the Accreditation of Joint Education (PAJE) reaffirmation results and institutional self-studies for the past 10 years. DOD uses these studies to document compliance with the Officer Professional Military Education Policy (OPMEP) standards and identify areas for improvement. These reviews have not identified systemic problems or required changes of PME over the past 10 years, and have concluded that PME institutions are successful in meeting policy and curricula standards, as well as the intent of the Skelton Report. The MECC also obtained and analyzed existing survey data—from various JPME institutions—of students, graduates, and supervisors of graduates as an indirect assessment of the programs to identify common areas and trends in JPME, as well as outcome assessments such as exams, quizzes, oral presentations, among other things, from the past 2 years as a more direct measurement of student learning. Consistent with the aforementioned RAND Standards report, while the MECC did leverage existing data, the MECC’s report also presented limitations of the existing survey data that it used for the study, stemming from variations in survey questions and measures that the separate JPME institutions used to conduct their own surveys. Regarding gap assessments, our 2004 Guide for Assessing Strategic Training and Development Efforts in the Federal Government highlights the need for determining the skills and competencies necessary to meet current and future challenges, and to assess any skill and competency gaps. According to the MECC’s report, to determine the ability of joint education to meet the Joint Force 2020 joint leader development requirements, the MECC conducted a three-phase gap analysis. These three phases included: (1) refinement of the desired leader attributes, which included breaking these attributes down into subattributes; (2) analysis of data provided by 27 officer and enlisted programs across the JPME system to identify gaps in the JPME program; and (3) gap analysis and documentation to break down the data into cohorts and rank the desired leader attributes. Specifically, 27 officer and enlisted programs, which included the services’ staff and war colleges and National Defense University, among others, were asked to assess the current capabilities of their joint education programs to achieve Joint Force 2020 outcomes by addressing the following three questions: Are current instructional methods effective, that is, how well do we Is the curriculum effective, that is, do we teach it? teach and deliver it? Are we achieving the educational outcomes, that is, can we measure it? According to its report, the MECC relied on the professional opinion of the academic deans of the JPME schools and directors of other joint education programs to make determinations about the effectiveness of their officer and senior enlisted programs based on the 35 subattributes identified by the MECC. To conduct the gap analysis, the MECC analyzed the 945 responses it received from the 27 programs and used a stoplight color coding scheme to summarize by cohort group each program’s assessment of its current capabilities to achieve the Joint Force 2020 outcomes. The gap analysis results were then coded, scored and ranked by officer cohort group (precommissioning, intermediate, senior, CAPSTONE) using four categories to describe the program’s current capabilities. In turn, this gap analysis helped inform the MECC’s 21 recommendations. While the MECC conducted a similar gap analysis for its senior enlisted JPME programs, the results of that analysis led the MECC to conclude that the department needed to further reconsider and redefine the desired leader attributes for enlisted personnel. The MECC has not yet fully developed timeframes and coordinated with all stakeholders, nor has it assessed the costs associated with implementing any of the study’s recommendations. As a result, it is not clear how the department will obtain concurrence among its stakeholders regarding an approach for moving forward, or how the department will hold its stakeholders accountable for implementing change. Furthermore, the department will not know whether the implementation of its recommendations will be cost effective. We have previously found that project planning is the basis for controlling and managing project performance, including managing the relationship between cost and time. Our 2004 Training Guide highlights the need for agencies to develop a formal process to help ensure that strategic and tactical changes are promptly incorporated in training and development efforts. Specifically, that report advises that agencies develop plans that describe or outline the way in which the agency intends to incorporate strategic and tactical changes into its training and developmental efforts. Our guide also highlights that including important agency stakeholders in the process can contribute to an open and continuous exchange of ideas and information, particularly when it comes to ensuring that strategic and tactical changes are promptly incorporated into training and developmental efforts. According to the July 16, 2012 memorandum that directed the MECC study, the March 1, 2013 deadline for the results of the study was intended to help shape the fall 2013 academic year. The MECC completed its report on June 24, 2013, and provided us with a copy of the final report on July 1, 2013. Subsequently, officials provided us with an informational paper that identifies an office of primary responsibility and outlines details on the status of the recommendations. However, this paper does not fully lay out DOD’s planned actions in a transparent manner. For nine of the recommendations, the document identifies potential target dates for completion, one of which is to be completed later in 2013, and eight of which are to be completed not later than September 2014. For the remaining 12 recommendations, DOD identified a target date for completion as either “not applicable” or “to be determined” because most of those recommendations fall under the purview of the services, the services’ schools, or National Defense University. Accordingly, officials states that consideration or implementation of any recommendations that fall under the purview of these stakeholders is at the discretion of those organizations and, according to officials, the department cannot identify or impose timeframes for implementing those recommendations. We recognize that the implementation of several of the recommendations fall under the purview of the services, the services’ schools, or National Defense University. However, the MECC has not yet reached out to these entities to identify estimated target dates for implementation to help guide these efforts. Further, the document does not identify interim milestones for specific actions between the date of the report and the target completion dates. The document also states that completion or implementation of a number of the recommendations is contingent on revisions to the Enlisted Professional Military Education Policy or the Officer Professional Military Education Policy, which officials estimated may take 12-18 months. Officials did tell us that the Joint Staff’s Joint Professional Military Education Division staff and members of the MECC are currently in the beginning stages of incorporating the changes into policy and that the individual schools may begin to incorporate changes resulting from the study at any time. However, at the time of our review, the department’s process for monitoring the implementation of the MECC’s recommendations was unclear. Without established milestones and timeframes for taking follow-on action, and mechanisms for coordinating with all key officials and stakeholders and holding them accountable, DOD may not be reasonably assured that the study’s recommendations will be implemented across the department or that its JPME program will achieve the potential benefits resulting from these recommendations. In addition, the department did not include a requirement that the MECC assess the costs associated with the JPME program or any recommendations resulting from the study. The MECC’s report acknowledges that the resource-limited environment facing the Department of Defense will make it difficult to sustain current practices, such as maintaining small class sizes and small student-to-faculty ratios, at the intermediate and senior courses or to consider additional requirements for the lower-level courses. The report states that leveraging new learning approaches, such as increased distance learning options, will not provide an inexpensive or free solution for increased joint education requirements. However, the MECC did not analyze or consider either the costs associated with implementing the study’s recommendations or efficiencies to be derived from implementing the study’s recommendations. Of the MECC’s 21 recommendations, the MECC identified five recommendations requiring further study and other possible investments, which suggests a potential need for additional funding resources to conduct those studies and implement any findings. For example, the report included recommendations to (1) conduct a study to identify and evaluate potential educational tools to achieve certain desired leader attributes, and (2) consider and explore opportunities to incentivize and reward lifelong learning. The costs associated with these recommendations are unclear and officials responsible for the study could not provide us with cost estimates. Prior studies of professional military education also did not or were unable to fully identify the costs associated with the overall JPME program, or with individual JPME programs. The Skelton Panel, for example, inquired into the cost per student at each school. The Office of the Secretary of Defense provided the panel with raw data produced with different methodologies by service, and sometimes by school, which resulted in widely varying costs for roughly similar programs. More recently, the House Armed Services Subcommittee for Oversight and Investigations made its own effort to ascertain whether a uniform cost accounting system existed for DOD’s professional military education system. The Department provided cost-per-student figures with standardized criteria. Comparative figures were no longer characterized by such enormous variations, but there were still a number of unexplained differences in the data provided by the Office of the Secretary of Defense. Our 2004 Training Guide states that agencies should strategically target training and development investments to help ensure that resources are not wasted on efforts that are irrelevant, duplicative, or ineffective. Our Training Guide further recommends that agencies should consider the appropriate level of investment and prioritize funding so that the most important training needs are addressed first. We have previously concluded that leading practices such as accounting for program resources enables managers to better manage existing resources and plan for future programmatic needs. Further, accounting for program resources is becoming increasingly important for decision makers within the department who must make difficult trade-off decisions in a sequestration budget environment. According to officials, the MECC omitted any and all cost elements from the study so as to not limit their assessment of the department’s needs for educating its leaders for Joint Force 2020 based on resources. Officials told us that they did not want to eliminate options for change based on the cost associated with them. However, by not assessing the cost of the MECC’s recommendations, as well as any efficiencies that may be achieved through their implementation, DOD may not be in a position to know if it has the resources to implement them or whether efficiencies will be achieved, and, without plans to assess cost in the near term as part of continued efforts to implement the results of the JPME study, decision makers may not know if implementing their recommendations will be cost effective. The experiences of the last 12 years of operations in Iraq and Afghanistan have underscored the importance of U.S. military officers’ ability to work jointly across the services and reinforced the need for an effective joint military education program. DOD’s recently completed study of JPME is an important step forward as DOD considers how to adapt its joint military education program to reach the goals it laid forth for Joint Force 2020. Nonetheless, some questions remain about how well its findings can be applied to the task of revising the department’s officer and enlisted JPME policies. Specifically, ensuring transparency and taking certain actions to ensure this study’s findings and recommendations can be utilized effectively is important. For instance, without establishing milestones and timeframes for conducting follow-on actions, and involving necessary stakeholders and developing an implementation approach that provides for accountability, the department may not be assured that the study’s findings and recommendations will be incorporated into the JPME program at institutions across the department and the services in a timely manner. Additionally, without reliable information on the costs of implementing the study’s recommendations, decision makers could be hindered in determining the most efficient allocation of departmental resources for JPME. DOD’s JPME program will continue to play a role in the future development of departmental leaders, and acting to help ensure the findings and recommendations of this and any future studies are both utilized and cost-effective could allow the department to make the best use of the resources it devotes to this program. To guide the implementation of actions DOD identified in its study on JPME, we recommend that the Chairman of the Joint Chiefs of Staff direct the Director for Joint Force Development to take the following two actions: establish well-defined timeframes for conducting follow-on actions, coordinate with all stakeholders, and identify key officials responsible for implementing the study’s recommendations to help ensure the usefulness, timeliness, and implementation of any actions DOD takes in response to the findings and recommendations contained in its study, and assess the costs of implementing recommendations made and efficiencies to be derived from the recommendations in order to implement DOD’s recommendations in a cost-effective manner. In written comments on a draft of this report, DOD concurred with our two recommendations to guide implementation of the actions that DOD identified in its review of joint education. DOD’s comments are reprinted in appendix IV. DOD also provided technical comments on the draft report, which we incorporated as appropriate. Regarding our first recommendation that DOD establish well-defined timeframes for conducting follow-on actions, coordinate with all stakeholders, and identify key officials responsible for implementing the study’s recommendations, DOD stated that, as the Joint Staff pursues implementation of the recommendations in its review, it will establish timelines, identify offices of primary responsibility, and coordinate with stakeholders as appropriate. DOD also noted that efforts are already underway to update the Chairman of the Joint Chief of Staff’s professional military education policies to ensure that these policy directives are consistent with the results and recommendations of the recent study. Regarding our second recommendation that DOD assess the costs of implementing recommendations made and efficiencies to be derived from the recommendations, the department again concurred, stating that, although assessing the costs of its recommendations is beyond the original scope and purpose in this study, it will consider costs and efficiencies prior to moving forward with the implementation of any recommendations. It also stated that a number of recommendations will be implemented through policy changes which are cost neutral. Further, it stated that the services will also consider such costs and efficiencies for recommendations under their purview. We are sending copies of this report to the appropriate congressional committees, the Chairman of the Joint Chiefs of Staff, and the Directorate for Joint Force Development. In addition, this report will also be available at no charge on the GAO Web site at http://gao.gov. If you or your staff have any questions regarding this report, please contact me at (202) 512-3604 or farrellb@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V. To identify the purpose of the Department of Defense’s (DOD) study of the Joint Professional Military Education (JPME) program, we reviewed relevant DOD guidance and other documents related to the study, including the final report dated June 24, 2013, and met with knowledgeable officials. We reviewed the Chairman of the Joint Chiefs of Staff’s July 16, 2012 white paper that called for and outlined the general focus of the study. We also reviewed the Director for Joint Force Development’s (J7) memorandum that was issued with the white paper and clarified the overall intent of the study and specific questions to help frame the study, and directed responsibility of the study to the Military Education Coordination Council (MECC). We analyzed the MECC’s meeting minutes and interim briefing report, which provided insight into how the MECC addressed the questions posed in the Director’s memorandum, as its approach differed somewhat from the framework provided in the memorandum. We then compared the aforementioned guidance documents and interim report findings to the final report to assess consistency. In addition, we met with knowledgeable officials from the Joint Professional Military Education Division within DOD’s Joint Staff’s Directorate for Joint Force Development who, in conjunction with the MECC, were responsible for guiding DOD’s effort. We also met with the officers of primary responsibility and/or team leads responsible for carrying out the various elements of the study. These meetings helped to ensure our understanding of the purpose and objectives of the study. We reviewed the final report section that focused primarily on the study’s findings and recommendations for change to gain an understanding of the MECC’s findings and recommendations. Further, we met with knowledgeable officials within the Directorate for Joint Force Development and the team lead for the findings and recommendations portion of the study to gain clarification and insight into the major takeaways from the study, areas in need of change, and any plans for the way ahead. To assess the methodology DOD used to conduct the joint professional military education study and its planning for follow-on actions, we reviewed the MECC’s final report and supporting documents. We interviewed DOD officials from the Joint Professional Military Education Division within the Directorate for Joint Force Development and officials from across DOD who were identified as officers of primary responsibility or team leads for specific study tasks associated with the study’s two objectives. We reviewed relevant reports and guidance by GAO; the Office of Management and Budget; the Office of Personnel Management; and private sector organizations such as the RAND Corporation, among others, to identify select leading practices for evaluating training, educational, and other programs. The following list includes select sources we consulted for leading practices, in addition to those listed throughout the report, and against which we assessed DOD’s JPME study: GAO, GAO Schedule Assessment Guide: Best Practices for Project Schedules, GAO-12-120G (Washington, D.C.: May 2012), GAO, Designing Evaluations: 2012 Revision, GAO-12-208G (Washington, D.C.: January 2012), RAND Corporation, Standards for High-Quality Research and Analysis (Santa Monica, Calif.: November 2011), The Office of Personnel Management, Training Evaluation Field Guide (Washington, D.C.: January 2011), GAO, Government Auditing Standards: 2007 Revision, GAO-07-162G (Washington, D.C.: January 2007), GAO, Human Capital: A Guide for Assessing Strategic Training and Development Efforts in the Federal Government, GAO-04-546G (Washington, D.C.: March 2004), The Office of Management and Budget, Circular A-94, Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs (Washington, D.C.: Oct. 29, 1992), and GAO, Designing Evaluations, GAO/PEMD-10.1.4 (Washington, D.C.: March 1991). We identified key practices contained in these documents for evaluating educational programs and information that were applicable for the purposes of this engagement. We then compared DOD’s methodology with these select leading practices for program evaluation. We also reviewed documents provided to us and identified by DOD officials as project plans for the study, as well as documentation of the department’s intended follow-on actions. We conducted this performance audit from March through October 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The MECC made a total of 21 recommendations, which collectively address the study’s two objectives and span four categories, including: (1) desired leader attributes, or educational outcomes; (2) joint education continuums; (3) lifelong learning and advancements in learning technologies; and (4) faculty quality. Several of these recommendations call for additional study by the National Defense University and the services to support the achievement of leader attributes. Other recommendations emphasize the importance of strengthening the educational outcomes at the primary level of joint education, making education programs more accessible, and using prior learning assessments to tailor education opportunities to students’ needs, among other things. Table 2 lists the recommendations by category. Brenda S. Farrell (202) 512-3604 or farrellb@gao.gov. In addition to the individual named above, David Moser, Assistant Director; Richard Burkard; Timothy Carr; Gustavo Crosetto; Melissa Emrey-Arras; Jennifer Madison; Terry Richardson; Jennifer Weber; and Erik Wilkins-McKee made key contributions to this report.

Summary: To facilitate unified operations across the services, DOD has provided JPME programs at departmental and service academic institutions for almost 30 years. In July 2012, the Director for Joint Force Development, who reports to the Chairman of the Joint Chiefs of Staff, tasked the MECC to review DOD's joint education objectives and institutions to help ensure that outcomes match requirements for the strategic environment projected for Joint Force 2020. Subsequently, the National Defense Authorization Act for Fiscal Year 2013 mandated GAO to report to Congress on the analytical approach used by the MECC not later than 90 days after the Director submitted the MECC's report to GAO, which the Director did on July 1, 2013. In this report, GAO (1) identifies the purpose of DOD's study of the JPME program, and (2) assesses DOD's methodology used to conduct the Joint Professional Military Education study and its planning for follow-on actions. GAO analyzed the MECC's final report and relevant planning documents, interviewed DOD officials who conducted portions of the study, and reviewed leading practices for evaluating programs by GAO and other entities. The purpose of the Department of Defense's (DOD) study of its Joint Professional Military Education (JPME) program was to identify (1) desired leader attributes as part of the JPME career-long learning experience needed to support DOD's strategic vision and (2) any gaps in the current educational program to facilitate the development of the leaders needed to achieve that vision. Specifically, a Military Education Coordination Council (MECC)--following direction from the Chairman of the Joint Chiefs of Staff and the Director for Joint Force Development--proposed six desired leader attributes, including, for example, the ability to anticipate and respond to surprise and uncertainty, and concluded that the existing institutional structure for providing JPME should be retained. The MECC's gap analysis, however, indicated that in order to support the development of these attributes, a greater emphasis on career-long self-directed learning is also needed, among other things. In line with its findings, the MECC made 21 recommendations to improve the JPME program that address increased accessibility of educational programs, changes to teaching methodologies and assessment mechanisms, and enhanced use of technology. DOD's methodology generally included leading practices for assessing training programs, but DOD has not yet fully planned for follow-on actions and engaged all stakeholders, nor has it assessed the costs of the MECC's recommendations to provide decision makers with more timely and complete information and help ensure that the study's results are implemented. Specifically, the MECC reviewed other related studies, conducted a gap analysis to identify gaps based on existing and future needs, used the best available data and acknowledged limitations--all practices identified by GAO and other government agencies and research institutions as leading practices for successful evaluations of training and other programs. By contrast, DOD documents state that the results of the study were intended to inform and shape the fall 2013 academic year, but the MECC did not complete its study until June 24, 2013 and provided its report to GAO on July 1, 2013. Further, DOD has not yet identified milestones and timeframes for implementing all of its recommendations. Subsequently, the department developed an update to inform actions for moving forward on its recommendations. DOD identified target dates for completion not later than September 2014 for 9 recommendations, but did not include interim milestones, and has not yet developed target dates for 12 recommendations. In addition, the MECC did not formalize plans to achieve the buy-in of all stakeholders for recommendation implementation. Without this information, it may be difficult for DOD to ensure that stakeholders agree on and are accountable for implementing the recommendations in a timely manner. Finally, the MECC did not analyze the costs or efficiencies associated with implementing its recommendations, but it identified 5 recommendations that could incur additional costs because they require further study. Leading practices such as accounting for program resources enable managers to better manage existing resources and plan for future programmatic needs. Without cost data on the study's recommendations or plans to assess cost in the near term as part of continued efforts to implement the results of the JPME study, decision makers could be hindered in determining the most efficient allocation of departmental resources for JPME. GAO recommends that DOD establish well-defined timeframes for conducting any follow-on actions and include stakeholders necessary for implementation, and assess the costs of implementing recommendations made in the MECCs recent study of joint professional military education. DOD concurred with both of GAOs recommendations.