Each military service has separate PME schools for senior- and intermediate-level officers. As defined by the Joint Staff’s Officer Professional Military Education Policy, the senior-level schools, typically for O-5 and O-6 ranked officers, focus on warfighting within the context of strategy. The intermediate-level schools, typically for O-4 ranked officers, focus on warfighting within the context of operations. (See table 1 for a list of PME schools and enrollment totals.) The senior- and intermediate-level PME schools are not alike in terms of program offerings for resident and nonresident students. As indicated in table 1, while all senior-level PME schools offer resident programs, only the Army War College and the Air War College have analogous nonresident programs. Also as indicated in table 1, all intermediate-level PME schools offer resident and nonresident programs. DOD has approximately 39,318 students enrolled in its senior- and intermediate-level PME schools. The vast majority of these enrollees are nonresident students. Of the total enrolled, approximately 3,788, or 10 percent, are taking course work as resident students; the rest, or 90 percent, are nonresident enrollees. Nonresident PME exists to provide PME to a larger population than can be supported in resident institutions. Nonresident PME provides alternative learning-style options for officers not selected for residence or unable to participate in residence due to operational commitments. The military services have had nonresident PME programs for many years. The Naval War College (NWC) has had a department for correspondence courses since 1914. The U.S. Army War College (USAWC) has provided a nonresident course offering since 1968. The Air Force’s nonresident programs were created in 1947 for its senior-level PME school and 1948 for its intermediate-level PME school. Paper-based correspondence is the traditional nonresident PME delivery mode. Students complete correspondence courses individually with limited faculty contact. Course materials and submissions are exchanged between students and faculty primarily by mail. PME schools have implemented other delivery modes, including seminars conducted at remote sites by PME faculty and CD-ROM distribution. Increasingly, PME schools are using ADL techniques in their nonresident program offerings. Several ADL applications are currently in use at senior- and intermediate- level PME schools, and all of them are focused on nonresident programs. They are offered at the U.S. Army War College, the Naval War College, and the Air Command and Staff College (ACSC). A planned ADL offering for reserve component staff is under development at the Joint Forces Staff College. See appendix IV for details on these programs. The addition of an ADL application for the Army Command and General Staff College nonresident PME course is anticipated for fiscal year 2005. DOD does not have specific performance goals and metrics to assess the effectiveness of ADL in PME. While DOD believes ADL has had a positive impact, its views are based on anecdotal information, rather than a systematic performance measurement. Thus, DOD cannot determine whether ADL is meeting performance goals in comparison to other delivery methods. Although numerous organizations are providing oversight of PME activities, with several specifically responsible for ensuring that PME meets general standards of accreditation, these organizations do not focus on student learning outcomes—that is, the knowledge, skills, and abilities a student acquires from a course. Instead, DOD accreditation activities, like those in the private sector, focus primarily on educational process inputs, such as quality of facilities and student faculty ratios. We and a private- sector organization have recently established guidelines and frameworks for assessing the effectiveness of educational programs that stress a focus on measurable outcomes. ADL is a new and evolving tool for which systematic evaluation requirements have not been established. ADL has a unique ability to capture, retain, store, and document interactions in an online environment, which provides the opportunity to demonstrate student skill improvements, and thus to customize performance metrics. However, we have found no evidence to indicate that DOD is utilizing this ability. Numerous oversight organizations review PME activities, with several organizations specifically designed to ensure that PME conforms to general standards of accreditation. The preeminent mechanism for oversight is the Joint Chiefs of Staff’s Process for Accreditation of Joint Education. The process is designed to provide oversight and assessment of PME institutions for purposes of strengthening and sustaining Joint Professional Military Education. It is a peer-review process involving a self-study component and a team assessment. The review sequence includes certification, accreditation, and reaffirmation of accreditation status. Accreditation can currently be granted for up to 5 years, and all PME programs with current ADL applications are Joint Staff-accredited. The Joint Staff also sponsors the Military Education Coordinating Council, an advisory body composed of high-ranking PME leadership. The purpose of the Council is to address key issues of interest for joint military education, to promote cooperation and collaboration among member institutions, and to coordinate joint education initiatives. The military services have responsibility for the service PME institutions in terms of managing PME content and quality and conducting all levels within the guidelines of the military educational framework. Consistent with Title 10 of the United States Code, the Secretary of Defense requires that each PME institution periodically review and revise curriculum to strengthen focus on joint matters and on preparing officers for joint duty assignments. PME is also reviewed by other internal and external organizations. Each PME institution has a Board of Visitors/Advisors that provides guidance over PME activities. The Board of Visitors/Advisors is composed of military and/or civilian academic officials who are nominated by PME schools and appointed by service secretaries to provide advice on educational and institutional issues. Service PME institutions have other internal and external advisory committees that perform activities such as providing advice, communicating feedback from major commands, and conducting curriculum review. Service Inspector General offices have conducted periodic reports and assessments on PME schools. The military services’ education and training commands also provide oversight of PME activities, though not day-to-day administration. Additionally, private-sector regional accreditation agencies assess senior- and intermediate-level PME programs. Their accrediting activities generally guide the Joint Staff’s review process. PME schools have not established, and oversight organizations have not reviewed, specific goals or metrics of performance effectiveness for ADL implementation. As was stated in our recently issued guide for establishing a framework for assessing training and development efforts in the federal government, “it is increasingly important for agencies to be able to evaluate training and development programs and demonstrate how these efforts help develop employees and improve the agencies’ performance.” The Sloan Consortium—a private-sector organization that maintains a repository of information on distance education—views metrics as crucial for assessing program effectiveness. For example, metrics can (1) demonstrate that the “learning effectiveness” of nonresident education is at least as good as that of its resident counterpart, (2) identify cost comparisons that can be used to develop better strategic plans, and (3) provide information on student retention and completion rates. As was stated in our report on oversight for the military academies, such elements embody the principles of effective management, in which achievements are tracked in comparison with plans, goals, and objectives, and the differences between actual performance and planned results are analyzed. PME schools identified advantages of ADL over other means of delivery, but the advantages appeared to be anecdotally derived. PME school officials stated that ADL has resulted in quality improvements in PME delivery, especially when compared with paper-based correspondence. These advantages include (1) better facilitation of student and faculty interaction; (2) increased flexibility in modifying course material; (3) reductions in time required to complete programs; (4) better leveraging of resources for administrative support; and (5) establishment of learning management systems that monitor student progress and produce management reports. But there were no indications that evidence for these advantages were based on an evaluative effort to compare differences between ADL and paper-based correspondence courses. Since PME schools have not detailed a comprehensive process for evaluating ADL benefits over paper-based correspondence, it cannot be determined whether ADL is meeting performance goals based on appropriate returns on investment, student retention, student access to courses, or other goals that schools use to measure program effectiveness. Additionally, we did not observe any oversight agency focus on specific metrics of ADL effectiveness. According to Joint Staff officials, they perform reviews of nonresident programs as part of their accreditation activities. However, their reports focus on the nonresident program as a whole and not on particular methods of delivery. ADL is a new and evolving tool, and a systematic assessment of these applications has not yet been required. The three regional accreditation agencies that review PME schools with ADL implementations show variances in nonresident program evaluation policy. One agency stated that nonresident programs are not separately evaluated, although the programs may be included within the scope of the institution’s existing accreditation. Another agency stated that additional procedures must be performed before nonresident programs are included within the scope of the institution’s accreditation. The third agency required schools to evaluate its nonresident programs to ensure comparability to resident programs. In addition, we have not observed any Office of the Secretary of Defense or Board of Visitors/Advisors reviews in relation to ADL effectiveness for nonresident PME. While we did not observe measures of effectiveness specifically geared toward ADL applications, PME schools with ADL applications did perform program effectiveness assessments for nonresident education by the way of student satisfaction assessments as part of the Joint Staff accreditation process. These assessments used in-course student surveys, graduate surveys, and supervisory surveys to obtain feedback as part of a systematic approach to instructional design and to update and improve curriculum offerings. USAWC performs surveys of students, alumni, and general officers with USAWC graduates in their commands. Students are surveyed for each course regarding particular aspects of the course and general degrees of satisfaction. A survey of alumni is conducted every 2 years. A general officer survey, designed to assess general officer impressions of alumni, will now be conducted annually instead of every 3 years, as in the past. Prior feedback from general officer surveys reported that the curriculum should emphasize application of strategic thinking to national security issues. USAWC also performs internal course evaluations as part of its curriculum assessment process. USAWC faculty members are required to undergo training to provide a degree of standardization in instruction and evaluation. This standardization, especially for evaluation, is more stringent for nonresident education. USAWC can conduct trend analyses for student performance and student satisfaction to determine statistical significances. NWC uses student and alumni surveys to assess the academic program’s effectiveness. Depending on the department, student assessments include daily sessions critiques, lecture critiques, end-of-course critiques, major exercise critiques, and exam critiques. Alumni are sent questionnaires 2 years after graduation asking for feedback on their educational experience. All academic departments conduct an extensive analysis of various student surveys to determine areas of the curriculum that are not meeting student needs so that these areas can be improved. Surveys are based on standards promulgated by accrediting agencies and external organizations to help objectively measure institutional excellence. Resident and nonresident student programs are measured the same since a single faculty is responsible for both. Peer evaluation of faculty members is used to sustain teaching method quality. ACSC uses internal and external evaluations at all phases of its curriculum development process. It conducts end-of-course surveys that focus on delivery and educational support and end-of-year surveys for students to provide feedback about whether they believed the school (1) prepared them to lead commands, (2) accomplished its mission, (3) was institutionally effective, and (4) was beneficial to professional development. Surveys are also given to graduates and graduate supervisors to obtain perspectives on whether the school (1) accomplished its mission and institutional effectiveness; (2) enhanced graduates’ ability to think operationally and critically; (3) prepared graduates to assume leadership duties, and (4) made the experience valuable in professional development. Student learning outcomes, as stated by the Council for Higher Education Accreditation—a national association representing accrediting organizations—are “properly defined in terms of the knowledge, skills, and abilities, that a student has attained at the end (or as a result) of his or her engagement in a particular set of higher education experiences.” PME schools generally are not assessed for student learning outcomes as a means of determining program effectiveness. The Joint Staff’s accreditation organization responsible for assessing PME schools has primarily focused on inputs to the educational process. As detailed in its policy, its educational standard assessment and self-study requirements focus on internal aspects such as organizational structure, facilities, curricula, student to faculty ratios, student body composition/mix, and faculty qualifications. However, as stated in our recently published guide for assessing training and development programs, the focus on evaluating activities and processes takes away from evaluating training and development’s contribution to improved performance, reduced costs, or greater capacity to meet new and emerging transformation challenges. The Joint Staff has identified the usefulness of student learning outcomes and is currently in the process of developing student learning outcomes for PME and procedures to include them in the accreditation process. Our recently published report on distance education states that there is increased interest in using outcomes more extensively as a means of ensuring quality in all forms of education, including nonresident education. The Council for Higher Education Accreditation has issued guidelines on nonresident education and campus-based programs that call for greater attention to student learning outcomes, and the congressionally appointed Web-based Education Commission has also called for greater attention to student outcomes. The Commission said that a primary concern related to program accreditation is that “quality assurance has too often measured educational inputs (e.g., number of books in the library, etc.) rather than student outcomes.” Private-sector educational institutions are just beginning to emphasize the evaluation of learning outcomes as a viable measure of program effectiveness. For example, the University of Maryland University College, a school with a comparably large distance education program and which serves a large number of military personnel, is piloting a project to identify and measure learning outcomes in five general areas—writing efficiency and oral communication, technology fluency, information literacy, quantitative literacy, and scientific literacy. The university will use knowledge captured by its distance education database to serve as a basis for this determination. Accrediting agencies and our recent report on training and development program assessments are also emphasizing the evaluation of learning outcomes as a measure of program effectiveness. Some of the regional agencies that accredit programs at the senior- and intermediate-level PME schools generally recognize the importance of student learning outcomes and have instituted practices that reflect some aspects of a systematic, outcome-based approach called for in GAO-04-279. However, these agencies vary in the extent to which standards and policies address student learning outcomes for distance education. Our training and development assessment guide states that agencies need credible information on how training and development programs affect organizational performance, and that decision makers will likely want to compare the performance of these programs with that of other programs. Furthermore, programs lacking outcome metrics will be unable to demonstrate how they contribute to results. We surveyed nonresident PME current students and graduates to obtain their perspectives on the achievement of PME learning objectives and PME’s impact on their career objectives. (See appendix III for presentation of survey results.) Because we only surveyed nonresident students, we could not compare the results with those of resident students. However, we believe the data can be useful for DOD to consider in its continuing study of program effectiveness. ADL has a unique ability to capture, retain, store, and document interactions in an online environment, which provides the opportunity to demonstrate student skill improvements, and thus to customize performance metrics. Since work is done on a computer, various data points are automatically collected as a student works, including the time spent, specific pages of the text visited, use of online help, and communication with others. University of Maryland University College officials pointed out ADL’s unique ability when compared with other delivery methods to retain, capture, store, and document baseline data that can be used as the basis for performance metrics. These officials said they would use such data in designing performance measures for learning outcomes. However, we found no evidence to indicate that DOD is using this ability. DOD may be missing an opportunity to enhance its ability to measure effectiveness. The processes for converting PME courses to ADL varied by school and by military service, and they feature a mixture of in-house and contractor approaches. PME schools generally focus their ADL applications on nonresident programs to improve their efficiency and effectiveness. In most cases, conversion decisions were made in collaboration with school academic boards. PME schools did not identify any systematic criteria as the basis for their decisions as to which courses or curricula to convert to ADL. They subjectively focused ADL conversion on the suitability of content for Web-based applications. Curriculum conversions were made because of a DOD-wide need (1) to improve access to a diverse officer corps and (2) to increase the efficiency of educational delivery. Since nonmilitary educational institutions also lack systematic criteria for converting courses or curricula to ADL for nonresident education, DOD’s approaches are in fact consistent with mainstream practice, and in some cases, compare favorably with best practices in nonmilitary education. The processes for converting PME courses to ADL varied by school and by military service, and they feature a mixture of in-house and contractor approaches. However, the conversions were focused on the schools’ entire nonresident programs. USAWC’s and ACSC’s ADL applications were developed and managed by in-house staff and faculty. USAWC and ACSC used staff consisting of instructional designers and courseware developers interacting with respective faculty to develop courses. NWC’s ADL application combined the use of contractor and in-house support. Contractor staff created Web-based applications for two of the three courses in NWC’s curriculum. NWC officials learned enough from contractor efforts to create a Web-based application for the remaining course with in-house staff. In all cases, the ADL applications were applied to affect the entire nonresident curriculum and, in most cases, were preceded by reviews and final decisions from the schools’ academic boards. The PME schools did not identify any systematic criteria that inform their selection of course for conversion to ADL. Rather, they made subjective decisions as to the appropriate parts of the curriculum that should be delivered online based on content suitability. While USAWC’s application is delivered mostly through the Internet, print media delivers a portion of the course content as well. USAWC’s application also includes two 2-week resident components in which students are brought together to achieve learning objectives best suited to resident instruction. These objectives include verbal communication, interaction in live settings, interpersonal skills used in direct relationships, and other skills that are important components of the resident experience. USAWC’s application also uses asynchronous “threaded discussions,” in which faculty members initiate online discussions with students on various academic topics. NWC’s and ACSC’s applications, which do not include resident components, blend content delivery using print media, CD-ROM, and the Internet. In NWC’s application, print media is used for material that is mostly text and requires limited interactive capabilities; CD-ROMs are delivered to students for material that is not routinely updated; and students are assigned to cohort teams that allow online interactive opportunities and group discussions. In ACSC’s application, almost all nonresident materials are provided to students on CD-ROMs as well as on the Internet to allow as much flexibility as possible to complete nonresident courses. Generally, PME school officials stated that ADL conversions were made because of nonresident PME’s need (1) to respond to multiple learning styles in a diverse officer corps, (2) to increase the efficiency of educational delivery, and (3) to improve the quality of the educational offering. Additionally, NWC saw ADL as a means of potentially affecting retention and increasing enrollment. USAWC saw ADL as a means of responding to student demands for a more efficient and relevant educational experience. ACSC saw ADL as an improved means for delivering course material to increasing numbers of deployed officers. In nonmilitary applications of ADL, we observed processes and criteria for conversion decisions that were similar to DOD’s. Course conversions followed analogous processes and criteria decisions, driven by factors such as interest and student enrollment that were not systematic. University of Maryland University College officials stated that their conversion process includes a dedicated staff of instructional designers and subject matter experts (usually faculty) who produce course conversion to distance learning content within an established framework to ensure maintenance of standards. Their criteria for conversion focus on high-student demand and high levels of interest and on course work that required less “hands-on” training, such as business and information technology courses. Further research of private-sector practices supports the observation that the lack of systematic criteria is consistent with mainstream practice in ADL adoption for nonresident education. DOD’s approaches for course conversion are thus consistent with mainstream practice, and in some cases, compare favorably with the best practices in nonmilitary education. For example, NWC’s ADL application received the Crystal Award in 2002 from the Association for Educational Communications and Technology based on “innovative and creative use of the medium, instructional value and relevance, soundness of instructional strategy, quality of production, and evidence of successful outcomes.” As of June 2004, USAWC’s nonresident education program is fully accredited by the Middle States Commission of Higher Education for the awarding of Master’s of Strategic Studies Degrees. USAWC is the first military institution to achieve degree-granting authority for its nonresident ADL-based program. PME schools identified a number of cultural, technological, and resource challenges that affect ADL implementation and may affect future maintenance or expansion of ADL efforts. Cultural issues such as appropriate extent of ADL incorporation, general perceptions about nonresident education, and limited ADL research in military education affect the degree of ADL implementation. Technological trade-offs and nonresident program resourcing also affect continued ADL efforts. PME officials question the appropriate extent to which ADL should be used in nonresident education and how closely it can, or should, enable nonresident education to approximate resident education. It is generally recognized that resident programs are better in providing acculturation, interactive skills, and simulations that are critical for professional development of officers, and that there are challenges in providing such aspects in nonresident education. But some observers believe that nonresident education should not be compared to resident education and that nonresident education represents a vital broadening experience in its own right. In addition, there are indications that an ADL approach could significantly enrich nonresident content by excluding teaching methods that do not work in residence, allowing students the flexibility to focus on material that requires further study without class disruption, and serving as the basis for applications that can be used to upgrade teaching methods in resident programs. ADL implementation could also be affected by certain negative perceptions concerning nonresident education that are held by students, and in some cases, reflected in policy. Our survey of nonresident PME current students and recent graduates indicated that about 49 percent of current students and 48 percent of graduates believe they are not as well prepared as are their resident student counterparts, regardless of nonresident delivery method. While not universal between the services, we observed instances of military education policies that reinforce the perception that nonresident education is not as desirable as resident education. The Air Force’s PME Instruction AFI36-2301 states that “ideally, all officers will attend PME in residence,” and that limited resources restrict resident attendance to the “best qualified.” Furthermore, “completing nonresident PME programs will not affect eligibility for resident PME programs.” Indeed, we were told of instances where officers who after completing the nonresident PME program subsequently enrolled in the corresponding senior- or intermediate-level course in residence. The extent of ADL implementation in nonresident education is affected by the role that PME completion plays in promotional consideration. Programs maintained to foster promotional consideration—that is, “personnel-oriented”—might not be compatible with programs emphasizing the learning outcomes brought about by ADL—that is, “education-oriented.” Our survey shows that promotional considerations, rather than learning outcomes, are the focus for students in nonresident education. An estimated 73 percent of current students and 84 percent of recent graduates listed as their predominant reason for participating in nonresident PME a desire to improve their promotional chances or to meet promotional requirements. In addition, of an estimated 22 percent of PME graduates who were promoted to a higher rank after PME completion, an estimated 88 percent stated that PME contributed to a “great” or “very great extent” in their promotions. But ADL adoption goals should focus more on learning outcomes than on simply “checking the boxes” for promotional opportunity enhancement. PME officials state that there are concerns that ADL advantages could be oversold to the point that ADL may be used to supersede resident programs and shift the burden of PME administration. DOD officials, already viewing ADL delivery methods as attractive from a cost savings perspective, are desirous to expand such programs, even at the expense of resident programs. However, PME officials believe that ADL expansion should be considered only after completely understanding its impact on military operations and recognizing the resident program’s role as the basis for a nonresident program. In addition, PME officials noted that ADL could be used as a means of shifting the burden from the school to the student and the student’s commands without providing appropriate command support, compensation, or resources. While ADL research exists for military training courses, there is only limited research on ADL’s impact on military education, especially in terms of its impact on learning that requires interactive elements. A DOD official stated that it is well-known in studies that distance learning (usually teleconferencing) instruction offers no significant differences as compared with classroom instruction. However, officials believe that more work should be done to collect information that ADL improves learning in military education and that these studies should focus on collaborative learning environments and the extent of their translation. In addition, further efforts should look at commercial education studies (undergraduate and graduate education) and its transfer to military education. PME school officials have stated that decisions are needed on trade-offs between increased demands for student access (more servers, more bandwidth, or reduced firewalls) and the maintenance for network security. Such decisions are complicated by what is viewed as a lack of standardization in DOD and within individual services on security requirements. In our nonresident survey, approximately 19 percent of current students and 16 percent of recent graduates experienced computer/Internet related problems affecting their PME experience. Some identified problems included servers, bandwidth issues, and security firewalls. PME schools considered using “.edu” domains to make courseware more available to students because of the difficulty of interacting “.mil” domains with systems outside of military organizations. However, such moves would be expensive and would conflict with increasing requirements to reduce the number of servers and personnel needed to operate these systems. Some reported problems involved limited bandwidth issues. While such problems can be overcome, they require time and money to resolve. Firewalls maintained for security purposes have caused schools to limit library access to nonresident students due to perceived security threats. Most ADL efforts at PME schools were fielded independently with limited budgets and staffing. USAWC’s and ACSC’s ADL applications were developed and supported with in-house staff responsible for managing resident and nonresident programs. These applications were fielded independently within the services. PME officials stated that PME schools’ ability to fund ADL applications is limited due to DOD’s priority to focus more on its training and operational activities. An emerging funding issue involves the use of copyrighted material in ADL applications. Increasing costs in using copyrighted material for course work could result in limiting course flexibility and methodologies. New technologies, such as ADL, create new requirements for faculty personnel with higher technical expertise and more equipment and structure than traditional programs. Faculty members must be skilled to perform in online and classroom settings. PME schools are beginning to observe they must offer faculty opportunities to teach courses in multiple medias or risk losing qualified faculty to competitors. Although PME schools receive oversight from a number of organizations, we observed that neither the schools nor the oversight agencies had focused on (1) establishing specific performance effectiveness goals for ADL implementation or (2) measuring learning outcomes as a means of evaluating program effectiveness. The Joint Staff’s accreditation reports on nonresident education do not detail performance goals for any particular delivery method. The military services, which have primary responsibility for PME oversight, view the accreditation process provided by the Joint Staff as the primary means of ensuring the effectiveness of nonresident education. DOD is not alone in this problem—the lack of metrics for performance effectiveness and learning outcomes is pervasive throughout all educational institutions. Our prior efforts indicate that most public and private institutions lack a framework with which to assess implementation of training and development efforts. However, agencies need clear information on how training and development efforts affect organizational performance, and decision makers will likely want to compare the performance of these programs with that of other programs. Without clear goals and an effective process for evaluating the results of ADL application, DOD cannot ensure that ADL is achieving appropriate return on investment, student retention, student access, and other goals in comparison with prior efforts. Furthermore, programs lacking outcome metrics will be unable to demonstrate how they contribute to results. Moreover, by not capturing and using student data that are uniquely available through ADL techniques, DOD is missing the opportunity to develop the basis for effectiveness metrics and knowledge about learning outcomes. We recommend that the Secretary of Defense direct the Under Secretary of Defense for Personnel and Readiness, in concert with the Joint Staff, service headquarters, and the PME schools, to take the following two actions: promote the development of specific performance effectiveness goals promote the use of ADL technologies to capture data to provide knowledge about learning outcomes. DOD partially concurred with our first recommendation and fully concurred with the second. DOD supports the use of specific effectiveness goals for PME, but believes such goals are not appropriate for any specific delivery method. While we acknowledge DOD’s concerns with focusing on a specific delivery method, we believe that ADL is unlike other means of delivery because of its potential to modernize the educational experience and because its use is rapidly expanding in other areas of PME. We believe it would be worthwhile for DOD to know specifically how well ADL performs, especially in comparison with other delivery methods, in order to better understand its appropriate use for PME. DOD concurred with our second recommendation and stated that current accreditation practices are already promoting the data collection capabilities of ADL technologies for assessing multiple delivery methods. DOD’s comments are included in this report as appendix V. DOD also provided technical changes, which we incorporated as appropriate. We are sending copies of this report to congressional members as appropriate. We will also send copies to the Secretary of Defense; the Secretaries of the Air Force, Army, and Navy; the Commandant of the Marine Corps; and the Chairman of the Joint Chiefs of Staff. We will make copies available to others on request. In addition, this report will be available at no charge on the GAO Web site at http:/www.gao.gov. If you or your staff have any questions, please call me on (757) 552-8100 or Clifton Spruill, Assistant Director, on (202) 512-4531. Major contributors to this report were Arnett Sanders, Maewanda Michael-Jackson, Jean Orland, David Dornisch, Terry Richardson, and Cheryl Weissman. We reviewed Department of Defense’s (DOD) implementation of advanced distributed learning (ADL) in senior- and intermediate-level professional military education (PME) to determine processes and criteria used for converting courses, metrics to assess ADL effectiveness and its fulfillment of learning objectives, and barriers and challenges in ADL implementation. We collected, reviewed, and analyzed relevant program information and conducted interviews with DOD officials responsible for ADL implementation in PME programs and oversight responsibilities. We initially obtained data from the Principal Deputy Under Secretary of Defense for Personnel and Readiness to identify PME programs with ADL applications. A review of the data indicated that there were three existing programs. We identified, interviewed, and obtained data from officials from PME schools with ADL applications. Those schools were U.S. Army War College, Carlisle, Pennsylvania; Naval War College, Newport, Rhode Island; and Air Command and Staff College, Montgomery, Alabama. We also interviewed and obtained data from officials at the Joint Forces Staff College in Norfolk, Virginia, on their pending ADL application. We also interviewed and obtained data from agencies within DOD responsible for oversight of PME activities. Those agencies included The Joint Chiefs of Staff’s Operational Plans and Joint Force Development Directorate (J-7), Joint Doctrine, Education, and Training Division, Joint Education Branch, Arlington, Virginia; The Office of the Under Secretary of Defense for Personnel and Readiness, Deputy to the Under Secretary for Readiness, Office of Readiness and Training Policy and Programs, Arlington, Virginia; The Office of the Under Secretary of Defense for Personnel and Readiness, Deputy to the Under Secretary for Military Personnel Policy, Arlington, Virginia; Department of the Army, Office of the Deputy Chief of Staff, Operations and Plans, Arlington, Virginia; Department of the Navy, Office of the Chief of Naval Operations, Personal Development and Accessions Division, Washington, D.C.; Department of the Air Force, Office of the Deputy Chief of Staff for Personnel, Learning, and Force Development, Arlington, Virginia; U.S. Marine Corps Combat Development Center, Training and Education Command, Quantico, Virginia; U.S. Army War College Board of Visitors, Carlisle, Pennsylvania; Naval War College Board of Advisors, Newport, Rhode Island; and Air University Board of Visitors, Montgomery, Alabama. To determine sufficient metrics to assess ADL effectiveness, we provided PME program officials with a detailed list of questions that included those relating to effectiveness and learning objectives. We reviewed written responses, if provided, and followed up with site visits and correspondence with oversight agencies to clarify or obtain additional information if necessary. We also obtained and analyzed data from a survey of nonresident PME current students and graduates, which included questions designed to obtain perceptions on program effectiveness. Details of the survey methodology are presented in appendix II. To determine processes and criteria DOD used for ADL conversion, we provided PME program officials with a detailed list of questions that included those relating to process and criteria decisions. We reviewed written responses, if provided, and followed up with site visits to clarify or obtain additional information if necessary. To determine whether criteria were consistent with those of other institutions performing distance education, we researched prior literature on this topic and conducted a site visit to the University of Maryland University College in Adelphi, Maryland. The school was identified in our prior reports on distance education as having a program with a large distance education population, as well as educating a significant number of military officers. We also contacted and received data from the Sloan Consortium, an organization designed to encourage collaborative sharing of knowledge and effective practices to improve online education. To determine barriers and challenges to ADL implementation, we provided PME program officials with a detailed list of questions that included those relating to barriers and challenges. We reviewed written responses, if provided, and followed up with site visits and correspondence with DOD oversight agencies to clarify or obtain additional information if necessary. We also obtained and analyzed data from a survey of nonresident PME current students and graduates, which included questions designed to obtain perceptions on barriers and challenges in completing PME courses. Details of the survey methodology are presented in appendix II. To obtain military officers’ perspectives on nonresident PME in terms of impact on careers, achievement of learning objectives, and obstacles and challenges, we conducted a statistically representative survey of current nonresident senior- and intermediate-level PME students and graduates of these schools from April 1999 to March 2003, roughly the period coinciding with initial ADL implementation at several senior- and intermediate-level schools. We present the survey questions and response results in appendix III. The population for the nonresident PME survey consisted of current students and graduates who fulfilled the following criteria: 1. Respondents were identified as enrolled in a senior- and intermediate-level nonresident PME program of study from April 1999 to March 2003. We decided on this time period to ensure that our respondents would have begun their programs after Web-based PME had been clearly established as a mode of instruction or have been in PME long enough to have meaningful responses to our questions. 2. Respondents participated in a senior- and intermediate-level nonresident PME program of study, as opposed to individual PME courses undertaken via continuing education programs. 3. Respondents are currently active in the U.S. military services or reserves, excluding U.S. civilians; U.S. Coast Guard members; and international members, either military or civilian. 4. Respondents participated (i.e., currently enrolled or graduated) in a nonresident PME program in one of the six senior- and intermediate-level PME schools: U.S. Army War College, Army Command and General Staff College, Air War College, Air Command and Staff College, Naval War College, and Marine Corps College of Continuing Education. The survey asked respondents about PME’s impact on furthering career objectives, their achievement of learning objectives, and obstacles and challenges of the programs. Specific questions concerned students’ satisfaction with their overall program, various modes of program delivery, and with technologies used; students’ time and duty management concerns; and reasons for participation in nonresident PME. To develop areas of inquiry for the survey, we reviewed our previous work related to distance education and PME. We reviewed a series of survey questionnaires developed by us and by DOD. We used these sources and our own analysis to develop an initial set of questions. We further developed and refined the questionnaire by obtaining and incorporating written comments regarding the initial questions from administrators and other representatives of the senior- and intermediate-level PME schools. In addition to an internal expert technical review by our Survey Coordination Group, we pretested the survey with five individuals whose personal characteristics corresponded to our eligibility criteria. We identified pretest subjects through our contacts who were current military personnel or who knew military personnel and our PME points of contact. We conducted the survey between January and April of 2004 on a random sample of 437 current students and graduates of nonresident PME programs using a self-administered Web-based questionnaire. We drew the names of our respondents from an overall population data set we constructed that combined separate data sets received from each of the senior- and intermediate-level PME schools. For each data set, we requested the officer’s name, school attended, month and year of initial enrollment, component (defined as either active duty or reservist), and mode of PME delivery. We requested e-mail addresses and, if needed, phone numbers from potential respondents after they were drawn from the population data sets. We stratified our sample by component in order to better understand any differences between these components. We activated the survey Web site and informed our sample respondents of the Web site, their logon name, and passwords by e-mail on January 30, 2004. To maximize the response rate, we sent five subsequent follow-up e-mail reminders to nonrespondents in February and March 2004. At the time of the third mailing, we also telephoned many of the nonrespondents to encourage them to complete the survey. We ended data collection activities on April 30, 2004. Of the 437 selectees included in our sample, we received 273 useable questionnaires. We defined useable as respondents who completed the survey and were not identified as out-of-scope. During the survey, we deemed 67 of the 437 to be outside the scope of our survey after determining that they did not meet at least one of our eligibility criteria. Disregarding these 67 responses, our overall response rate was 73.8 percent (273/370). Table 2 shows the final disposition of the sample (the 437 respondent accounts activated) by strata. The estimates we make in this report are the result of weighting the survey responses to account for effective sampling rates in each stratum. These weights reflect both the initial sampling rate and the response rate for each stratum. As with many surveys, our estimation method assumes that nonrespondents would have answered like the respondents. For the estimates we present in this report, we are 95 percent confident that the results we would have obtained had we studied the entire population are within +/-10 or fewer percentage points of our estimates (unless otherwise noted). Because we surveyed a sample of recent nonresident PME students, our results are estimates of student and graduate characteristics and thus are subject to sampling errors. Our confidence in the precision of the results from this sample is expressed in 95 percent confidence intervals, which are expected to include the actual results for 95 percent of the samples of this type. We calculated confidence intervals for our results using methods appropriate for a stratified probability sample. We conducted in-depth pretesting of the questionnaire to minimize measurement error. However, the practical difficulties in conducting surveys of this type may introduce other types of errors, commonly known as nonsampling errors. For example, measurement errors can be introduced if (1) respondents have difficulty interpreting a particular question, (2) respondents have access to different amounts of information in answering a question, or (3) errors in data processing occur. We took extensive steps to minimize such errors in developing the questionnaire, collecting the data, and editing and analyzing the information. The Web-based data management system we used provides a systematized process for processing, transferring, and analyzing data that also protects against nonsampling errors. In addition, we performed tests to ensure the reliability and usefulness of the data provided by the PME schools. These included computer analyses to identify inconsistencies both within and across the data sets and other errors in the data sets from which we developed our overall sampling frame. We also interviewed agency officials knowledgeable about the data. We determined that the data were sufficiently reliable for the purposes of this report. Welcome DoD Distance Learning Student or Graduate. The U.S. General Accounting Office (GAO) - an agency of the U.S. Congress has been requested to review various aspects of nonresident professional military education (PME). Part of that effort is to evaluate whether the appropriate decisions are being made in regards to nonresident education for intermediate and senior-level military officers. As part of this effort, we are assessing opinions of nonresident PME graduates and current students on (1) the achievement of PME learning objectives, (2) obstacles and challenges in completing PME, and (3) PME’s impact on furthering career objectives. The survey should take approximately 10 minutes to complete. Participation in this survey is voluntary but encouraged. Your responses will be confidential and the results of the survey will be reported in aggregate form only. Before choosing an answer, please read the full question and all response choices carefully. Thank you in advance for your participation. Please consider only the following when answering the questions on this survey: 1. Nonresident Professional Military Education programs of study that you have participated in, as opposed to individual PME courses undertaken via continuing education programs. 2. Nonresident Professional Military Education programs of study that you began in April 1999 or after. Our survey of nonresident Professional Military Education students was divided into two main parts, one with questions appropriate for current students and one with similar questions worded slightly differently for graduates. There are also questions on demographics to which both current students and graduates responded. Survey questions and responses for graduates are indicated in italics and those for current students are in plain text. The information provided here represents weighted data. For information on weighting, see appendix II. Questions 1 and 31 are intentionally omitted because they contained instructions telling respondents which questions to answer. Q2. The name of program in which currently enrolled: Q32. From which school did you graduate? Air War College Nonresident Program (percent) Army Command and General Staff College Nonresident Program (percent) Army War College Distance Education Program (percent) Naval College of Distance Education (percent) (percent) Q3. In what month and year did you begin your PME program? Q33. In what month and year did you begin your PME program (if you graduated from more than one program, answer for the most recent one)? (percent) (percent) (percent) (percent) Q4. What mode of instruction have you used most often in your nonresident Professional Military program? Q34. What mode of instruction did you use most often in your nonresident Professional Military program? (percent) (percent) Q5. In what month and year do you expect to complete your PME studies program? Q35. In what month and year did you complete your Professional Military Education program? (percent) (percent) (percent) (percent) (percent) (percent) (percent) (percent) (percent) (percent) Q6. In a typical week, approximately how many hours did you spend in Professional Military Education-related activities, including preparation, study, working on-line, and time in class? Q36. In a typical week, approximately how many hours did you spend in Professional Military Education-related activities, including preparation, study, working on-line, and time in class? Q7. Does the military or your employer afford you time during your work-week for Professional Military Education? Q37. Did the military or your employer afford you time during your work-week for Professional Military Education? (percent) Q8. How many hours do you work in paid employment in a typical work-week (outside of Professional Military Education-related activities)? Q38. During the period of time that you were completing your Professional Military Education program, how many hours did you work in paid employment in a typical work-week (outside of Professional Military Education-related activities)? Q9. Listed below are various reasons why someone would participate in a nonresident PME program. What is your greatest reason for participating in a nonresident Professional Military Education program? Q39. Listed below are various reasons why someone would participate in a nonresident PME program. What was your greatest reason for participating in a nonresident Professional Military Education program Current Students (percent) (percent) To gain knowledge in my field or in fields of interest to me To improve my chances of, or meet the requirements for, promotion Q10. To this point in time, how satisfied or dissatisfied are you with your Professional Military Education program? Q40. Overall, how satisfied or dissatisfied are you with the Professional Military Education program in which you graduated? Somewhat satisfied (percent) (percent) (percent) (percent) Q11. To what extent, if any, has your Professional Military Education program benefited your military career to this point in time? Q41. To what extent, if any, has graduation from your Professional Military Education program benefited your military career to this point in time? (percent) (percent) (percent) (percent) 12 Q12. To what extent, if any, do you believe the knowledge you are acquiring in your Professional Military Education program will improve your effectiveness in future assignments? Q42. To what extent, if any, do you believe the knowledge you acquired in your Professional Military Education program has improved your effectiveness in job assignments? (percent) (percent) (percent) (percent) Q43. Have you been promoted to a higher rank since you completed your Professional Military Education program? (percent) Q44. To what extent, if any, do you believe that completion of your Professional Military Education program contributed to your promotion? (percent) (percent) (percent) (percent) Q13. To what extent, if any, does your Professional Military Education program enable you to acquire the knowledge you are expected to obtain? Q45. To what extent, if any, did your Professional Military Education program enable you to acquire the knowledge you were expected to obtain? (percent) (percent) (percent) (percent) Q14. Was/is any part of your Professional Military Education program taken through seminar/classroom-based instruction? Q46. Was any part of your Professional Military Education program taken through seminar/classroom-based instruction? (percent) 50 Q15. Overall, how would you rate the quality of this seminar/classroom-based learning? Q47. Overall, how would you rate the quality of this seminar/classroom-based learning? (percent) (percent) (percent) (percent) Q16. Was/is any part of your Professional Military Education program taken through paper-based correspondence? Q48. Was any part of your Professional Military Education program taken through paper-based correspondence? (percent) Q17. Overall, how would you rate the quality of this paper-based correspondence learning? Q49. Overall, how would you rate the quality of this paper-based correspondence learning? (percent) (percent) (percent) (percent) Q18. Was/is any part of your Professional Military Education program taken through the World-Wide Web or Internet?) Q50. Was any part of your Professional Military Education program taken through the World-Wide Web or Internet? (percent) Q19. Overall, how would you rate the quality of this web/Internet-based learning? Q51. Overall, how would you rate the quality of this web/Internet-based learning? (percent) (percent) (percent) Q20. How easy or difficult has it been for you to use web/Internet-based learning? Q52. How easy or difficult was it for you to use web/Internet-based learning? (percent) (percent) (percent) (percent) Q21. How easy or difficult have you found interaction with faculty during your web/Internet-based learning? Q53. How easy or difficult did you find interaction with faculty during your web/Internet -based learning? (percent) (percent) (percent) (percent) Q22. How easy or difficult have you found interaction with other students during your web/Internet-based learning? Q54. How easy or difficult did you find interaction with other students during your web/Internet-based learning? (percent) (percent) (percent) (percent) Q23. How well does the courseware/course software work on the computer equipment to which you have access for taking web/Internet- based learning? Q55. How well did the courseware/course software work on the computer equipment to which you had access for taking web/Internet- based learning? (percent) (percent) (percent) (percent) 2 Q24. How reliable is your network access for taking web/Internet-based learning (e.g. ability to connect; to upload and download assignments, etc). Q56. How reliable was your network access for taking web/Internet-based learning (e.g. ability to connect; to upload and download assignments, etc)? (percent) (percent) (percent) (percent) Q25. Compared to resident Professional Military Education students in the school in which you are enrolled, of the following options, do you believe you are prepared: Q57. Compared to resident Professional Military Education program graduates of your school, of the following options, do you believe you are prepared: as well as resident students. (percent) worse than resident students. (percent) (percent) Q26. Overall, what has been the primary challenge, if any, affecting your Professional Military Education program? Q58. Overall, what was the primary challenge, if any, affecting your Professional Military Education program? Current Students (percent) (percent) Not applicable, I am not experiencing & have not experienced any challenges to this point in time. Q27. Have you experienced any computer/Internet-related problems affecting your Professional Military Education program? Q59. Did you experience any computer/Internet-related problems affecting your Professional Military Education program? (percent) 16 Q28. What specific computer/Internet-related problems have you incurred? Q60. What specific computer/Internet-related problems have you incurred? (percent) (percent) Q29. At any point during your Professional Military Education program, have you had to defer/disenroll from your studies? Q61. At any point during your Professional Military Education program, did you have to defer/disenroll from your studies? (percent) Q30. What was the primary reason you had to defer/disenroll from your studies? Q62. What was the primary reason you had to defer/disenroll from your studies? Q63. What is the highest degree or level of school that you have completed? Current Students (percent) Q64. In what branch of the military do you serve? (percent) Q65. What duty capacity best describes you during the majority of your Professional Military Education program? (percent) Q66. What component best describes you during the majority of your Professional Military Education program? (percent) Q67. Are you a member of the National Guard? (percent) 55Q68. What military occupational category best describes you during the majority of your Professional Military Education program? Current Students (percent) (percent) Q69. What was your rank when you began your Professional Military Education program? (percent) (percent) 3 Q71. If you have any other comments related to your PME education, training, assignments, distance learning, or any other matters related to this questionnaire, please note them here. We observed three current ADL applications at PME senior- and intermediate-level schools. These schools have geared their ADL efforts to their nonresident programs. The programs vary from service to service in terms of enrollment, structure, duration, and credits received for graduation. In addition, we observed additional features of nonresident programs that affect the nature of their ADL applications. The U.S. Army War College (USAWC), the Army’s senior-level PME school, initiated its Web-based nonresident education program in April 1999. The program went online in an evolutionary process until the spring of 2002, whereby students received both text and online versions. Since the spring of 2002, all nonresident students have received their education via a combination of ADL technology and appropriate text. Nonresident students are board selected to participate in the program. It is a 2-year Web-based program that is the only delivery method offered to nonresident students. The program has a “blended” component, whereby 2 of its 12 courses are taken in residence at USAWC. Also, distance courses are presented to students as a group or cohort; that is, students enroll at the beginning of the nonresident school year and must complete a sequenced load of 5 courses during the first year, followed by an additional 5 courses during the second year. The resident courses are of 2-week duration and are conducted at the end of each academic year. The nonresident program is designed to parallel the resident program, and graduates from both programs are awarded Master’s Degrees in Strategic Studies. The Naval War College’s (NWC) nonresident education programs are concentrated in its College of Distance Education, its only nonresident college and one of five colleges under the NWC umbrella. The College of Distance Education, an intermediate-level PME school, offers several nonresident options. The fleet seminar program has existed in various forms at the school since 1974; the Web-enabled correspondence program has been operating fully since October 2002; and the CD-ROM-based correspondence program, effective in April 2004, was designed to replace the phased-out paper-based correspondence course. Nonresident options are open to all officers and qualified personnel. The Web-based course can be completed in 18-24 months. While there is no formal resident portion to this course, students are assigned to cohort teams to facilitate team and faculty communication. This nonresident course is closely aligned with the resident course, and graduates are allowed to obtain graduate hour credits. In the case of several seminars of the fleet seminar program, students can apply for admission to a program of graduate study leading toward a Master’s of Arts Degree in National Security and Strategic Studies. The Air Command and Staff College (ACSC), the Air Force’s intermediate-level PME school, implemented its nonresident program in its present form in September 1999. There are two methods for completing the nonresident program: by seminar or by correspondence. The ACSC nonresident program is open to all officers and qualified personnel. The most recent version of the program consists of six courses organized into two semesters. The seminar method, which can take up to 11 months to complete, is conducted weekly, is typically composed of 3-18 students, and is led by assigned seminar leaders in order to facilitate group discussions. The correspondence program, a self-study program delivered in a balanced manner consisting of paper, CD-ROM, and Web-based delivery, requires up to 18 months to complete. Students move interchangeably between both programs, but they must achieve a minimum score of 70 percent on each of the six examinations and must complete four interactive Web-based exercises. The nonresident programs are designed to mirror resident programs, and there are multiple versions in use by ACSC nonresident students. These programs do not award master’s degrees, but the American Council of Education recommends up to 21 semester hours of graduate credit for course completion. National Defense University’s Joint Forces Staff College (JFSC) is piloting an Advanced Joint Professional Military Education pilot course for senior- and intermediate-level reserve officers. Initially launched in September 2003, it is designed to last 38 weeks. The period consists of 35 weeks of Web-based education and 3 weeks of resident education with 1 week occurring after the first 8 weeks of Web-based education, and the last 2 weeks at the end of the 38-week period. JFSC, already responsible for the resident Joint PME Phase II course used to complete the education process for joint specialty officers, was tasked to develop a Joint PME course for reserve officers in response to the fiscal year 1999 National Defense Authorization Act and the Joint Staff Guidance in May 2000. While there is no joint specialty officer requirement for reserve officers, JFSC was required to prepare reserve officers for joint duty assignments by providing a course similar in content to its resident Joint PME course, and to do so by utilizing current distance learning applications. There are additional features of PME programs that affect the nature of their ADL applications. Those features include: Student Board Selection—Nonresident students are selected to attend the PME schools either through an annual board selection process or through open admissions. Only USAWC selects its nonresident students; the other programs with ADL applications have open-admission policies. Joint Professional Military Education—A significant portion of the PME curriculum involves study of joint service issues along with service-specific issues. Officers who successfully complete senior- or intermediate-level PME course work are awarded Joint PME Phase I credit, which is required for those who wish to serve as joint specialty officers. All nonresident programs with ADL applications grant Joint PME Phase I credit. Service Promotion Impact—PME officials stated that PME program completion and other forms of higher education are factors used in consideration for promotion and vary among the services. Generally, the Air Force requires completion of a corresponding PME level of study before a candidate is considered for the next promotion level. The Army, while not as strict as the Air Force, places a high value on PME and graduate education in promotion decisions. The Navy, placing a higher premium on operational experience, currently is less inclined to recognize PME as a credential for promotion. Learning Objectives Between Programs—PME officials stated that, as outlined by Joint Staff policies, learning objectives for nonresident courses are required to be the same for resident courses, regardless of the method of delivery. PME schools have instituted internal control processes to ensure the achievement of learning objectives for all programs, irrespective of delivery method. Generally, PME schools apply similar evaluation systems and criteria to both resident and nonresident programs. Levels-of-Learning—PME schools teach to differing achievement levels across and within the services, and they have designed their curricula accordingly. School officials refer to these achievement levels as “levels-of-learning” based on a taxonomy defined in the Joint Staff policy. (See table 3 for a detailed definition of levels-of-learning designations.) For the schools with ADL applications, their desired levels of learning for nonresident programs may or may not be equivalent to the corresponding resident programs: USAWC—Synthesis/Analysis (same as for resident program). NWC—Application (resident program calls for synthesis/analysis). ACSC—Comprehension (resident program calls for synthesis/resident). JFSC (Planned)—Application (same as for resident program). The following is GAO’s comment on the letter from the Department of Defense dated July 23, 2004. 1. When we initiated this engagement in February 2003, a key objective was to determine (1) the assumptions for DOD’s decision to move officer senior- and intermediate-service schools from 1-year residency to shorter periods by using ADL and (2) which courses and schools would be affected. Immediately after fieldwork commenced, however, DOD informed us that it was no longer actively pursuing that approach. In April 2003, after consulting with the congressional requester, we informed our DOD point of contact regarding our pursuit of the engagement’s remaining objectives. The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to www.gao.gov and select “Subscribe to Updates.”

Summary: As part of its transformation to prepare the armed forces to meet current and future challenges, the Department of Defense (DOD) is expanding its use of advanced distributed learning (ADL) techniques in senior- and intermediate-level officer professional military education (PME). ADL instruction does not require an instructor's presence, and it facilitates the use of varied learning management systems. To date, the application of ADL has been targeted to nonresident students. To determine whether DOD uses a systematic process for evaluating the results of ADL application, GAO was asked to examine DOD's metrics for assessing program effectiveness, to compare DOD's criteria for converting courses to ADL with those of private-sector institutions, and to identify the challenges to ADL implementation. DOD does not have specific performance goals and metrics with which to assess ADL effectiveness in PME. Furthermore, although GAO and private-sector organization have established frameworks for assessing the effectiveness of educational programs by focusing on metrics for learning outcomes--that is, the knowledge, skills, and abilities that students attain through learning activities--DOD's oversight focuses instead on educational inputs such as facilities, student to faculty ratios, and student body composition. Since ADL is still a new and evolving tool, systematic evaluative processes have not yet been required. Without clear goals and an effective process for evaluating the results of ADL application, DOD cannot ensure that its program is achieving an appropriate return on investment and other goals. The criteria for converting PME courses and curricula to ADL vary by school and by military service, are based on subjective choices as to which content is suited for online delivery, and are focused solely on nonresident programs. The private sector similarly lacks systematic criteria in its use of ADL. However, DOD's implementation of ADL programs for PME compares favorably with private-sector institutions. Cultural, technological, and resource challenges affect ADL implementation. For example, some military policies reflect a lower estimation of the value of nonresident PME, and many respondents to a survey of ADL students and alumni indicated that its quality and achievement of outcomes did not compare favorably, in their view, with those of resident education programs. The technological challenges of balancing computer access with network security, along with resource challenges of funding and increased burdens on limited administrative staff, are additional concerns.