metadata: authors: 0: author: Angel-Urdinola,Diego
1: author: Castillo Castro,Catalina
2: author: Hoyos,Angela
count: World
docna: 0: docna: Meta-Analysis Assessing the Effects of
            Virtual Reality Training on Student Learning and Skills Development
docty: Policy Research Working Paper
subtopic: Educational Sciences,Health Care Services Industry,Health Service Management and Delivery,Gender and Economics,Economics and Gender,Gender and Poverty,Gender and Economic Policy
teratopic: Education,Industry,Health, Nutrition and Population,Macroeconomics and Economic Growth,Gender
repnme: repnme: Meta-Analysis Assessing the Effects of
            Virtual Reality Training on Student Learning and Skills Development
display_title: Meta-Analysis Assessing the Effects
            of Virtual Reality Training on Student Learning and Skills Development
listing_relative_url: /research/2021/03/32925750/meta-analysis-assessing-effects-virtual-reality-training-student-learning-skills-development
topicv3: Education, Skills Development and Labor
            Market m1326135 1484,Education m1327048 470,Gender and Skills m1327153 912,Health m1504661 3480,Surgery m1504689 3616
publishtoextweb_dt: 2021-03-18T00:00:00Z
dois: 10.1596/1813-9450-9587
historic_topic: Education,Industry,Health, Nutrition and Population,Macroeconomics and Economic Growth,Gender
origu: Education LCR (HLCED)
no_of_pages: 52
author_keys: 0: author_key: 438716
1: author_key: 1106912
model_extractions: Policy Research Working Paper 9587 Meta-Analysis Assessing the Effects of Virtual Reality Training on Student Learning and Skills Development _Diego F. Angel-Urdinola Catalina Castillo-Castro Angela Hoyos_ Education Global Practice March 2021
Policy Research Working Paper 9587

# **Abstract**

Training using virtual reality has been applied in many fields of education, but primarily in the fields of health and safety, engineering and technical education, and general education. Numerous studies assessing the use of immersive training in education have yielded promising results in educational outcomes, but there is not yet in the literature a systematic analysis of the effects of virtual reality training on student learning. This paper presents a meta-analysis of the results of available studies that assess virtual reality training’s impact on student learning and skills development, and which rely on robust evaluation methods. The study’s primary purpose is to identify the extent to which immersive training can successfully develop students’ skills across different fields of education and the size of the effects encountered. The analysis presented here relies on 31 primary studies and more than 90 experiments. The results indicate that, on average, virtual reality training is more effective than traditional training in developing technical, practical, and socio-emotional skills. The results are particularly promising in fields related to health and safety, engineering, and technical education. The results also indicate that students who are exposed to virtual reality training are more efficient in using inputs and time and/or avoiding performance errors than students receiving traditional training.

This paper is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at dangelurdinola@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# Meta-Analysis Assessing the Effects of Virtual Reality Training on Student Learning and Skills Development

Diego F. Angel-Urdinola, Catalina Castillo-Castro, Angela Hoyos The World Bank<sup>1</sup> JEL Classification: I20, I24

**Keywords:** Virtual reality, Education, Learning, Skills Development. **Corresponding Author:** Diego F. Angel-Urdinola (dangelurdinola@worldbank.org)

> 1 This paper is a product of the Education Global Practice. The paper was prepared as a background paper for the _Pilot Program to Test Virtual Reality Training Programs for Technological and Technical courses in Higher Education (TF0A8313)_ , supported by the Korea World Bank Partnership Facility (KWPF). The authors acknowledge useful comments and support from Christine H. Joo, Robert Hawkins, and Michael Trucano.
# **I. Introduction**

Recent events, such as the health pandemic introduced by the COVID-19 virus, have contributed to speed up alternative mechanisms to offer digital instruction to substitute and complement in-class instruction. The expansion of digital and computer assisted learning is becoming a global trend, making it of extreme importance to identify technology tools that work, while being scalable and cost-effective (Escueta, Quan, Nickow, & Oreopoulos, 2017). Even before the pandemic, it had become particularly challenging for education systems to supply digital learning opportunities that provide students the hands-on pedagogical experiences necessary to develop practical skills, especially for programs that require the use of laboratories.

Virtual reality (VR) training is often known as the process of learning in a simulated or artificial environment. VR training has existed in the realm of education for over half a century but has dramatically expanded over the past 15 years as VR simulators are becoming less expensive to develop and increasingly realistic. The term VR applies to computer-simulated environments that can imitate physical presence in places in the real world, as well as in imaginary worlds (Lorenzo, Pomares, & Lledó, 2013), and simulate the illusion of participation in a synthetic environment with an external observation of such surroundings (Gigante, 1993). VR simulations can be constructed employing 3D graphics using a desktop computer (non-immersive) or using a head-mounted display (immersive) (Makransky, Terkildsen, & Mayer, 2019). In non-immersive VR, the simulated environment is displayed on a conventional computer with sound and graphics coming through the computer’s speaker and monitor, and the interaction is controlled through a regular computer mouse. Immersive VR uses a head-mounted display in which a high graphical fidelity screen is mounted in front of the user’s eyes with separate lenses for each eye and with sound delivered through earphones. The interactions in the context of high-immersion VR are controlled through head-motion tracking in conjunction with a computer system that allows users to look around a simulated 360-degree environment.

In some educational fields, the development of adequate cognitive, technical, and socioemotional skills remains a challenge for trainees and their tutors, partly because of the limited availability of hands-on training or access to proper content and learning situations. As a response, educators are starting to rely on VR simulations to develop learning experiences that would otherwise not be easily accessible to students. VR simulations can provide students practical 2
training opportunities without pressure, danger, and allowing repeated interventions. Also, VR simulations can provide students access to situations and learning environments (such as traveling within a cell, simulated scenarios for public speaking, among others) that would otherwise be very difficult or impossible to access. Such opportunities can accelerate students' learning curve in a simulated environment, reproducing real-life conditions and situations without time or space limitations and much fewer risks than real environments. In addition, VR simulations offer the great advantage of providing students and teachers a standardized, reproducible environment for repeated and optimized training (Apostolellis, Bowman, & Chmiel, 2018; Cheung, Fong, Fong, & Wang, 2013; Ferracani, Pezzatini, & Del Bimbo, 2014; Huang, Rauch, & Liaw, 2010; Sharma, Agada, & Ruffin, 2013).

Another advantage of using VR simulations is that gamification, performance metrics, and collaborative features (using avatars) can be embedded in the software, enabling continuous peer interaction, active learning, enjoyment, and performance feedback – all elements that enhance proficiency-based training. Indeed, constructivism is often cited as a theoretical framework that supports the implementation of learning in virtual environments. Constructivism suggests that students learn by constructing knowledge and incorporating it into their existing knowledge structure. Thus, constructivist learning environments can increase active learning, motivation, interactivity, and personalized learning (Madathil et al., 2017).

Proponents of VR simulations claim that higher motivation and presence are the main two channels through which VR training simulations can influence student learning (Mikropoulos, & Natsis, 2011). As a result, VR simulations have been regarded as a pedagogical method with the potential to increase student learning, as they increase self-motivation to learn and allow embedding to the educational experience constructivist pedagogy, collaboration, and gamification (Kavanagh, Luxton-Reilly, Wuensche, & Plimmer, 2017).

The impact of media on student learning outcomes has been highly debated among educational technologists where much of the prior literature has shown no significant difference between technology-based and traditionally delivered instruction and media. However, the counterargument contends that using the correct media could impact students’ cognitive skills and that the media itself is a critical component of instructional design (Madathil et al., 2017).

3
Given the rising importance of identifying digital education platforms that work, this paper conducts a meta-analysis of the results of available experiments that assess the impact of VR training on learning and skills development. The study's primary purpose is to identify the extent to which VR training is conducive to learning and skills development. A secondary objective is to assess, to the extent possible, if VR training is also an efficient mechanism to deliver training. The analysis presented here relies on a total of 31 primary studies and over 90 different experiments.

There has not been a systematic assessment of the effects of VR training on learning, other than in the field of surgical education in the early 2000s (Haque, & Srinivasan, 2006). This study was conducted with a limited number of studies and focused on assessing the extent to which VR training could help students perform surgical procedures faster (i.e. improve their time-intask). To bridge the knowledge gap, this study focuses on a more recent time period (2005-2020), during which VR technology has significantly evolved, and covers other fields such as engineering, science, technical education, and general education. Moreover, this study analyzes the effects of VR more holistically as a mechanism to develop cognitive, technical, and socioemotional skills. As such, the findings of this paper represent an essential contribution to the literature and intend to guide education institutions and policy makers to have more information about the effects of VR training as they expand their offer of digital learning opportunities to students.

Based on the information available, our research shows that VR training is, on average, more effective than traditional training as a mechanism to develop students' technical, practical, and socio-emotional skills. Results are particularly promising in fields related to health and safety, engineering, and technical education. Results reveal that for each additional hour (1⁄4 hour) of VR training, students score 3 percent higher in technical (cognitive) learning assessments than students exposed to the same curricular content delivered through traditional training methods. Results also indicate that students exposed to VR instruction report on average 30 percent higher scores in socio-emotional skills assessments after completing their training than their peers receiving traditional instruction. Results also suggest that students exposed to VR training are up to 30 percent more efficient using inputs, time, and/or avoiding performance errors than students exposed to traditional training, per additional hour of instruction.

4
The paper is structured as follows. Section II provides a review of the literature on the use of VR for pedagogical purposes. Section III presents the data and methodology used to conduct the meta-analysis. Section IV presents the results of the meta-analysis and intends to quantify the observed effects of VR training on students’ learning outcomes and skills development. The conclusion follows in Section V.

# **II. Literature review**

Recent studies assessing the effect of the use of VR simulations in education show promising findings, in different areas, from increased time-on-task, enjoyment, motivation, and learning. Nonetheless, there is not a recent systematic analysis of the effects of VR training on student learning and skills development. A recent review of the literature (Kavanagh, LuxtonReilly, Wuensche, & Plimmer, 2017) shows that VR simulations are used in many education fields, but primarily in health and surgical education, engineering and technical education, and general education (mainly in STEM related fields). Since each of these fields uses VR with slightly different pedagogical purposes, our literature review discusses each of these fields independently.

# _2.1. VR training for health and safety_

The use of VR training has shown great potential within the field of health, especially in the area of surgery, as it offers trainees the opportunity to practice several surgical procedures in a safe environment and at a comparatively low cost. Simulators provide excellent benefits to surgical trainees by allowing for repeated practice of a specific skill set in a controlled and safe environment, before ever entering the operating room. VR training allows developing surgical training experience that can enable junior trainees to undertake self-directed training while practicing and learning the fundamentals of surgery procedures without putting patients at risk and without needing supervision from an attending surgeon. Also, VR training can provide junior trainees relevant experience at an early stage in their surgical training while giving them an exposure to otherwise scarce educational resources, such as cadaveric parts (Zhao, Kennedy, Yukawa, Pyman, & O'Leary, 2011). A meta-analysis on the effects of VR training for surgery training was first made in 2006 (Haque & Srinivasan, 2006). While the study was limited in scope (only assessed the impacts of VR simulators on task completion time), it concluded that VR simulators did lessen the time trainees take to complete a given surgical task.

5
Other studies have shown that simulation-based training of surgical skills can improve medical personnel performance in the operating room and diminish complication rates related to inexperience (Gallagher et al., 2005). For instance, VR laparoscopic simulators and robotic surgery have been extensively used in health practice (Gurusamy, Aggarwal, Palanivelu, & Davidson, 2008; Valdis, Chu, Schlachta, & Kiaii, 2016). Laparoscopic and robotic surgery have become a standard approach for many surgical specialties, as they reduce patient’s surgical trauma, faster postoperative recovery, shorter hospital stays, and are associated with better cosmetic results. By using virtual reality simulator training, surgeons are expected to improve their proficiency and speed up their learning curve to master these procedures (Larsen et al., 2009).

Similarly, the use of VR training for eye surgery as well as for other uncomfortable procedures for both the patient and the examiner, such as transvaginal examinations and infant sedation, has been promoted extensively in the medical practice (Chao, Chalouhi, Bouhanna, Ville, & Dommergues, 2015; Zaveri et al., 2016). Finally, the use of VR training for some procedures, such as bone surgery and total hip arthroplasty, has shown to be effective addressing limited access to resources that are necessary for making practical training possible, namely real human bones, as well as decreasing surgical errors (such as the incorrect alignment of the hip).<sup>2</sup> Safety and risk prevention are fields where VR training has also shown significant potential. As disasters and accidents are recurrent in all areas, training on safety and risk prevention is essential to mitigate their incidence and provide a rapid response and minimize casualties. VR training allows participants to emulate situations that may otherwise not be accessible with traditional learning methods. Immersive VR simulators have the potential to expose individuals to situations where high-level performance is critical but difficult to rehearse, such as mass disasters, evacuation drills, firefighting, and other hazardous or toxic conditions (Farra et al., 2018). Training emergency response personnel for catastrophes, for instance, is difficult due to the inability to replicate a given disaster environment comprehensively. In addition, there is an ethical concern about exposing trainees to the emotional and physical stresses encountered in real casualty situations (Andreatta et al., 2010). Available disaster drills often rely on mock patients, and they

> 2 Although cadaver temporal bones remain the gold standard of simulated training for temporal bone surgery, their increasing scarcity worldwide has meant that additional training tools are currently being explored. In addition to a shortage of cadaver bones, the increasing workload of attending surgeons has meant that the time that can be devoted to teaching and education has decreased.

6
can be very costly. Also, available disaster drills do not provide opportunities for on-demand repetitive training. In such contexts, VR training could represent a more cost-effective and accessible alternative than large-scale real-life exercises.

# _2.2. Engineering, science, and technical education_

Applications of VR training in engineering, science, and technical education have been most common in the fields of aviation, design, mechanics, industrial safety, and robotics (Buiu, & Gansari, 2014; Wei, Dongsheng, & Chun, 2013). In these fields, VR training provides students similar to real-life environments and access to state-of-the-art technology and equipment without the need to make significant capital investments in laboratories.

When teaching engineering, science, and technical education, laboratory sessions constitute an essential part of the training. They provide hands-on experiences that allow students to learn the necessary skills required to manage, configure, troubleshoot, repair equipment, specialized instruments, and machinery. Laboratories enable students to practice and acquire skills before performing tasks in real professional situations. Nonetheless, many technical programs fall short in providing practical experience to students, as the set-up and functioning of laboratories require important capital investments in equipment, as well as in maintenance and updates. Also, the necessary equipment needed to perform hands-on labs is not always available or accessible, especially in developing countries and in rural areas.

A proposed solution to address the difficulties to set-up laboratories is to substitute them with virtual laboratories. A virtual lab is an interactive simulation of a real lab. Virtual labs are essentially synthetic environments with attributes that include interactivity and real-time feedback (Lampi, 2013). The purpose of virtual labs is to develop student proficiency in the execution of practical skills. Virtual labs have traditionally been used in fields that require a skills proficiency that guarantees learners’ safety before they can operate real equipment. Pilot training, military equipment training, and nuclear power plant training have a long-documented history in utilizing virtual labs (Lampi, 2013). Moreover, virtual labs offer additional advantages such as remote access for distance education, low cost, reliability, security, flexibility, and convenience to the student. The authenticity of the learning experience in a virtual lab depends on the extent to which 7
the simulation causes learners to engage in cognitive processes comparable to those provided by a real laboratory. VR has been used to improve the fidelity of virtual labs as 3D environments, allowing the possibility of recreating real environments and even developing psychomotor skills when using virtual labs (Stone, <mark>Watts, & Zhong,</mark> 2011).

# _2.3. VR training for general education_

VR training has also been used to impart general education courses in several areas, such as STEM education, astronomy, anatomy, nurse education, and the arts (Kavanagh et al., 2017). Given its pedagogic potential and increasing market availability, it is crucial to examine the effectiveness of emerging VR technologies to deliver content to students in general education settings. Some advocates for using VR technologies for general instruction claim that immersive learning using VR brings motivational benefits that can lead to improved student learning (Makransky, et al., 2019). This is so because VR simulations can replace or amplify real-world learning environments by allowing students to interact and manipulate objects and parameters, thus promoting constructivist learning. VR simulations can also enable students to observe otherwise unobservable phenomena and provide students a higher sense of physical, environmental, and social presence. Students work harder when they are more interested in the material, either intrinsically (individual interest) or as elicited by the situation (situational interest) (Parong, & Mayer, 2018).

Nonetheless, creating educational applications for VR could be a laborious and costly endeavor, so it is crucial to investigate whether these applications are useful for learning or not (Allcoat, & von Mühlenen, 2018). Unnecessary features introduced by VR simulations may hinder learning compared with traditional methods or compared to less sophisticated multimedia channels, such as videos and well-designed slideshows (Parong et al., 2018). Some authors claim that VR simulations may not adhere to the coherence principle of multimedia, which states that people learn better when extraneous words, sounds, and pictures are excluded from rather than included in the student learning environment. This occurs because VR simulations, especially those that are fully immersive, often add material and features (visual effects, sounds, detailed environments) which could divert attention from the important material. In other words, VR simulations may include content that is not relevant to the instructional goal. Given that learners have a limited amount of cognitive processing capacity, if VR simulations entail unnecessary 8
detail, learners may not engage adequately with the essential materials that trigger cognitive processing and learning (Mayer, 2009; Mayer, 2014).

Finally, other authors claim that the usefulness of VR for general education might also depend on the type of subject of learning. Indeed, VR simulations may not necessarily be equally suitable for all subject areas. For example, it might be less beneficial for learning to play a musical instrument that requires tactile feedback, such as arts education. Still, it may be particularly helpful for teaching subjects where it is important to visualize the learning materials in 3D (e.g., biology or geometry) (Allcoat, & von Mühlenen, 2018). Other authors argue that technologies themselves, such as VR, do not directly cause learning but can afford specific tasks that themselves may result in learning (Dalgarno, & Lee, 2010).

# **III. Data and Methodology**

A first step for conducting an informative meta-analysis is to gather relevant studies and accurately extract and report information from these primary studies (Uttl, White, & Gonzalez, 2017). Data were collected through a review of available studies assessing the impacts of VR training on learning and skills development. Studies included in the meta-analysis follow some predetermined criteria. First, they need to be published in a peer-reviewed journal or as a doctoral thesis, as a proxy for research quality. Second, to account for significant technological developments in access and quality of VR simulators (hardware and software), the sample only includes studies conducted within the last 15 years (2005-2020 period). Third, studies included in the meta-analysis assess the impact of VR training on student learning through value-added experiments or experimental evaluations that use randomized control trials (RTC).<sup>3</sup> Fourth, studies included in the sample assess skills development using objective and clearly measurable metrics, such as learning assessments or performance evaluations (pre- and post-test). The review of studies relied on web and academic databases, such as ACM Digital Library, IEEE Xplore, Web of Science, ERIC, and Scopus. Data obtained from these primary studies were compiled systematically, including the following information:

> 3 RCT experiments are defined as those where individuals are allocated at random (by chance alone) to receive one or several interventions. One of these interventions is the standard of comparison or control. The control may be a standard practice or no intervention at all. Value-added studies quantify changes in desired outcomes (for instance, skills development of student learning) by quantifying these outcomes before and after individuals benefit from the intervention.

9
- Year of implementation;

- Field of study, using three main categories: (i) Health and safety, (ii) Virtual laboratories for engineering, science, and technical education; and (iii) General Education

- Type of VR training used: (i) immersive; (ii) non-immersive

- Beneficiary grade level: (i) Basic Education (K to 12); (ii) Technical-Vocational Education and Training (TVET), (iii) Higher Education, (iv) On-the-job training<sup>4</sup>

- Number of individuals who participated in the study

- Type of evaluation conducted: (i) RCT; (ii) Value-added

- Type of skills assessed: (i) Cognitive Skills, understood as the acquired knowledge to understand and retain complex ideas, adapt effectively to the environment, learn from experience, and reason; (ii) Technical Skills, understood as the expertise and ability needed to perform a specific job, including the mastery of the materials, tools, or technologies, and time on task<sup>5</sup> ; and (iii) Socio-emotional skills, understood as the ability to navigate interpersonal and social situations effectively included leadership, teamwork, cooperation, self-control, self-confidence, self-efficacy, and grit

- A description of the instrument and scale used to assess student’s skills

- Evaluation results (e.g. results of pre and posttest and their statistical significance)

- VR exposure time, in hours.

The meta-analysis assesses three main outcomes of VR training courses: learning performance ( _L_ ), value-added ( _VA_ ), and learning efficiency ( _LE_ ). While not all papers report all three outcomes, papers included in the analysis report at least one of them. Learning performance is quantified as the average percentage gain in test-scores obtained after the training is completed (i.e. % difference in posttest scores), between students who receive the VR training, or treatment group ( _T_ ) and students who receive traditional training, or control group ( _C_ ). Learning value added is defined as the net gains accrued as a result of the training, measured by differences in posttest and pretest scores that assess similar competences. Most studies report this information for the treatment group only, but a few allow to assess differences in learning value added between

> 4 TVET includes technical basic education, technical higher education, and vocational training programs. Higher Education includes academic undergraduate and graduate programs.

> 5 Time on task refers to the time spent to successfully complete a task or procedure.

10
students in the treatment and control groups. Learning efficiency is measured as the % difference in inputs or time utilization (such as training time, time-on-task, materials used) between students exposed to VR training vs. students exposed to traditional training methods. These outcomes are defined as follows: Since not all test-scores in (1) and (2) reported in the studies use a similar metric, a first step to assure comparability of outcomes across studies is to conduct a monotonic transformation to normalize all test-scores ( _r_ ) between zero and one and into a comparable metric ( _s_ ), as follows: Where _min_ and _max_ are the minimum and maximum allowed values for the test-score of the original metric used.

Since each study ( _i_ ) included in the meta-analysis display important differences in sample size ( _N_ ), intervention exposure times in hours ( _E_ ), and type of skills assessed ( _j_ ) (cognitive, technical, and socio-emotional); we use weights in our estimations to take into account that larger sample size studies and higher intervention exposure are associated with less sampling error than studies with smaller size and less intervention exposure. As such, to assess average effects by skill type, we compute for each of the outcomes ( _O_ ) assessed, a sample weighted average and a sample weighted variance, as follows: Since the main objective of the study is to identify the extent to which VR contributed to learning and skills development, the instruments used by the primary studies to assess learning and 11
skills are of prominent importance. The assessment of technical skills often relies on direct observation of the trainee’s performance on a predetermined task or procedure. Expert practitioners conduct observations and evaluate student performance based on a series of predetermined metrics. Observations tend to be anonymous to prevent the observer’s bias. The protocols for the observation and the metrics to assess performance are often available and previously applied by the industry to certify professional skills, especially in the fields of medicine and engineering. The assessment of cognitive skills is generally measured using standardized tests, developed by professors and practitioners based on the curricula imparted in the training course. Finally, the assessment of socio-emotional skills is mainly conducted through students’ selfreported perceptions of self-efficacy and attitudes towards learning. Tables A2, A3 and A4 in the Annex provide a detailed description of the instruments used by different experiments to assess skills proficiency.

# _3.1. Descriptive statistics_

After conducting a thorough review of the literature, a total of 31 primary studies met the criteria specified above. Most studies (29) were conducted in OECD countries, notably in the United States, the United Kingdom, and Canada (18 of 31). While many studies attempt to assess the effects of VR on learning and skills development, not all conduct a credible evaluation of the impacts and, among those that do, many do not report complete information on the impacts of the VR training and their statistical significance. Nonetheless, the studies that met the criteria (31) include 92 different experiments that assess the effects of VR training on students’ skills development. Detailed information about each experiment is provided in Tables A1 to A12 in the Annex section.

Figures 1 and 2 present descriptive statistics of the 92 experiments included in the metaanalysis. A total of 78 experiments assess the impacts of VR training on learning outcomes using RCTs. Most experiments (79 of 92) assess immersive VR training. A total of 50 experiments were conducted in higher education settings, while 37 experiments studied the effects of VR training on cognitive skills and 29 studied the effect on technical skills. Only 13 experiments were conducted in basic education settings (k to 12). In terms of the educational field, the experiments were evenly distributed in health and safety (35) and virtual labs for engineering, science, and technical education (35). A total of 22 experiments focused on general education (Figure 1). While 12
categorizing experiments across education fields was straightforward for those experiments related to health and safety, there were some topics that overlapped between experiments in the fields of engineering, science, and technical education with those pertaining to general education. The determining factor to sort these studies in one of these two field, was the use or not of a virtual laboratory. If the training imparted aimed to emulate and/or substitute for a real laboratory, the experiment was included in the field of engineering, science, and technical education – independently of the subject (see Table A1).

**Figure 1: Descriptive statistics of the primary experiments in the meta-analysis.**

<!-- Start of picture text --> Total number of experiments (N=92)<br><!-- End of picture text --> <!-- Start of picture text --> Impacts assessed using RCTs 78<br>Immersive VR 79<br>Basic education (K to 12) 13<br>Grade Level<br>TVET / OJT 29<br>Higher education 50<br>Efficiency 12<br>Socio-emotional 14 Type of<br>skill<br>Technical 29<br>assessed<br>Cognitive 37<br>General education 22<br>Field of<br>Health and safety 35<br>Engineering, science, and TVET 35 study<br>0 10 20 30 40 50 60 70 80 90<br><!-- End of picture text --> _Source_ : Author’s elaboration Developing comparable and fair metrics constitutes a critical aspect to accurately assess the impacts of VR training on learning and skills development. As mentioned above, in order to provide a fair assessment of the available literature, the meta-analysis weights results based on two aspects, notably, intensity of the treatment (i.e. exposure time to VR training) and experiment sample size (Figure 2). These two variables display important variations in the primary studies included in the analysis. While half of the experiments expose students to more than one hour of VR training, 20 experiments report very short VR exposure (less than 15 minutes), while 22 report an exposure that surpasses 5 hours. Moreover, while most experiments included in the analysis are 13
medium size (benefiting between 21 and 100 students), some experiments have very limited sample sizes, of fewer than 20 beneficiaries (16 experiments in total), while others (9) include larger scale experiments that reach more than 100 beneficiaries.

**Figure 2: Exposure time and sample size of the primary studies in the meta-analysis**

Total number of experiments (N=92) <!-- Start of picture text --> 45<br>41<br>40<br>Number of experiments Number of experiments according<br>35 according to the experiment to the VR exposure time<br>sample size<br>30<br>26<br>25<br>25 22<br>20<br>20<br>16<br>15 12 13<br>9<br>10<br>5<br>0<br>0 to 20 21 to 50 51 to 100 > 100 0 to 15 16 to 30 31 to 59 1 to 5 > 5 hours<br>minutes minutes minutes hours<br><!-- End of picture text --> _Source_ : Author’s elaboration Figure 3 provides a list of all studies included in the meta-analysis, as well as information about their relative weight (i.e. sample size multiplied by exposure time, normalized to 100%), education field, and training topic. Results in the figure show studies in the field of engineering, science, and technical education tend to provide students with longer VR exposure and be conducted at a larger scale. As such, these studies are given a higher relative weight in the analysis.

Finally, studies included in the meta-analysis cover a diverse range of topics, from surgical education to welding. General education topics are also studied, such as from frog dissection and job interview training. In the field of health and safety, most studies included in the analysis pertained to surgical education (laparoscopic surgery, bone dissection, robotic cardiac surgery, robotic suturing, cataract surgery, and hip arthroplasty), although studies in other areas were also included, such as studies pertained to safety and risk prevention (2), one study on medical procedures in gynecology, and one study on nursing education.

14
**Figure 3: Studies in the meta-analysis according to their relative weight (in %)**

|**Stone et al.(2011)**||Welding|
|---|---|---|
|**Oser (2013)**||Genetics|
|**Yang & Heh (2007)**||Physics|
|**Tatli and Ayas (2013)**||Chemistry|
|**McLaurin & Stone (2012)**||Welding|
|Tschannen et al.(2012)||Nursing education|
|Smith (2015)||Job interviewing|
|Hwang & Hu (2013)||Geometry|
|**Lampi (2013)**||PC network configuration|
|**Finkelstein et al.(2005)**||Circuits design|
|Valdis et al.(2015)||Robotic surgery|
|**Tanyildizi & Orhan (2007)**||Synchronous motors|
|Kiely et al.(2015)||Robotic suturing|
|Farra et al.(2018)||Emergency evacuation|
|Kockro et al. (2015)||Anatomy of the heart|
|Logishetty (2018)||Hip surgery|
|Zhao et al.(2011)||Bone dissection|
|**Rupasinghe et al.(2011)**||Corrosion prevention|
|Webster et al. (2015)||Corrossion prevention|
|Skou-Thomsen et al.(2017)||Cataract surgery|
|Makransky et al.(2019)||Biology - mammalian proteins|
|Chao et al.(2015)||Gynecologic ultrasound|
|Larsen et al.(2009)||Laparoscopic surgery|
|Alhalabi (2016)||General Science|
|Allcoat et al. (forth)||Science - solar panels|
|Allcoat & von Mühlenen (2018)||Biology - plant cells|
|Akpan & Strayer (2010)||Frog dissection|
|Parong & Mayer (2018)||Biology - human cells|
|**Buttussi et al.(2018)**||Aviation safety|
|Zaveri et al.(2016)||Pediatric sedation|
|Vincent et al. (2008)||Emergency triage|
|-15.0<br>-10.0<br>-5.0|0.0<br>5.0<br>10.0<br>15.0|20.0<br>25.0<br>30.0<br>35.0|

_Note_ : studies in bold are in the field of engineering, science, and technical education and underlined studies are in the field of health and safely. Other studies are in the field of general education.

Most studies included in the field of health and safety focused on assessing the impact of VR training on students’ technical skills. In the field of engineering, science, and technical education, all studies are focused on assessing the effectiveness of virtual labs (e.g. for welding, physics, chemistry) to develop students’ practical and cognitive skills. Most studies pertaining 15
general education were in STEM related fields and focused on assessing the effects of VR training on students’ cognitive skills vis a vis otherwise similar content imparted through more traditional mechanisms, such as slideshows and videos (see Table A5 in the annex).

# **IV. Main results**

This section presents the main results of the meta-analysis. Table 1 presents the observed average effects of VR training on student learning performance, as proxied by the % difference in posttests between students exposed to VR training vs. students exposed to traditional training. The effects are calculated based on information of a total of 42 available experiments, and account for the experiments’ sample size and VR exposure time. To foster comparability, we report the effects of VR training on students' technical skills per one hour of training and the effects of VR training on students' cognitive skills per quarter-hour of training. This reporting choice reflects that many experiments that aim to improve students' cognitive skills are of minimal duration (in some cases, they entail an exposure of fewer than five minutes), which would be difficult and misleading to extrapolate more extendedly.

**Table 1. Average effects of VR training on student learning** **<u>performance</u>**

||% Difference inpo<br>|sttests results(VR vs trad<br>|itional training)<br>|
|---|---|---|---|
||**Technical skills**<br>(effect per hour of training)|**Cognitive skills**<br>(effect per 1⁄4 hour of<br>training)|**Socio-emotional skills**<br>(effect per training<br>course)|
|Average impact|2.95|2.51|29.8|
|Standard Deviation|10.1|8.07|-|
|Number of experiments|15.0|24.0|3.0|
|Registering a positive effect|7.0|17.0|2.0|
|Registering negative effect|0.0|1.0|0.0|
|Registering no effect|19 8.0|6.0|1.0|
|Average VR exposure time (hours)|8.8|1.93|22.0|
|Average experiment size|57.3|89.6|80.0|

Source: Author’s elaboration.

A total of (15) 26 out of the 42 experiments included in the meta-analysis show a (neutral) positive effect of VR training on learning performance. Only one experiment indicates that VR training is associated with lower learning performance when compared to traditional training. Findings emanating from these experiments indicate that, on average VR training is indeed more effective than traditional training as a mechanism to develop technical, practical, and socioemotional skills. Results are particularly promising in fields related to health and safety, 16
engineering, and technical education. Results reveal that for each additional hour (1⁄4 hour) of training, students exposed to VR training score 3 percent higher in technical (cognitive) learning assessments, when compared to students exposed to the same curricular content delivered through traditional training methods. Results also indicate that students who complete a VR training course report, on average, 30 percent higher scores in socio-emotional skills assessments.

Table 2 presents the observed average effects of VR training on learning value added, as proxied by the % difference between posttests and pretests of students exposed to VR training. These results are indicative of the capacity of VR training to positively improve students’ skills. The effects are calculated based on information of a total of 27 available experiments, and account for the experiments’ sample size and VR exposure time. Results indicate that, on average, VR training contributes to gains in technical (practical) skills that average 17 (8) percent per hour (1⁄4 hour) of training. Moreover, students who completed a VR training course, report on average, gains averaging 20.5 percent in self-reported socio-emotional skills.

**Table 2. Average effects of VR training on student learning value added**

||% Difference b<br>|etweenposttests andprete<br>|sts(VR only) <br>|
|---|---|---|---|
||**Technical skills**<br>(effect per hour of<br>training)|**Cognitive skills**<br>effect per 1⁄4 hour of<br>training)|**Socio-emotional skills**<br>(effect per training<br>course)|
|Average impact|17.3|8.0|20.5|
|Standard Deviation|20.9|31.3|26.1|
|Number of experiments|10.0|7.0|10.0|
|Registering a positive effect|9.0|7.0|7.0|
|Registering negative effect|0.0|0.0|0.0|
|Registering no effect|1.0|0.0|3.0|
|Average VR exposure time in hours|3.5|2.42|0.24|
|Average experiment size|23.4|89.0|39.4|

Source: Author’s elaboration.

Average results hide important patterns that arise when assessing the effects of VR training by education type of skill and education field. Results in the next subsections are organized by the type of skill assessed by the available experiments (technical, cognitive, and socio-emotional) and, within each skill, if there are any observed patterns, there is brief discussion of the impacts of the training by education field.

- _4.1. Impacts of VR training on technical skills_

Technical skills are often measured using ability tests, whereby students are required to perform specific task and are graded based on how their performance fares against predetermined 17
standards. The meta-analysis includes a total of 14 experiments that assess student learning of technical skills, proxied as the % difference observed in posttests per hour of training received between students who participated in VR training, or treatment group, and students who participated in traditional training, or control group. The results of these experiments are presented in Figure 4. Results in the chart provide information about the authors who conducted the experiment and the field and topics of the training.

**Figure 4:  % Difference in posttests between students exposed to VR vs non-VR training per hour of training [Technical Skills]**

<!-- Start of picture text --> Chao et al. (2015) Gynecologic ultrasound<br>Larsen et al. (2009) Laparoscopic surgery<br>Logishetty (2018) Hip arthroplasty surgery<br>Farra et al. (2018) Emergency evacuation<br>Valdis et al. (2015) Robotic heart surgery<br>Tschann et al. (2012) Nursing procedures<br>Osner (2013) Genetics<br>McLaurin & Stone (2012) Vertical groove weld (3G)<br>McLaurin & Stone (2012) Vertical filet weld (3F)<br>McLaurin & Stone (2012) Flat groove weld (1G)<br>McLaurin & Stone (2012) Horizontal filet weld (2F)<br>Lampi (2013) PC Network configuration<br>Lampi (2013) PC Network troubleshooting<br>Zaveri et al. (2016) Pediatric sedation<br>-60.0 -40.0 -20.0 0.0 20.0 40.0 60.0 80.0 100.0 120.0<br><!-- End of picture text --> _Note_ : studies in bold are in the field of engineering, science, and technical education. Underlined studies are in the field of health and safely. Other studies are in the field of general education. The bold vertical dotted line represents the average observed effects. Gray vertical dotted lines represent the standard deviation of the observed effects.

Not surprisingly, experiments that assess technical skills pertain mainly to the fields of health and safety (notably surgery performance) and engineering, science, and technical education (i.e. they relate to virtual labs for training students to perform technical tasks). Results indicate that for experiments in the field of health and safety, VR training is generally associated with posttest scores in trainees’ ability assessments that are 20 to 60 percent higher per every extra hour of instruction than those observed in students exposed to traditional training. However, this is not the case for studies in engineering, science, and technical education where students who are exposed to VR laboratories fare just as well as students who access traditional laboratories.

18
At the aggregate level, the average difference in post-test scores between students in the treatment and control groups is 3 percent per additional hour of training, with a standard deviation of 10 percent. This average effect accounts for the fact that experiments in the field of health and safety, where impacts tend to be positive and large, are generally limited in terms of sample size and exposure time to VR training (see Table A1). The fact that students exposed to VR training in the fields of engineering, science, and technical education do not display higher posttest scores than students exposed to traditional training does not indicate that VR training in these fields is not effective. What the result suggests is that VR training is as effective as traditional training methods. As will be discussed below, when the effects of VR training on learning efficiency are assessed, this result is quite relevant because VR training for technical education could be more cost-effective that traditional training in cases where it is cheaper and safer to use simulators than traditional laboratories that are expensive to set up, maintain, and update.

The meta-analysis also includes 10 experiments, most of them in the field of health and safety, that assess the effects of VR training on student learning value added, as proxied by the % difference in post-test minus pre-tests for students who participate for VR training (Figure 5).

**Figure 5:  Value added (% difference between posttests and pretests) per hour of training for students in the Treatment group [Technical Skills]**

<!-- Start of picture text --> Vincent et al. (2008) Accuracy in emergency triage<br>Vincent et al. (2008) Emergency triage<br>Kiely et al.(2015) Robotic suturing - Satisfactory knots<br>Skou-Thomsen et al.(2017) Cataract surgery - intermediate MDs<br>Skou-Thomsen et al.(2017) Cataract surgery - Novice MDs<br>Kiely et al.(2015) Robotic suturing - Total knots<br>Kiely et al.(2015) Robotic suturing - GEARS test<br>Kiely et al.(2015) Robotic suturing - GOALS + test<br>Smith (2015) Job Interview performance<br>Skou-Thomsen et al.(2017) Cataract surgery - Experienced MDs<br>-80.0 -30.0 20.0 70.0 120.0 170.0 220.0<br><!-- End of picture text --> _Note_ : studies in bold are in the field of engineering, science, and technical education. Underlined studies are in the field of health and safely. Other studies are in the field of general education. The bold vertical dotted line represents the average observed effects. Gray vertical dotted lines represent the standard deviation of the observed effects.

19
Results systematically confirm the effectiveness of VR training to develop technical skills in the field of health and safety. Available experiments indicate that one hour of VR training increases students net learning outcomes by 17 percent on average (with a standard deviation of 18 percent). Some results even indicate that VR training can be conducive to double student learning gains, especially in topics such as emergency response, where it is otherwise hard to provide students with access to real emergency situations. Results also show that the effects of VR training on value added can vary depending on the seniority of the trainees. For instance, results in Skou-Thomsen et al. (2017) indicate that VR training is conducive to higher learning gains for novice and intermediate surgeons. However, such training shows no statistically significant learning gains for more experienced surgeons.

Finally, the meta-analysis results include 4 experiments that assess differences learning value added between students exposed to VR training vs. students exposed to traditional training. In the field of robotics surgery, these experiments suggest that students exposed to VR training display 28 percent higher learning gains per hour of training on average than students exposed to traditional training (Figure 6).

**Figure 6:  % Difference in value added per hour of training (VR vs. Non-VR) [Technical Skills]**

<!-- Start of picture text --> Kiely et al.(2015) Robotic suturing - GOALS + test<br>Kiely et al.(2015) Robotic suturing - GEARS test<br>Kiely et al.(2015) Robotic suturing - Satisfactory knots<br>Kiely et al.(2015) Robotic suturing - Total knots<br>-50.0 -30.0 -10.0 10.0 30.0 50.0 70.0 90.0 110.0<br><!-- End of picture text --> _Note_ : studies in bold are in the field of engineering, science, and technical education. Underlined studies are in the field of health and safely. Other studies are in the field of general education. The bold vertical dotted line represents the average observed effects. Gray vertical dotted lines represent the standard deviation of the observed effects.

# _4.2. Impacts of VR training on cognitive skills_

Cognitive skills are generally measured using standardized tests, whereby students are required to answer a set of questions based on the curricula imparted. The meta-analysis includes a total of 24 experiments that assess the impact of VR training on students’ cognitive skills, proxied as the % difference observed in posttests per every 1⁄4 hour of training received between students 20
who participated in VR training, or treatment group, and students who participated in traditional training, or control group. Contrary to VR training experiments aimed to develop technical skills, available experiments seeking to improve cognitive skills tend to have shorter exposure times to the technology (See Table A1 in the annex). As a result, as mentioned before, while we assess the impacts of VR training per hour of instruction when courses aim to develop technical skills, when developing cognitive skills, it seems more adequate to assess the effects per 1⁄4 hour of instruction.

Once factoring out the relative weight of each experiment, results of available experiments reveal that the VR training is associated with a 3 percent higher learning than traditional training per every 1⁄4 hour of instruction, with a standard deviation of 8 percent. Most experiments assessing cognitive skills pertain to the fields of general education (in topics related to STEM), as well as virtual laboratories. The results of these experiments are presented in Figure 7. While results from available experiments display some dispersion, most experiments (18 out of 24) indicate that VR training has positive impacts in student learning. In some experiments (8 out of 24), these impacts are quite high and show that students exposed to VR training have results in cognitive assessments that are 20 to 80 percent higher than those of students exposed to traditional training methods per every additional 1⁄4 hour of instruction. Nonetheless, a total of 6 experiments show no significant effects of VR training on learning vis a vis traditional instruction. Only one study, in the topic of biology, finds that the effect of VR training on learning is negative (Parong & Mayer, 2018).

When assessing by field of study, the results presented in Figure 7 indicate that that students trained in virtual laboratories generally display higher cognitive learning (proxied by test scores) than students exposed to traditional laboratories. The intuition behind this result is that virtual labs allow for illimited repetition of experiments, are self-paced, and generally provide direct feedback to students. Such features are particularly useful for student learning in topics that require understanding of abstract concepts, such as physics (Yang, & Heh, 2007).

Results do not reveal a clear pattern when it comes to general education. Most related experiments aim to assess the impacts VR training has on learning compared to other more traditional instruction methods such as lecture and/or classes that use other type multimedia aid such as a video, a textbook, or a slideshow (Table A5 in the annex provides more details). Some studies indicate that students who receive VR training perform better in cognitive assessments than 21
students exposed to a traditional lecture or videos (Alhalabi, 2016; Allcoat, & von Mühlenen, 2018).

**Figure 7:  % Difference in posttests between students exposed to VR vs non-VR training [Cognitive Skills]**

|Allcoat & von Mühlenen (2018)||Plant cells (VR vs Video)|
|---|---|---|
|Alhalabi (2016)||Famous inventors|
|Allcoat & von Mühlenen (2018)||Biology concepts (VR vs Video)|
|Alhalabi (2016)||Networking|
|Allcoat & von Mühlenen (2018)||Biology concepts (VR vs Text)|
|Alhalabi (2016)||Transportation|
|Akpan & Strayer (2010)||Anatomy|
|Alhalabi (2016)||Anatomy|
|**Rupasinghe et al.(2011)**||Use of boroscope (oral test)|
|**Finkelstein et al.(2005)**||Circuits (essay)|
|**Finkelstein et al.(2005)**||Circuits (test)|
|**Tatli and Ayas (2013)**||Chemistry|
|**Rupasinghe et al.(2011)**||Use of Eddy labs (oral)|
|**Tanyildizi & Orhan (2007)**||Synchronous motors|
|Farra et al. (2018)||Emergency Preparedness|
|**Hwang & Hu (2013)**||Science|
|**Yang & Heh (2007)**||Synchronous motors|
|**Yang & Heh (2007)**||Synchronous motors|
|Kockro et al. (2015)||Anatomy|
|Allcoat & von Mühlenen (2018)||Plant cells (VR vs Text)|
|**Rupasinghe et al.(2011)**||Use of Eddy labs (written)|
|**Rupasinghe et al.(2011)**||Use of boroscope (written)|
|**Tatli and Ayas (2013)**||Chemistry|
|Parong & Mayer (2018)||Biology|
|-80.0<br>-60.0<br>-40.0<br>-20.0|0.0<br>20.0<br>40.0|60.0<br>80.0<br>100.0<br>120.0|

_Note_ : studies in bold are in the field of engineering, science, and technical education. Underlined studies are in the field of health and safely. Other studies are in the field of general education. The bold vertical dotted line represents the average observed effects. Gray vertical dotted lines represent the standard deviation of the observed effects.

Other studies (Allcoat, & von Mühlenen, 2018; Hwang, & Hu, 2013) indicate that students who receive VR training perform just as well as students that use textbooks to complement traditional lectures. Allcoat and von Mühlenen (2018) and Hwang and Hu (2013) find that students exposed to VR training memorize better the parts of a plant cell and display better skills to calculate area and volume of 3D figures than students exposed to similar content but whose lessons were imparted using more traditional learning methods, such as videos and lectures. Parong & Mayer (2018) find that students exposed to slideshows perform better than students exposed to VR training on subjects such as biology. Finally, Kockro et al., 2015 found no significant difference 22
in students’ knowledge of the human ventricular system’s anatomy after providing them with a training course using VR content versus a traditional slideshow.

The meta-analysis also includes 7 experiments that assess the effects of VR training on learning value added, as proxied by the % different in post-test minus pre-tests for students who participate for VR training (Figure 8). Results in all available experiments are indicative that VR training is conducive to positive learning gains for the development of cognitive skills. Such gains in some experiments can be as high as 2.5 times the baseline cognitive knowledge, as proxied by standardized tests. Nonetheless, experiments with such acute increases in learning gains are often very small in size and limited in terms in VR exposure (i.e. VR training of less than 5 minutes) (Allcoat, 2021; Buttussi, <mark>& Chittaro</mark> , 2018). Once accounting for experiment size and exposure time, results indicate that VR training contributes to student learning gains averaging 8 percent per 1⁄4 hour of training, with a rather large standard deviation of 31 percent.

**Figure 8:  Learning value added (% difference between posttests and pretests) per 1⁄4 hour of training for students in the Treatment group [Cognitive Skills]**

<!-- Start of picture text --> Allcoat et al. (forth) Science - solar panels<br>Buttussi et al.(2018) Aviation safety - VR wide view<br>Buttussi et al.(2018) Aviation safety - VR narrow view<br>Akpan & Strayer (2010) Science - anatomy<br>Webster et al. (2015) Science - corrossion prevention<br>Tatli and Ayas (2013) Chemistry - lab instruments<br>Tatli and Ayas (2013) Chemistry - chemical changes<br>-120.0 -70.0 -20.0 30.0 80.0 130.0 180.0 230.0 280.0 330.0 380.0<br><!-- End of picture text --> _Note_ : studies in bold are in the field of engineering, science, and technical education. Underlined studies are in the field of health and safely. Other studies are in the field of general education. The bold vertical dotted line represents the average observed effects. Gray vertical dotted lines represent the standard deviation of the observed effects.

Finally, results in the meta-analysis include 5 experiments that assess differences in learning value added between students exposed to VR training vs. students exposed to traditional training. Results are mixed. Some experiments indicate that students exposed to VR training experiment lower learning gains than students exposed to traditional training (Zaveri et al., 2016; Makrasky et al., 2019). Other experiments indicate that VR training contributes to similar or higher 23
learning gains than traditional training (Allcoat, 2021; Farra et al., 2018; Tanyildizi & Orhan, 2007). Nonetheless, when accounting for experiment size and exposure time, the average observed differences in learning gains per 1⁄4 hour of training between students exposed to VR vis a vis those exposed to traditional learning is close to zero (Figure 9), indicating that VR training is, on average, as effective a mechanism to enhance student’s cognitive skills when compared to traditional training. Nonetheless, due to the limited number of experiments that assess learning value added across VR and non-VR recipients, results need to be used with care and may not allow to adequately generalize.

**Figure 9:  % difference in value added per 1⁄4 hour of training (VR vs. Non-VR) [Cognitive Skills]**

<!-- Start of picture text --> Tanyildizi & Orhan (2007) Synchronous motors<br>Farra et al.(2018) Emergency evacuation<br>Allcoat et al. (forth) Science - solar panels<br>Makransky et al.(2019) Biology - mammalian proteins<br>Zaveri et al. (2016) Pediatric sedation<br>-50.0 -40.0 -30.0 -20.0 -10.0 0.0 10.0 20.0 30.0 40.0<br><!-- End of picture text --> _Note_ : studies in bold are in the field of engineering, science, and technical education. Underlined studies are in the field of health and safely. Other studies are in the field of general education. The bold vertical dotted line represents the average observed effects. Gray vertical dotted lines represent the standard deviation of the observed effects.

# _4.3. Impacts of VR training on socio-emotional skills_

The assessment of socio-emotional skills is often conducted through students’ self-reported perceptions of self-efficacy and attitudes towards learning. Such perceptions are often quantified using a Likert scale type of questionnaire that students complete before and/or after they complete their training (see Tables A2 to A4 in the annex). Such perceptions are often indicative of the main channels that explain why VR training can be more effective and more conducive to learning than traditional training. The meta-analysis includes a total of 13 experiments that assess the effects of VR training on the effects of students’ socio-emotional skills.

24
||**Figure 10: % Dif**<br>**Stone et al.(2011)**|**ference in posttests per course (VR vs**<br>**[Socio-emotional Skills]**|**non-VR**|**training)**<br>Collaborative learning|
|---|---|---|---|---|
||**Stone et al.(2011)**|||Open communication|
||**Stone et al.(2011)**|||Improvement seeking|
|-60.0|-40.0<br>-20.0|0.0<br>20.0<br>40.0<br>60.0|80.0|100.0<br>120.0|

_Note_ : studies in bold are in the field of engineering, science, and technical education. Underlined studies are in the field of health and safely. Other studies are in the field of general education. The bold vertical dotted line represents the average observed effects. Gray vertical dotted lines represent the standard deviation of the observed effects.

Results from Stone et al. (2011) indicate that students who receive welding training in virtual labs report higher self-reported peer communication and collaborative learning in post-test than students who receive traditional training. This may occur because virtual laboratories can allow for digital peer interactions, which can be repeated multiple times, and allow for real-time / automated feedback to students. However, the authors do not find significant differences in student self-reported improvement seeking of students who received their training in a virtual lab (Figure 10). The remaining 10 experiments assess the effects of VR training on student learning selfreported skills value added, as proxied by the % difference in post-test minus pre-tests for students who participate for VR training (Figure 11). Available results indicate that students who complete VR training tend to report 20% higher levels of confidence and self-efficacy towards learning after they complete their courses. For instance, results from Vincent et al. (2008) indicate that after being exposed to VR training, students report higher levels of confidence in several dimensions (e.g. being effective, making decisions, and using resources) when addressing a health emergency.

Interestingly, results for Buttussi et al. (2018) indicate that such self-reported increases in confidence can vary using different types of VR technology. In particular, the authors find that students exposed to narrow view VR technology (which allows them to focus more on features of the simulation) tend to report higher increases in self-reported self-efficacy addressing a plane emergency than students exposed to wide VR technology (which may expose students to more visual distractions).

25
**Figure 11:  Learning value added (% difference between posttests and pretests) per completed VR training course [Socio-emotional Skills]**

<!-- Start of picture text --> Buttussi et al.(2018) Self-efficacy assessing an emergency<br>(VR narrow view)<br>Confidence on ability to prioritize<br>Vincent et al. (2008) resources<br>Self-efficacy addressing an emergency<br>Buttussi et al.(2018)<br>(VR Wide view)<br>Vincent et al. (2008) Confidence on ability to identify risks<br>Confidence on ability to prioritize<br>Vincent et al. (2008) treatment<br>Confidence that others will consider<br>Vincent et al. (2008) me effective<br>Vincent et al. (2008) Confidence on being effective<br>Akpan & Strayer (2010) Attitude towards use of PC<br>Akpan & Strayer (2010) Attitude towards science<br>Akpan & Strayer (2010) Attitude towards frog dissection<br>-30.0 -10.0 10.0 30.0 50.0 70.0 90.0 110.0<br><!-- End of picture text --> _Note_ : studies in bold are in the field of engineering, science, and technical education. Underlined studies are in the field of health and safely. Other studies are in the field of general education. The bold vertical dotted line represents the average observed effects. Gray vertical dotted lines represent the standard deviation of the observed effects.

Finally, results from Akpan and Strayer (2010), which use VR training to simulate a frog dissection, do not find a significant effect of the use of VR training in students change in attitude towards learning anatomy and/or using PC assisted instruction.

# _4.4. Impacts of VR training on learning efficiency_

In the context of this study, learning efficiency is defined as any savings in the form of inputs, time, or performance errors that VR training could contribute to. One of the promises of using simulators vis-a-vis traditional instruction is their potential to save training costs and minimize the risks and errors faced when novice students intend to master some skills they will use in real life. Ideally, VR training should contribute to more efficient use of inputs, more expedited completion of tasks (or time-in-task), and fewer performance errors. The meta-analysis includes a total of 12 experiments that study the effects of VR training on the utilization of inputs (such as materials and time to complete a task) and performance errors. A total of 11 out of 12 experiments included in the meta-analysis find that VR training is associated with higher learning efficiency levels when compared to traditional training. In fact, experiment results indicate that, on average, students who are exposed to VR training are, on average, up to 30 percent more efficient (using inputs, time, and/or avoiding performance errors) than students exposed to traditional training per additional hour of instruction. Results from available experiments indicate 26
that VR training can help welding students to be more efficient using materials, such as plates and electrodes (Stone et al. 2011) and can expedite the time students take to perform surgical procedures such as laparoscopy surgery (Larsen et al. 2009), robotic heart surgery (Valdis et al., 2015) and hip arthroplasty (Logishetty, Rudran, & Cobb, 2018). Results from Logishetty, Rudran, & Cobb (2018) also indicate that students who are exposed to VR training are less likely to make mistakes when performing real surgery procedures (see Table A12 in the annex). Of course, due to the limited number of experiments used to draw these conclusions, these results need to be interpreted with care and should not be generalized. Nonetheless, these findings are indicative of the potential of VR simulators to be not only an effective, but also an efficient learning mechanism, especially in the fields of health and safety and technical education.

**Figure 12:  % decrease in inputs / performance errors per 1 hour of VR training compared to traditional training [Learning Efficiency]**

<!-- Start of picture text --> Lampi (2013) Configuration time - min<br>Lampi (2013) Troubleshooting time - min<br>Stone et al. (2011) Number of groove plates used<br>Finkelstein et al. (2005) Time on task - min<br>Stone et al. (2011) Number of plates used<br>Stone et al. (2011) Pounds of electrodes used<br>Valdis et al. (2015) Time on task - min<br>Logishetty (2018) Operation time - min<br>Larsen et al. (2009) Time on task - min<br>Valdis et al. (2015) Time on task - min<br>Logishetty (2018) Anteversion error - degrees<br>Logishetty (2018) Inclination error - degrees<br>-120.0 -100.0 -80.0 -60.0 -40.0 -20.0 0.0 20.0 40.0 60.0<br><!-- End of picture text --> _Note_ : studies in bold are in the field of engineering, science, and technical education. Underlined studies are in the field of health and safely. Other studies are in the field of general education. The bold vertical dotted line represents the average observed effects. Gray vertical dotted lines represent the standard deviation of the observed effects.

# **V. Conclusions**

The development of students’ skills remains a challenge for education systems worldwide. To address this challenge, educators are beginning to explore the possibility of using information technology to create learning experiences that would otherwise not be accessible to students. Simulations that rely on VR technology can provide students access to learning environments that 27
would otherwise be very difficult, expensive, or impossible to access. VR simulations can provide students practical training opportunities without pressure, danger, and allowing repeated practice. Such opportunities have the potential of accelerating students’ learning curve in a simulated environment, reproducing real-life conditions and situations without time or space limitations, and with much fewer risks.

This study constitutes an attempt to assess the effects of VR instruction holistically, as a mechanism to develop students’ skills. Given its pedagogic potential and its increasing market availability, it is crucial to examine the effectiveness of emerging VR technologies for pedagogical instruction. Creating educational applications for VR could be a laborious and costly endeavor, so it is essential to investigate whether these applications are useful for learning or not and, to the extent possible, to assess their cost-effectiveness. Results capitalize from a thorough review of 31 primary studies and over 90 experiments that intend to assess the effects of VR instruction on student learning. Our findings reveal that VR instruction is, on average, more effective than traditional training as a mechanism to develop students’ skills. Results indicate that for each additional hour (1⁄4 hour) of training, students exposed to VR training score 3 percent higher in technical (cognitive) learning assessments, when compared to students exposed to the same curricular content delivered through traditional training methods. Results also indicate that students exposed to VR instruction report, on average, 30 percent higher scores in socio-emotional skills assessments after completing their training.

Results are particularly promising in fields related to health and safety, engineering, and technical education. Results from available experiments confirm systematically that VR instruction yields positive results as a mechanism to train surgeons and medical personnel. It offers trainees the opportunity to practice medical procedures safely and at a comparatively low cost. Available experiments confirm VR simulators' effectiveness to improve surgeons’ proficiency to perform procedures such as laparoscopic surgery, robotic surgery, eye surgery, transvaginal examinations, infant sedation, and bone surgery, to name a few. Some results even indicate that VR training can be conducive to much higher student learning gains, especially in topics such as emergency response, where it is otherwise hard to provide students access to real emergencies.

VR training can provide students similar-to real-life laboratories and equipment without making significant capital investments. Available experiments show that virtual laboratories can 28
be as effective as real laboratories to develop students’ skills, but they can be a more efficient, safe, and cost-effective mechanism of instruction. Our findings indicate that students exposed to VR training are up to 30 percent more efficient using inputs, time, and/or avoiding performance errors than students exposed to traditional training, per additional hour of instruction. The intuition behind this result is that virtual labs allow for illimited repetition of experiments, are self-paced, and generally provide direct feedback to students.

Nonetheless, results do not reveal a clear pattern when it comes to the use of VR instruction for general education. Some studies indicate that students who receive VR training perform better in cognitive assessments than students exposed to a traditional lecture or videos. Other studies indicate that students exposed to other less expensive multimedia platforms, such as slideshows and videos, learn more than students exposed to VR training. As such, VR instruction may not be adequate as a mechanism for instruction in all educational fields. Indeed, VR training may provide students too much information, which may deviate their attention from the aspects of the curricula that matter most.

VR training can also help students develop their socio-emotional skills. Simulations can develop and promote collaborative features that enable peer interaction, active learning, and performance feedback. In turn, these features can promote student motivation and presence, which are two channels that can positively influence student learning. Available experiments show that students exposed to VR training report higher peer communication and collaborative learning in standardized assessments, than students who receive traditional training. Available experiments also show that students who complete VR training report higher confidence and self-efficacy towards learning after completing their courses.

It will be essential to continue to assess the cost-effectiveness of VR training, which is something beyond the scope of this study. While VR training's cost-effectiveness is likely to vary depending on many parameters such as course duration, cost of the actual equipment that VR intends to simulate, educational field, risks of making mistakes in a non-simulated environment, and type of technology used, it is not always assured. Indeed, this type of instruction could be costeffective only if it provides savings or reduces potentially expensive risks compared to other alternative multimedia or traditional laboratories. Imparting VR courses entails software development and equipment, maintenance, support, and updates, which require sustained 29
investments. To date, not many studies assessing the effects of VR training have focused on conducting a cost-benefit or cost-effectiveness analysis of VR instruction compared to traditional training methods. Having more such information will be crucial to assess the scalability potential of VR training across education systems.

Finally, this study's results primarily draw their conclusions based on experiments in developed countries. As such, these results may not necessarily hold in all educational settings because several factors necessary for VR training to succeed (e.g., connectivity, availability of equipment and IT support, students' and teachers' dominium of necessary digital skills, among others) are not necessarily assured in many education institutions in developing countries.

In summary, this study finds that VR training tends to be an effective mechanism of instruction to develop students’ technical, practical, and socio-emotional skills. Nonetheless, these results cannot be generalized, and it is important to continue to assess the pros and cons of using VR for pedagogical instruction for different subjects as well as its cost-effectiveness and scalability.

30