Conversational Tutoring in VR Training: The Role of Game Context and State Variables (2025)

Abstract: Virtual Reality (VR) training provides safe, cost-effective engagement with lifelike scenarios but lacks intuitive communication between users and the virtual environment. This study investigates the use of Large Language Models (LLMs) as conversational tutors in VR health and safety training, examining the impact of game context and state variables on LLM-generated answers in zero- and few-shot settings. Results demonstrate that incorporating both game context and state information significantly improves answer accuracy, with human evaluations showing gains of up to 0.26 points in zero-shot and 0.18 points in few-shot settings on a 0-1 scale.

## introduction vr is a powerful tool for fields such as healthcare and emergency response training, offering handson learning without real-world risks. however, current systems rely on joystick inputs, static messages, or pre-programmed responses, limiting engagement and personalized feedback essential for skill development. llms offer a promising solution to these interaction barriers by enabling humanlike dialogue and more natural, context-aware interactions. despite their potential, their role as conversational tutors in vr training is largely unexplored. this work presents the first use of llms as virtual tutors in emergency response vr training, addressing interaction gaps with dynamic, contextaware communication. by integrating game context and state variables, it enhances llm response accuracy and relevance, achieving significant quality improvements. contributions include advancing conversational ai in vr training and demonstrating the importance of contextual information for llm performance, paving the way for more interactive and effective training in critical scenarios like emergency response and health and safety. the paper is structured as follows: section 2 reviews related work and highlights gaps addressed in this study. section 3 introduces the use case, and section 4 details the proposed approach. section 5 outlines the experimental setup, followed by results and analysis in section 6. finally, section 7 concludes with key findings and future directions. ## related work llms as chatbots llms derive from research in language modeling, originally statistical n-gram models @xcite , passing to neural lms @xcite which later incorporate the attention mechanism @xcite and finally today's transformer architecture @xcite . their success lies in pre-training on vast amounts of data, where they develop a nuanced ability in natural language and retrieving real-world facts, @xcite and instruction-tuning @xcite , where they learn to follow instructions to engage with humans as chatbots. although they generate fluent text, llms need further training to be used in specific scenarios. for instance, transfer learning consists of fine-tuning a model on annotated in-domain data. as this annotated data is often limited, one can instead enrich input prompts with relevant context via in-context learning @xcite . in this work, we leverage this technique to improve the llm's ability to utilize contextual information effectively. ## use case the addressed use case involves immersive vr training for fire extinguishing. this allows users to practice techniques safely, cost-effectively, and sustainably while improving skill retention. the game guides the user through a sequence of 7 procedural steps, ranging from checking the fire extinguisher's pressure gauge to performing a test shot, approaching and attacking the fire, and, finally, stepping back to observe the results. the game context includes detailed descriptions for each step, emphasizing their importance and providing additional insights. it also outlines key simulation errors caused by extinguisher and fire type incompatibility, along with navigation aids. the game also incorporates 19 state variables that represent environmental factors, such as extinguisher type, fire class, and user proximity, along with user actions like checking the pressure gauge, performing a test shot, and attacking the fire, all of which evolve as the game progresses. an excerpt of the game context and the state variables is provided in appendix a. ## proposed approach as shown in figure 1 , the proposed approach models the game context and state variables that define and execute the vr training scenario, using the llm as an interactive conversational tutor. the llm prompt incorporates the following key information: the llm can process this information in a zeroshot manner but may benefit from few-shot examples to improve accuracy. when the conversational tutor intervenes, the above information is passed to the llm along with the user's request, allowing it to guide the user through the vr exercise. to adapt the system to a different vr training scenario, only the game context and state variables would need updating. ## experiments to evaluate the impact of game context and state variables in the proposed approach, we have conducted an ablation study using three distinct prompt configurations with varying amounts of information across five different open-source llama family llm models. the evaluation has been performed under both zero-shot and few-shot setups, with performance assessed through automatic metrics and human evaluation. ## results for the remaining models, g-eval is the most consistent metric across model sizes and versions. figure 2 shows zero-shot g-eval results for all figure 3: zero-shot vs. few-shot g-eval results using the gc + sv prompt models and prompts. it is clear that the gc + sv prompt consistently outperforms the other configurations across all models. furthermore, the g-eval metric shows a clear upward trend across all models as the prompts progress from the vanilla prompt to the gc prompt and finally to the gc + sv prompt, highlighting the positive impact of incorporating more information into the prompt on performance. focusing on the gc + sv prompt, figure 3 reveals that few-shot prompting enhances perfor-mance for larger models but offers no benefit for smaller models. this disparity likely stems from the complexity of the few-shot examples, which include game state variables represented in json format for each case. accurately interpreting this detailed information appears to be a capability that only the larger models can effectively manage. finally, the human evaluation results in ## conclusions and future work this paper explores using llms as conversational tutors in vr health and safety training, leveraging game context and state variables as key contextual information. experiments show the best results when combining these contextual elements in few-shot settings with large models. however, further improvements are necessary in modeling state variables to enhance their integration into llm responses. future work will refine the integration of state variables, explore other vr training applications, and investigate using prior conversation turns as additional context.