A Voice-Controlled Dialogue System for NPC Interaction using Large Language Models (2025)

Abstract: This paper explores the integration of voice-controlled dialogue systems in narrative-driven video games, addressing the limitations of existing approaches. We propose a hybrid interface that allows players to freely paraphrase predefined dialogue options, combining player expressiveness with narrative cohesion. The prototype was developed in Unity, and a large language model was used to map the transcribed voice input to existing dialogue options. The approach was evaluated in a user study (n=14) that compared the hybrid interface to traditional point-and-click methods. Results indicate the proposed interface enhances player’s degree of joy and perceived freedom while maintaining narrative consistency. The findings provide insights into the design of scalable and engaging voice-controlled systems for interactive storytelling. Future research should focus on reducing latency and refining language model accuracy to further improve user experience and immersion.

## introduction voice interaction in video games remains a niche yet promising feature, especially as advances in technology offer new possibilities for immersion and interaction of the player. traditional approaches to voice-controlled dialogues with non-playable characters (npcs) in games generally fall into two categories: reading out pre-written dialogue lines or free speech input in ai-generated dialogues. the former often limits player expression, while the latter can lack narrative consistency and control. this paper aims to present a novel approach that serves as a middle ground between these two approaches, combining the flexibility of player input with structured narrative cohesion. the goal of this paper is to explore the implementation of a voice-controlled interface (vci) that allows players to freely phrase their responses while still choosing from pre-defined dialogue options. by evaluating this hybrid approach, we aim to determine its impact on the player experience, particularly in the context of narrative-driven games. specifically, we address the following research questions: the key contribution of this work lies in an approach to enable spoken interaction in a narrativedriven game that balances player freedom with narrative consistency. we present findings that highlight the potential of this approach in enhancing immersion and user satisfaction while maintaining cohesive storytelling. the remainder of the paper is structured as follows: section 2 presents and discusses other approaches that include voice control into games and discusses how our approach differs. section 3 contains the core concept of the voice-controlled dialogue system with a description of its realization in section 4. sections 5, 6, and 7 present the user study design, the results and their discussion. ## related work voice control as a narrative device in video games has gained significant attention for its potential to enhance player immersion @xcite . natural voice interactions are generally well-received, as they enhance player flow and reduce identity dissonance @xcite . players often mimic character voices @xcite , deepening immersion, though this can be challenging when there are differences in player and character attributes such as gender @xcite . persistent issues with voice interfaces include unnatural interactions, difficulty recalling commands, slower response times compared to button inputs @xcite , and recognition failures @xcite . this section reviews notable approaches specifically for voice-controlled dialogues with non-playable characters (npcs). one established approach is the use of read-outloud interfaces, where players speak predefined dialogue lines to interact with npcs @xcite . here, players cannot freely phrase their voice input but are restricted to the phrasing of the dialogue option they are choosing. this method is reliable and can enhance immersion by encouraging players to embody their characters. for instance, flowers for dan dan (osking and doucette, 2019) used a read-out-loud interface where players verbally selected dialogue options by reading the text of the dialogue option, resulting in higher emotional engagement compared to traditional point-and-click controls. similarly, the dragonborn speaks naturally modification for skyrim @xcite adopted this approach to create more immersive player-npc interactions without the need for complex ai systems. the main advantage of read-out-loud interfaces is their practical integration into existing games, as they rely on predefined dialogue options and require minimal changes to the game's dialogue system. however, the restrictive nature of reading out predefined dialogue lines may limit the player's sense of agency, reducing immersion over extended play sessions. dynamic dialogue generation represents another approach, where npc responses are generated in real-time using ai techniques such as natural language processing (nlp) or large language models (llms). this approach provides players with greater freedom and more natural interactions by allowing them to speak freely rather than selecting from predefined options. for example, the game façade @xcite , later modified by @xcite , employed a "wizard of oz" technique to simulate natural speech input. building on top of its underlying ai systems-natural language processing, autonomous character behaviour, and a drama manager-this approach fostered dynamic and immersive conversations. fraser et al. @xcite ex-tended this concept by incorporating sentiment analysis to adapt npc responses based on player emotions, thereby enhancing engagement. similarly, bot colony @xcite and vaudeville (bumblebee-studios, 2023) utilized ai-driven dialogue systems to generate npc responses. while the use of llms in dialogues with npcs, such as those in vaudeville, can create human-like dialogue that enhances player engagement, they also present challenges including hallucinations, inconsistencies, and difficulty maintaining narrative coherence. @xcite sentiment-driven approach demonstrated improvements in emotional immersion; however, concerns regarding scalability in larger game environments and negative player reactions to ai-generated dialogue remain @xcite . building on these existing methods, this paper proposes a middle-ground solution that integrates the strengths of both approaches. by allowing players to use free-form speech while mapping their input to predefined dialogue options, our method seeks to maintain immersion and deliver a natural interactive experience without compromising narrative control. this hybrid approach offers a more scalable and robust solution for voice-controlled dialogues in narrative-driven games by addressing the challenges identified in earlier research. ## concept the core of the proposed approach lies in a middleground solution for integrating voice interaction in narrative-focused video games. it combines predefined dialogue options with the player's ability to paraphrase freely. to achieve this, players are given predefined dialogue choices that contain very concisely worded versions of the core messages. however, instead of asking the players to read them out loud, they are encouraged to paraphrase these options in their own words. for example, an option like "ask for more information" can be expressed as "could you give me more details?". while in some cases the participants nonetheless opted to read the text as given or nearly so, others were more creative in their formulations. hence, a pre-defined dialogue flow controls the overall dialogue while users can speak freely and naturally. the overall architecture of modelling the dialogue and processing new user input is shown in figure 1 . new user input is first processed in the understanding component that utilizes a large language model to map the user input to one of the possible dialogue options. these options are part of the pre-defined dialogue flow and represented in textual form. the large language model is then prompted to either map the user input to one of the dialogue options given the previous system response, or to map it to misunderstood. ## prototype development the proposed concept is realized in a prototype implementation of a narrative-driven game. the prototype was built using unity, chosen for its flexibility and extensive library of assets. unity handled all game mechanics, visual elements, character interactions, and user interface components. custom c# scripts managed core game interactions, such as dialogue flow, npc responses, and player controls. the game environment and characters were created using free unity asset store resources, providing a functional game world for voice interaction testing. a screenshot of the game is shown in figure 2 . in the tutorial shown in the figure, the player is instructed by an npc to help search for flowers in the forest by selecting one of two options of how to respond. voice interaction: player speech input was captured in unity and processed through the whisper ai service for transcription. given that the language model performed better with english input, the transcribed german text was translated into english via the google cloud translation api before further processing. dialogue management: a structured dialogue graph, implemented with unity's internal tools and custom c# scripts, served as the backbone for dialogue flow. each node in this graph represented a specific narrative point linked to predefined player options. player input was mapped to these options using the llama-2 13b language model, hosted on an nvidia a100 gpu. the model received a prompt that included the transcribed and translated response, the current npc dialogue, and available dialogue options. the model then returned the option number that best matched the player's intent. prompt engineering was applied by using langchain to improve mapping accuracy and reduce latency. an excerpt of the system message is shown in figure 3 . user interface: developed within unity, the user interface displayed available dialogue options and provided immediate visual feedback. when players used voice input, the ui indicated ongoing processing and highlighted the chosen option after recognition, helping players understand the system's response to their spoken input. ## user study the user study aimed to evaluate the proposed voice-controlled interface (vci) by comparing it with a traditional point-and-click interface (pci). the study combined usability testing, a/b testing, and surveys to assess the system's impact on user experience, perceived freedom, and system accuracy. the study involved 14 participants, consisting of an equal number of male and female individuals, v low low neutral high v high aged between 23 and 36 years (mean age: 29, standard deviation: 4). most participants (11) held university degrees, and all were native german speakers. participants were personally recruited and included a mix of friends, acquaintances, and individuals with no close connection to the researchers. this group represented varied levels of familiarity with gaming and voice interfaces. while participants had moderately high experience with video games, their exposure to voice interfaces was comparatively limited (see fig. 4 ). participants alternated between the two interfaces to counterbalance order effects, with one group using the vci first and the other starting with the pci. the procedure included the following phases: the study was conducted on a laptop equipped with the unity-based prototype. voice input was captured using a logitech webcam microphone, chosen for its accuracy over the laptop's built-in microphone. participants completed questionnaires on the same laptop. audio recordings documented verbal interactions, while logs captured system responses, dialogue choices, and observational notes. two primary data sources, questionnaires and play-through documentation, informed the study's findings. questionnaires: participants responded to a series of structured questions using seven-point likert scales. the questionnaires were adapted from existing instruments, namely the sassi @xcite for assessing the speech interface with regard to the usability aspects, and the guess @xcite for measuring video game satisfaction and user experience. the adapted questionnaire covered five scales: playthrough documentation: logs recorded dialogue choices, llm prompt-responses, and voice input accuracy. audio recordings and observational notes provided qualitative insights into user behavior, naturalness of interactions, and system responsiveness. the study faced several limitations that must be acknowledged. the small and relatively homogeneous sample, consisting of younger participants with higher education, is not representative of the broader gaming population. the limited duration of the study restricted participants' familiarity with the interfaces, potentially limiting the learning curve and long-term usability assessment. some prototype limitations, such as latency, translation inaccuracies, and limited npc animations, likely influenced user perceptions of the system. factors like mood, time of day, and external distractions could also have impacted participant performance and feedback. ## results the results of the user study are presented in this section, focusing on the impact of the voicecontrolled interface (vci) on immersion, user experience, perceived freedom, and accuracy of spoken inputs. a total of 14 participants completed the study, which involved gameplay with both the vci and a traditional point-and-click interface (pci), followed by corresponding questionnaires. an overview of the results for the vci is shown in the questionnaire responses were collected on a 7-point likert scale ranging from "strongly disagree" (1) to "strongly agree" (7). for positively phrased statements, higher values indicate a more favorable response, while for negatively phrased statements, the scale was reversed to ensure consistency in interpretation, where higher values always reflect a positive attitude towards the vci. ## discussion this section offers a comprehensive discussion of the user study results and final reflections on the voice-controlled interface (vci) prototype, synthesizing the findings, implications, limitations, and directions for future research. ## conclusion this study contributes to the growing field of voice interaction in video games, particularly npc interactions. the hybrid vci approach-combining predefined dialogue options with the ability to paraphrase-has proven to be an engaging feature that enhances a player's sense of freedom and overall user experience. allowing players to interact naturally, in their own words, creates a more personalized experience that can align well with narrativedriven games. while the study's findings show the potential of voice-controlled interfaces, the technical and methodological challenges identified must be addressed for long-term success. improvements in system speed, reliability and more advanced character design are critical for impacting the sense of immersion to a greater extent. with these enhancements, voice interaction could become an integral part of video game dialogue systems, providing a richer and more immersive player experience.