The T uring Quest: Can Transformers Make Good NPC s? (2023)

Abstract: In this paper, we study the viability of the deployment of language models towards non-playable character (NPC) scripts, by introducing a novel pipeline for the automatic construction of NPC scripts using Transformer-based believable scripts for a variety of game genres and specifications. In addition, we propose a self-diagnosis method inspired by previous work to develop language models, tailored specifically to desirable NPC qualities such as coherency, believability, and degree of repetition. Finally, we propose a new benchmark, called The Turing Quest, which we use to show that the pipeline, when applied to GPT-3, can generate for a variety of game genres and contexts, NPC scripts that can fool judges in thinking they have been written by humans. We believe that these findings can greatly benefit both the gaming industry and its global community of users, since many current games continue to base their NPCs on manually-curated scripts that are resource-demanding and may curb the immersiveness and enjoyment of the user.

## introduction over the past decade, there has been a growing interest in applying deep learning models to natural language generation (nlg) for open-domain dialogue systems and conversational agents. in parallel, the gaming industry has been striving to create more immersive experiences for players by enhancing their interactions with non-playable characters (npcs). however, the potential of utilizing state-ofthe-art deep learning models, such as transformerbased models, to create npc scripts remains largely unexplored. pre-trained transformer-based language models (plms) like openai's gpt-3 @xcite and chatgpt @xcite have demonstrated impressive conversational abilities @xcite . in certain contexts, the text generated by these models can be nearly indistinguishable from human-written text @xcite without the aid of external tools or watermarks @xcite . the use of these models in real-world applications has been expanding in areas such as customer service automation @xcite , educational conversational agents @xcite , and mental health dialogue systems @xcite . despite their growing prevalence, the effectiveness and generalization capabilities of plms in various contexts remain uncertain. one such uncharted domain is the creation of "non-playable characters" or npcs in video games. when comparing chatbots to npcs, the latter can be considered as a narrative-driven variant of goal-oriented chatbots. however, npcs and chatbots serve different purposes and operate in distinct environments. generating npc scripts presents unique challenges, as the dialogue must be consistent with the game's plot, genre, and the npc's character to maintain player immersion and suspension of disbelief @xcite . according to @xcite , npc believability hinges on "the size and nature of the cognitive gap between the [npc that] players experience and the [npc] they expect". players anticipate npcs with individualized and possibly dynamic traits, which should be reflected in their dialogue. while incorporating personality into dialogue systems is well-studied @xcite @xcite , the challenge of generating goal-oriented, believable npc scripts that align with a game's narrative and thematic elements, while preserving player immersion, remains substantial. the ability to automatically generate contextually appropriate dialogue for a specified character could have an effect on the design paradigms of future video games. while manually scripted narratives and plot points will continue to hold their value, developers could augment player immersion by allowing an array of npcs to dynamically respond to a player's in-game progression. traditionally, game design involves scripted dialogues only for npcs that contribute directly to a quest or story line, thereby limiting the extent of player interaction. it is not often possible for a player to initiate a conversation with a companion about an ongoing quest or solicit their views, creating an impression that, from an npc's perspective, the player's existence is confined to the quests they undertake. simply implementing an interactive companion system necessitates writing dialogues for every quest for all possible companions-a laborintensive task. expanding this system to encompass a majority of a game's npcs would further compound these challenges, increasing the amount of labour to an unreasonable degree. the vast amount of dialogue required for each narrative stage would significantly exceed typical time and resource constraints of most developers. despite the potential enrichment of the player experience, the practicality of creating such an immersive, dialogue-rich environment using solely human-authored dialogue in game development remains questionable. in this study, we investigate the application of transformer-based models like gpt-3 to the task of creating npcs and generating believable scripts. to this end, we develop an npc construction pipeline capable of generating dialogue based on the npc's attributes alone. our pipeline com-prises three key modules: a) a feature characterization schema that classifies npcs based on personality traits and world descriptions, b) an automatic prompt creation process that employs the schema to generate tailored prompts for conditioning language models, and c) a dialogue generation phase that uses the customized prompts to generate scripts with transformer-based plms. figure 1 provides an example of dialogue generated through this pipeline. we also devise and automate an evaluation metric for npc dialogue quality, drawing inspiration from related literature @xcite . lastly, we propose the turing quest: a test using human judges to assess the believability and quality of generated npc scripts. ## related work in recent years, there has been a growing interest in dialogue systems and conversational agents. however, the exploration of dialogue generation for npcs in video games, despite their similarities to chatbots, remains limited. although most video games in the past decade include npc dialogue, research on automating its creation using artificial intelligence (ai) is still in its infancy. ## npc construction pipeline the objective of the npc construction pipeline is to automatically generate coherent, contextually appropriate, and engaging utterances for an npc, given the dialogue history between the npc and a player, as well as the contextual information about the npc and the game. the pipeline consists of three modules, which serve to a) characterize the npc according to a generalized representation schema that captures crucial information about the npc's role, personality, and game context, b) generate short prompts based on the characterization, providing contextually relevant pretexts for the language model (lm), and c) generate utterances based on these prompts using an lm optimized for npc dialogue generation. ## evaluation to assess the performance of the npc construction pipeline and the resulting generated dialogue, we designed a comprehensive evaluation metric that examines dialogue quality based on coherency, believability, degree of repetition, alignment of the npc's dialogue with their role, and fittingness of the npc's dialogue within their world. these categories draw from and adapt kalbiyev (2022)'s metric for evaluating video game dialogue. each metric is assigned a score between one and five, with the sum of these scores indicating the overall quality of the dialogue. self-diagnosis harnesses the capacity of transformer-based language models to detect patterns within text and their few-shot learning performance to enable rapid, automated evaluation of dialogue without prior fine-tuning. we conducted a human evaluation of 66 different npc scripts to assess the accuracy and reliability of our self-diagnosis approach. after each conversation was evaluated and scored, we found a correlation between parameters and their average score. by including our full npc header, we were able to generate dialogue of higher quality. we then conducted a single-blind test where human judges were asked to determine whether an npc script was generated by ai or written manually by a human. ## conclusion we developed a novel pipeline capable of automatically generating npc scripts comparable or of superior quality to human-written npc dialogue using transformer-based plms. we then created a self-diagnosis module which provides a method to evaluate and compare the quality of npc dialogue quantitatively. finally, our proposal of the turing quest allows us to determine the capabilities of a language model when applied to the task of npc dialogue generation and whether a script passes as human-written. while the npc construction pipeline allows for modularity even in between responses, that aspect was not explored in depth in this paper. we will explore dialogue generation for dynamic npcs with evolving roles or attributes in future research.