Craft an Iron Sword: Dynamically Generating Interactive Game Characters by Prompting Large Language Models Tuned on Code (2022)

Abstract: Non-Player Characters (NPCs) significantly enhance the player experience in many games. Historically, players’ interactions with NPCs have tended to be highly scripted, to be limited to natural language responses to be selected by the player, and to not involve dynamic change in game state. In this work, we demonstrate that use of a few example conversational prompts can power a conversational agent to generate both natural language and novel code. This approach can permit development of NPCs with which players can have grounded conversations that are free-form and less repetitive. We demonstrate our approach using OpenAI Codex (GPT-3 finetuned on GitHub), with Minecraft game development as our test bed. We show that with a few example prompts, a Codex-based agent can generate novel code, hold multi-turn conversations and answer questions about structured data. We evaluate this application using experienced gamers in a Minecraft realm and provide analysis of failure cases and suggest possible directions for solutions.

## introduction the recent advent of large pre-trained language models such as @xcite and @xcite has fostered spectacular advances in text-generation. in this work, we focus on the potential application of these large language models in video games. in games, non-player characters (npcs) enhance the player experience by providing interaction, often involving conversation. currently players' conversations with npcs are highly scripted: in a typical scenario players must select from a set of preset responses that they can give to the npc. moreover, this interaction is limited to natural language responses, and does not directly involve dynamic game state change as part of the interaction. below, we explore some first steps towards creating functionally agentive npcs with which players can hold free-form conversations that are grounded in the game and which players can instruct to perform actions that change the game state by having the npc adaptively generate code that calls functions exposed by the game api. this is done by a single language model that generates both natural language and code. to this end, we use openai codex @xcite (a gpt-3 model finetuned on github data). we demonstrate that by simply including examples of both natural language conversations and code in the prompt, codex can generalize to interesting new settings, opening up intriguing possibilities for enhanced player experiences and game development. we employ minecraft as our test bed. first, this is an open-world game where players creatively build artifacts in the environment. this makes minecraft a good use case for providing npcs that can converse and perform tasks for the player, something that minecraft currently does not do. second, minecraft has rich game apis in scripting languages, foot_1 that permit models to write function calls that allow the npc perform in-game actions. we investigate these codex-powered npcs through an exploratory user study. we ask experienced gamers to interact with the npc to accomplish tasks in a minecraft realm: obtain crafting recipes, mine resources, craft items and, lastly, break out of two escape rooms. figure 1 shows two sample interactions. we analyze these interactions, and discuss fail cases and what modifications might be needed to handle them. we also present discussion of some interesting avenues of future research in the gaming space that might be achieved by fine-tuning on game apis. ## related work the minecraft gaming environment is increasingly widely used as a platform for researching agents and machine-human collaboration. @xcite is a test-bed for machine learning architectures trained on reinforcement learning. @xcite showcases dialog in which players provide npcs with information and the npcs retain episodic memory and identify player's sentiments. @xcite lays out the motivation for building assistants in minecraft. @xcite describes a framework for dialogenabled interactive agents using high-level, handwritten composable actions. @xcite ) study collaborative conversation between a builder and an architect about structure building. iglu: interactive grounded language understanding in a collaborative environment has emerged a competition to explore interactions in a minecraft environment. @xcite . the model we explore here is distinct from the previous minecraft-related work in that it generates novel code that allows the npc 1) to perform contextually viable actions (moving around, mining, crafting, etc), 2) to answer questions about structured minecraft data (such as crafting recipes) and 3) to engage in multi-turn conversations. this richness does not emerge in a vacuum: it draws on several convergent lines of research. large pre-trained language models (plms) such as @xcite , @xcite and gpt-j have become the predominant paradigm for text generation. research in neural modelling of dialog has focused on powerful new models derived from these, such as di-alogpt @xcite , meena @xcite , @xcite , and lamda @xcite that offer rich potential for open-ended conversational applications. the application of new prompting functions to large plms enables them to perform few-shot or zero-shot learning to adapt to new scenarios with little or no data @xcite . this approach, too, is rapidly being mainstreamed in dialog generation. @xcite employ prompting to select different dialogue skills, access multiple knowledge sources, generate human-like responses, and track user preferences. (zheng and huang, 2021) use prompt-based few-shot learning for grounded dialog generation, in an approach similar to ours. plms that have been tuned on code repositories, typically github, are have begun to be used to automate coding processes and generate code according to programmer's textual specifications, e.g., @xcite and @xcite . @xcite suggest that models pre-trained on code may also benefit semantic parsing for natural language understanding. @xcite explore conversational program synthesis within this framework, and is close in spirit to the current work by virtue of its focus on emergent conversational properties. ## methodology our model is based on few-shot prompting of a large language model, in which a small number of sample instances in the prompt generalize to new unseen input @xcite . we use codex (the code-davinci-002 model) and the mineflayer api, together with minecraft (java edition v.1.17.1). our goal is to have the npc respond to the player's input appropriately according to whether the input requires a purely natural language response or a call to a function to perform some action. figure 2 shows a section of the prompt we provide to the model. the prompt begins with the following statement: "this file contains minecraft bot commands and the code needed to accomplish them using the mineflayer javascript library. if asked something conversational, the bot should use bot.chat() to answer." this tells the model that the prompt includes natural language commands and the code needed to accomplish them. we include in the prompt the natural language commands and the code that need to be generated to enable basic npc functionalities 3 . a new command from the player is appended to this seed prompt and sent to the codex model. in the abstracted code, we evaluate the generated completion. when the completion includes a function call to the game api, the corresponding action is performed by the npc inside the game. when it includes a call to the bot.chat() function, (discussed below) the response string is displayed on the chat interface. for each subsequent input, the prompt includes the seed prompt plus the previous player commands and model completions. when the prompt exceeds the allowed token limit (2048 tokens), we revert to the seed prompt and report to the player that the context has been reset. we further refine this prompting approach using the following strategies: using a stop sequence: since we want only to generate npc responses (and not an entire conversation), we use a stop sequence (comment operator). player input always starts with the stop sequence. ## user study we conduct a user study to evaluate our npc. we invite eight participants who have previously played minecraft. each participant had an hour to complete the study and to answer the post study survey questions. the study consists of five parts: ## npc capabilities on analyzing interactions in the user study, we find that the npc exhibits the following capabilities: parse unseen commands: the npc can understand player commands that are not in the prompt but correspond to an existing functionality. it can map the command 'make me a chest' (not in the prompt) to the right function calls and create the chest or explain why it can't. generalize to new functionality: for some lowlevel functionality, the npc can generalize to un-6 appendix b contains the full list of questions. 7 only 7 of the 8 participants filled the survey. seen functions. for example, since the command 'move forward' is included in the prompt, the npc knows how to call the right functions to move in other directions (backward, right and left). hold multi-turn conversation: the npc can retain the context (both code and language) and maintain a multi-turn conversation in which the npc both responds using natural language and takes actions within the game. generate language about code: the npc can remember multi-turn context and answering questions about the code (and the language) generated in previous contexts. it can answer questions such as "what did you just do?" and "what directions have you moved?" . switch between code & language generation: depending on the player command and the previous context, the npc is able to automatically decide when to respond using natural language and when to generate a function call. question answering: the npc is able to answer questions about its inventory (e.g. "what do u have in your inventory?", "do you have x?", "how many of x do you have" etc), about crafting recipes (e.g. "how can i make a chest?", "how many cobblestones do i need to make a furnace") and answer questions that require both inventory and crafting recipe information (e.g. "how many more cobblestones do i need to make a furnace?"). it can also answer questions generally about minecraft. (the training data for gpt-3 includes minecraft information available on the web.). ## conclusion codex-powered npcs can integrate both conversational and task-oriented language interactions almost seamlessly with code generation in asset-rich contexts, and suggest huge potential for new kinds of gaming experience, including the generation of side quests (appendix f). gaming, moreover, is a rich sandbox-like environment for exploring complex agent interactions with code and addressing issues faced by large language models. the behavior of npcs shed light many of the challenges encountered by large pretrained models of language and code in sustaining persona, goals, and intents over the course of interactions. it remains to be seen whether solutions can be found within existing training and tuning strategies or whether they must be sought outside these models. these are important, ongoing research questions, as are the huge challenges remaining in mapping these interactions to image recognition and to game state.