B yte S ized32: A Corpus and Challenge Task for Generating Task-Specific World Models Expressed as Text Games (2023)

Abstract: In this work we investigate the capacity of language models to generate explicit, interpretable, and interactive world models of scientific and common-sense reasoning tasks. We operationalize this as a task of generating text games, expressed as hundreds of lines of Python code. To facilitate this task, we introduce ByteSized32, a corpus of 32 reasoning-focused text games totalling 20k lines of Python code. We empirically demonstrate that GPT-4 can use these games as templates for single-shot in-context learning, successfully producing runnable games on unseen topics in 28% of cases. When allowed to self-reflect on program errors, game runnability substantially increases to 58%. While evaluating simulation fidelity is labor intensive, we introduce a suite of automated metrics to assess game fidelity, technical validity, adherence to task specifications, and winnability, showing a high-degree of agreement with expert human ratings. We pose this as a challenge task to spur further development at the juncture of world modeling and code generation.

## introduction simulating the world through mental models is a crucial component of human problem solving, inference, and cognition @xcite @xcite . large language models (llms) have demonstrated precursors of this ability, such as encoding a wide range of common-sense world knowledge from their training data @xcite . similarly, llms have been used directly as interactive world simulators in text games like ai dungeon @xcite , where their capacity to predict tokens in context is leveraged to convert natural language user inputs (e.g. open treasure chest) into plausible environmental observations (e.g. you open the chest and find within a glittering sword). ## the bytesized32 corpus to support the task of generating simulations in the form of text games, we construct a corpus of highly-templated text games written in python that can serve as examples in a few-shot generation paradigm. each game is between 500 and 1000 lines of code (including comments), and encodes a task which requires either scientific or commonsense reasoning to complete. for example, a game might require an agent to apply a bandage, load a dishwasher, or make a campfire. each game has two components: the game code itself, as well as a detailed task specification in the form of structured comments at the top of each game that provide a detailed summary of the game task, critical objects, actions, distractors, and a high-level solution plan. these components are described below, with corpus summary statistics provided in ## evaluating generated simulations evaluating model-generated text games presents a number of challenges. first, because games are largely open-ended and constrained only by a short task prompt, the evaluation metrics must be robust to a wide range of potential errors and behaviors. second, evaluating open-ended simulations typically requires manual human evaluation, which is costly and labor intensive. here, we propose a set of fully-automatic metrics that measure both technical aspects of games -such as whether the simulation runs error-free -as well as content aspects that measure how well generated simulations adhere to task specifications. we then validate these automatic instruments with human ratings. ## experiments here we investigate the capacity for a large language model such as gpt-4 foot_0 to generate the hundreds of lines of code required to generate a working text game simulation centered around unseen tasks on each of the technical and quality metrics described above. alongside, we examine the extent to which these models can use reflection to increase generation performance. ## results here, we evaluate all generated games (@xmath0, reporting results both before and after self-reflection. the results of the technical similarly, self-reflection increases the winnability of games. as shown in self-reflection does not improve every metric we evaluate. each of the submeasures of specification compliance, including generating task-critical objects, task-critical actions, and game distractors observes a small decrease or no difference postreflection when measured automatically -suggesting that self-reflecting on technical validity measures affords limited utility to these measures of simulation content. still, we observe strong overall performance in specification compliance, with generated games including task-critical objects in nearly every case, task-critical actions in 93.8% of games, while struggling with distractors -generating these in only 18.8% of games. ## discussion to what extent can gpt-4 generate long structured text games using single-shot in-context learning? at a high level, our results indicate that gpt-4 is frequently capable of generating syntactically valid, templated, and playable programs that ## conclusion in this work, we present bytesized32, a corpus of small world models expressed as text games centered around specific common-sense tasks. using a simulation as code generation paradigm, we show that it is possible to use these games, expressed as hundreds of lines of python code, as templates for in-context learning, and generate novel simulations for unseen tasks. we futher show that it is possible to iteratively self-reflect on these large simulations, and improve on technical validity and physical reality alignment metrics by as much as 29% and 8% respectively. while manually evaluating simulations is labor intensive, we empirically demonstrate that a number of measures of simulation accuracy can be automatically evaluated with moderate-to-strong agreement with expert human ratings. we release this work as open source, and as a challenge task at the intersection of world modeling and code generation, to spur further development in expressing the knowledge contained in language models in more formal and interpretable forms.