STARLING : Self-supervised Training of Text-based Reinforcement Learning Agent with Large Language Models (2024)

Abstract: Interactive fiction games have emerged as an important application to improve the generalization capabilities of language-based reinforcement learning (RL) agents. Existing environments for interactive fiction games are domain-specific or time-consuming to generate and do not train the RL agents to master a specific set of skills. In this work, we introduce an interactive environment for self-supervised RL, STARLING, for text-based games that bootstraps the text-based RL agents with automatically generated games (based on the seed set of game ideas) to boost the performance and generalization capabilities to reach a goal of the target environment. These games let the agent hone their skills on a predefined set of tasks. We create and test an environment with 100 games, generated using this automated framework that uses large language models (GPT3) and an interactive fiction game engine (based on Inform7) to provide the user with the ability to generate more games under minimal human supervision. Experimental results based on both the human participants and baseline text-based RL agents reveal that current state-of-the-art text-based RL agents cannot use previously learned skills in new situations at the level humans can. These results enforce STARLING’s potential to serve as a sandbox environment for further research in self-supervised text-based RL.

## introduction interactive fiction games such as zork can be utilized as an important test-bed to improve the generalization capabilities of text-based reinforcement learning (tbrl) agents @xcite . in these games, both the observed state of the game and the actions taken are in natural language. to play these games, the agents (or human players) need to understand the observed text from the environment and take relevant action toward the goal. these games encourage agents to understand the underlying state of the game and take actions to interact with the environment. in order to be successful, agents must use previously learned skills in new situations to complete an overarching goal. current environments of interactive fiction games suffer from two major problems. first, environments such as textworld commonsense measure simple commonsense reasoning based on one-hop relationships between entities (e.g., @xmath0. second, environments such as scienceworld (even though many variations of task-based games are available) and jericho are domain-specific so agents that play these environments may perform well while conducting specific tasks like completing science experiments but lack generalized skills to apply them to other situations @xcite . most importantly, in order to generate a large number of games to train the rl agents to master skills in these environments, we will have to employ human annotators to manually design, generate, and deploy the game. therefore, the purpose of this work is to develop an efficient approach that generates a large amount of text-based games to train the rl agents to master the desired skills and excel at the target environments such as twc, scienceworld, etc. as developing a set of text-based games is a time-intensive manual process, we propose self-supervised text-based rl learning, "star-ling", an interactive environment that utilizes large language models (llm) and an integrated interactive fiction game engine (inform7 @xcite ) to easily produce games in any domain. we generate a set of 100 text-based games using gpt3 @xcite based on the input game ideas (seed list) that emphasize the need for the everyday skills such as boiling water, cooking pasta, etc. in (pre-)training text-based rl agents. these games require agents to use a specific sequence of actions to achieve the goal and successfully complete the game. for example, while cooking pasta, an agent must first gather the ingredients, fill pot with water, boil the water, and put the pasta in the pot. we then deploy the pre-trained rl agent on the target environments. this novel game-generation method can easily be used by others to create their own games and be adapted for future applications to build challenging rl agents in various domains. figure 1 shows the overview of the proposed approach for self-supervised text-based reinforcement learning using llm. ## experiments in this section, we report the experimental results of the proposed approach: starling. we pre-train the rl agent on the generated 100 games (train split). we evaluate the pre-trained agent (star-ling) on three benchmark environments for textbased games: textworld commonsense (twc) with easy, medium, and hard difficulty levels (murugesan et al., 2021a) 2) scienceworld with 4 tasks and variations @xcite . 3) zork1 from jericho. ## conclusion in this paper, we proposed a novel self-supervised training for a text-based reinforcement learning agent, starling, with the help of the generalized skill generation capability of large language models like gpt3. we generated a set of text-based games that require agents to learn basic skills such as cooking pasta, boiling water, etc., and utilize sequential decision-making over the modality of text. the proposed starling uses the gpt3 pretrained language model to automatically generate these games. this approach can be used to create additional games or adapted to build games for new domains with minimal human intervention. we showed that the starling agent pre-trained on the games generated by llm outperforms vanilla tbrl. we evaluated starling on three environments: scienceworld, twc, and zork1. in all these environments, starling showed enhanced skills in the target environment.