Self-Supervised Behavior Cloned Transformers are Path Crawlers for Text Games (2023)

Abstract: In this work, we introduce a self-supervised behavior cloning transformer for text games, which are challenging benchmarks for multi-step reasoning in virtual environments. Traditionally, Behavior Cloning Transformers excel in such tasks but rely on supervised training data. Our approach auto-generates training data by exploring trajectories (defined by common macro-action sequences) that lead to reward within the games, while determining the generality and utility of these trajectories by rapidly training small models then evalauating their performance on unseen development games. Through empirical analysis, we show our method consistently uncovers generalizable training data, achieving about 90% performance of supervised systems across three benchmark text games.

## introduction complex tasks often involve decomposing problems into sequential steps to accomplish a goal. this is particularly the case for text games, which are interactive virtual environments where an agent progressively selects actions until a task is complete. examples of such tasks range from cooking a recipe @xcite , finding a treasure @xcite , executing a science experiment @xcite , to evaluating common sense reasoning @xcite @xcite . text game agents are frequently modeled using reinforcement learning (e.g. @xcite @xcite @xcite , wherein an agent explores an environment with the goal of learning a policy that chooses actions leading to reward and task completion. more recently, text game agents recast reinforcement learning as tories from the current state are extracted up to a specified horizon, which extends to the first reward. the generality of these paths is assessed by training a compact t5 model and evaluating its performance on unseen development games. highperforming trajectories are subsequently extended through further exploration in an iterative process until the game is ultimately completed. in figure 1 , we present our approach to overcome this data supervision limitation by automatically crawling, grouping, and evaluating candidate paths that can serve as useful and generalizable training data. by rapidly evaluating these candi- ## 2 game variation 1: take 24 bananas, put 24 bananas in answer box game variation 2: take 12 apples, put 12 apples in answer box game variation 3: take 16 pears, put 16 pears in answer box ... path group: take(x), put (x, y) ## approach figure 1 provides an overview of our approach. the goal of our work is to find a set of generalizable training data that can be a substitute for human gold playthroughs of text games, and suitable for training a behavior cloning transformer agent. while simply exhaustively pathcrawling all possible trajectories in a game offers a method to find winning paths, not all winning paths are gener-alizable (see figure 2 ), so such a method would yield poor training data, while also being computationally expensive. here, we show that by training small models on candidate training data, and evaluating performance on unseen development games, we can quickly evaluate the generalizability of that data. in terms of tractability, by iteratively pathcrawling only up to the next reward, and then continuing pathcrawling only from the most generalizable paths, we can limit the size of the search to only paths most likely to be generalizable and form quality training data. in this work, we use text games that have parametric variations of each task, such as different math problems to solve for an arithmetic game, which is standard practice for evaluating a model's generalization ability (e.g., @xcite @xcite . path crawling: our process begins with a path crawler exploring all potential trajectories in a game from the start state until achieving non-zero reward or reaching a limited horizon. this crawler also navigates multiple parametric game variations, each with distinct objectives, such as making a stirfry versus baking a cake in a cooking game. path grouping into parameterized macro-action sequences: text game actions consist of a command verb (e.g., take, put, open) with one (e.g. open closet) or two (e.g. put apple in cupboard) arguments, where a sequence of actions (i.e. a full or partial playthrough) is called a trajectory. when creating candidate training sets for the behavior cloning transformer, we attempt to group together trajectories that -as best as we can tell -appear to solve different parametric variations of a game using roughly the same sequence of actions. we do this by abstracting the trajectories into variabilized action sequences. for example, in the textworld common sense game, a trajectory such as "take hat", "put hat on hat rack" from one variation, and "take dirty shirt", "put dirty shirt in washer" from another variation, could both be abstracted into the same parameterized macroaction sequence "take(x), put(x, y)". we call a set of trajectories from different game variations that can all be described using the same parameterized macro-action sequence a path group. in the experiments reported below, path groups are typically created by grouping paths from 100 training variations of a game. evaluating path groups: the above process yields n path groups, each representing a unique parameterized macro-action sequence. these groups provide the training data for a behavior cloning model. although n is typically large, we've observed that generalizable sequences are often among the shorter paths (though not the shortest -figure 2 provides a counterexample). for efficiency, we select the k shortest path groups and train a t5-based behavior cloning transformer individually on each, assessing the model performance on 100 unseen game variations from the development set. 2 the process continues until we identify a group surpassing a predefined performance threshold t . if no single group meets this early stopping criterion, we attempt to merge path groups to discover higher-scoring combinations, detailed further in appendix c.2.1. incremental path crawling: reinforcement learning problems can offer sparse or dense rewards. to minimize the search space and the total number of path groups to evaluate, we initially crawl and assess paths up to the first reward instance. upon identifying a path group with gener2 an analysis of performance versus number of parametric variations included in training is provided in the appendix. alizable performance to this point, the model continues path crawling, commencing with this macroaction sequence and assessing generality until the next reward signal. this cycle persists until reaching a winning state, effectively segmenting the task into subtasks demarcated by non-zero rewards. benchmarks: our exploration of using path crawling for generating self-supervised training data spans three benchmark text games. textworld common sense (twc) @xcite tasks agents with assigning household items (e.g. a dirty shirt) to their canonical locations (e.g. a washing machine). arithmetic engages agents in a math problem, followed by a pick-and-place task using the calculated result. sorting requires agents to arrange objects by quantity. each game presents parametric variations altering the environment and task-specific objects across episodes. for all experiments, we generate distinct training, development, and test sets, each with 100 variations of each game. objects crucial to tasks are unique across sets. refer to the appendix for game specifics and playthrough examples. ## models we assess four models to showcase the effectiveness of the self-supervised behavior cloning transformer. further model details and hyperparameters can be found in the appendix. ## results and discussion how do the trajectories identified by selfsupervision compare to gold trajectories? as shown in how can the performance of this method versus the drrn and gpt-4 be contextualized? our goal in this work is to demonstrate selfsupervision of a behavior cloning transformer using the pathcrawling method described in section 2. nevertheless, other baselines offer valuable context. the poor performance of the drrn, a robust reinforcement learning baseline for text games, underscores the difficulty of exploring expansive search spaces without guiding context -one of the motivating factors towards the application of pretrained language models to guide exploration more effectively. on the other hand, while the gpt-4 model performs comparably to the self-supervised behavior cloning model, it is likely to be at least three orders of magnitude larger. this highlights the potential for small models to perform well given suitable exploration mechanisms. how many training examples are required to learn a task? figure 3 shows the performance of the t5-agent trained with varying numbers of gold trajectories. generally, task performance increases roughly linearly with the number of training examples provided. an exception to this is the arithmetic game, where nearly all 100 training 0 20 40 60 80 100 0 0.2 0.4 0.6 0.8 1 arithmetic sorting twc proportion of training examples that are gold(%) score figure 4: performance of the supervised behavior cloning agent on each of the three benchmark games, when trained with varying proportion of gold trajectories interspersed with randomly sampled non-gold trajectories. error bars represent the standard deviation of 5 runs per data point on the development set. arithmetic ss take(x), read(x), look-around, take(y), put(y, z) gold take(x), read(x), take(y), put(y, z) sorting ss take(x), put(x, y), take(z), put(z, y), take(a), ... gold take(x), put(x, y), take(z), put(z, y), take(a), ... text world common sense ss (take(x), look-around, put(x, y)) (take(x), inventory, open(y), put(x, y)) gold (take(x), put(x, y)) (take(x), open(y), put(x, y)) how is this method affected by training hyperparameters? we have observed substantial differences in performance across various training hyperparameters (such as number of training epochs, or training random seed), and suspect part of the difference between supervised and self-supervised performance may be due to these fluctuations. ## conclusion we present a self-supervised behavior cloning transformer for text games, that leverages a process of incremental path crawling in action spaces leading to rewards, while evaluating the generality of groups of crawled paths on unseen games, to generate highquality training data. our method achieves comparable performance to models trained on gold trajectories, efficiently exploring the environment without needing human playthroughs or gold agents. our model excels over a robust reinforcement learning baseline and attains about 90% of supervised models' performance, emphasizing the potential of efficient exploration mechanisms, even for smaller models.