Behavior Cloned Transformers are Neurosymbolic Reasoners (2023)

Abstract: In this work, we explore techniques for augmenting interactive agents with information from symbolic modules, much like humans use tools like calculators and GPS systems to assist with arithmetic and navigation. We test our agent’s abilities in text games – challenging benchmarks for evaluating the multi-step reasoning abilities of game agents in grounded, language-based environments. Our experimental study indicates that injecting the actions from these symbolic modules into the action space of a behavior cloned transformer agent increases performance on four text game benchmarks that test arithmetic, navigation, sorting, and common sense reasoning by an average of 22%, allowing an agent to reach the highest possible performance on unseen games. This action injection technique is easily extended to new agents, environments, and symbolic modules.

## introduction interactive fiction games (or text games) evaluate ai agents abilities to perform complex multistep reasoning tasks in interactive environments that are rendered exclusively using textual descriptions. agents typically find these games challenging due to the complexities of the tasks combined with the reasoning limitations of contemporary models. overall performance is generally low, with agents currently solving only 30% of classic interactive fiction games such as zork (ammanabrolu @xcite @xcite . similarly, reframing benchmarks such as question answering into text games where agents must interactively reason with their environment and make their reasoning steps explicit causes performance to substantially decrease @xcite , highlighting both the capacity figure 1 : an overview of our approach on an example game evaluating arithmetic ability. at each step, the agent receives an observation from the environment, then takes an action. by providing actions that interface to symbolic modules (such as a calculator), the agent is able to use external knowledge to help solve the task. while large language models are capable of a variety of common sense reasoning abilities @xcite , contemporary agents typically struggle on tasks such as navigation, arithmetic, knowledge base lookup, and other tasks that humans typically make use of external tools (such as gps systems, calculators, and books) to solve. this is at times frustrating, because the tasks they perform poorly on can sometimes be solved in a few dozen lines of code. in this work, we show that combining both approaches is possible for text game agents, with our approach shown in figure 1 . we develop symbolic modules for arithmetic, navigation, sorting, and knowledge base lookup in python, paired with new benchmark games for testing these capacities in interactive text game environments. we empirically demonstrate that injecting actions from those modules into the action space of each game can allow transformerbased agents to make use of that information, and achieve near-ceiling performance on unseen benchmark games that they otherwise find challenging. ## related work neurosymbolic reasoning offers the promise of combining the inference capabilities of symbolic programs with the robustness of large neural networks. in the context of text games, @xcite develop methods to decompose text games into a set of logical rules, then combine these rules with deep reinforcement learning @xcite or integer linear programming @xcite to substantially increase agent performance while providing a more interpretable framework for understanding why agents choose specific actions @xcite . more generally, neurosymbolic reasoning has been applied to a variety of multi-step inference problems, such as multi-hop question answering @xcite , language grounding @xcite , and semantic analysis @xcite . because text games require interactive multistep reasoning, agents have most commonly been modelled using reinforcement learning (e.g. @xcite @xcite , though overall performance on most environments remains low (see @xcite . recently, alternative approaches modeling reinforcement learning as a sequence-to-sequence problem using imitation learning have emerged, centrally using behavior cloning @xcite , decision transformers @xcite , and trajectory transformers @xcite . these approaches model interactive multi-step reasoning problems as a markov decision process, where an agent's observation and action history up to some depth are provided as input, and the transformer must predict the next action for the agent to take. behavior cloning and decision transformers have recently been applied to text games with limited success @xcite . here, we show that the performance of a behavior cloned transformer can substantially increase when augmented with neurosymbolic reasoning. ## approach figure 1 illustrates the workflow of our approach. at each time step t, based on the observation o t , the symbolic module will generate a set of valid actions a m t , and the text game environment will have a distinct set of valid actions a e t . let the valid action set at step t be a @xmath0. given o t and a t , the agent needs to choose an action a @xmath1. note that in principle, any agent could be adapted to use this approach, since we simply inject actions from the symbolic modules into the environment action space. at a given time step, our approach checks if a t is a valid symbolic action. if a @xmath2.g. div 22 11 in figure 1 ), the symbolic module will generate the next observation o t+1 , otherwise the text game environment will take a t and generate o t+1 (e.g. take 2 bananas in figure 1 ). ## environments and symbolic modules we evaluate our approach to neurosymbolic reasoning using four text game benchmark environments centered around pick-and-place tasks, including one existing benchmark and three new developed for this work. each environment supports parametric variation to generate many different games. these environments are outlined below, with additional details and example playthroughs found in appendix b. all environments were implemented using the textworldexpress game engine (jansen and côté, 2022). text world common sense (twc): a benchmark common sense reasoning task @xcite where agents must collect objects from the environment (e.g. dirty socks), and place those objects in their canonical common sense locations (e.g. washing machine). the symbolic module for this game allows agents to query a knowledge base of (subject, relation, object) triples (e.g. (cushion, hascanonicallocation, sofa)). mapreader: a navigation-themed pick-and-place game similar to coin collector @xcite ). an agent starts in a random location (e.g. the kitchen), and is provided with a target location (e.g. the garage). the agent must navigate to the target location, pick up a coin, then return to the starting location and place it in a box. the agent is further provided with a map that can be used for efficient route planning. the navigation symbolic module paired with this environment scrapes the observation space for location information (e.g. you are ## models in this section, we introduce the reinforcement learning and behavior cloning agents used in our experiments. additional details and hyperparameters are provided in appendix a. deep reinforcement relevance network (drrn): the drrn @xcite is a fast and strong reinforcement learning baseline that is frequently used to deliver near state-of-the-art performance in a variety of text games (e.g. @xcite @xcite . at each step, the drrn separately encodes the observation and candidate actions using several grus @xcite . a deep q-network is then used to estimate q-values for each (observation, candidate action) pair. the candidate action with the highest predicted q-value will be chosen as the next action. behavior cloning: behavior cloning @xcite is a form of imitation learning similar to the decision transformer @xcite that models reinforcement learning as a sequence-tosequence problem, predicting the next action given a series of previous observations. we follow the strategy of @xcite in adapting behavior cloning to text games, where the model input at step t includes the task description, current state observation, previous action, and previous state observation (d, o t , a t-1 , o t-1 ). during training, the agent is fine-tuned on gold trajectories, where the training target is to generate action a t from the gold trajectories. during evaluation, the agent performs inference online in the text game environment. for experiments reported here, we used a t5-base model @xcite . ## results and discussion the results of both drrn and behavior cloning experiments across each benchmark are shown in ## conclusion in this paper, we present an approach to neurosymbolic reasoning for text games using action space injection that can be easily adapted to existing text game environments. for models that are capable of exploiting the information provided by the symbolic modules, this technique allows agents to inexpensively augment their reasoning skills to solve more complex tasks. we empirically demonstrate this approach can substantially increase task performance on four benchmark games using a behavior cloned transformer.