A Minimal Approach for Natural Language Action Space in Text-based Games (2023)

Abstract: Text-based games (TGs) are language-based interactive environments for reinforcement learning. While language models (LMs) and knowledge graphs (KGs) are commonly used for handling large action space in TGs, it is unclear whether these techniques are necessary or overused. In this paper, we revisit the challenge of exploring the action space in TGs and propose \epsilon -admissible exploration, a minimal approach of utilizing admissible actions, for training phase. Additionally, we present a text-based actor-critic (TAC) agent that produces textual commands for game, solely from game observations, without requiring any KG or LM. Our method, on average across 10 games from Jericho, outperforms strong baselines and state-of-the-art agents that use LM and KG. Our approach highlights that a much lighter model design, with a fresh perspective on utilizing the information within the environments, suffices for an effective exploration of exponentially large action spaces.

## introduction an intelligent agent that communicates in natural language space has been a long goal of artificial intelligence @xcite . text-based games (tgs) best suit this goal, since they allow the agent to read the textual description of the world and write the textual command to the world @xcite . in tgs, the agent should perform natural language understanding (nlu), sequential reasoning and natural language generation (nlg) to generate a series of actions to accomplish the goal of the game, i.e. adventure or puzzle @xcite . the language perspective of tgs foists environments partially observable and action space combinatorially large, making the task challenging. since tgs alert the player how much the game has proceeded with the game score, reinforcement learning (rl) naturally lends itself as a suitable framework. due to its language action space, an rl agent in tgs typically deals with a combinatorially large action space, motiving various design choices to account for it. as two seminal works in this space, @xcite trained a language model (lm) to produce admissible actions foot_1 for the given textual observation and then used, under the predicted action list, deep reinforcement relevance network to estimate the q value. as an alternative, @xcite constructs a knowledge graph (kg) to prune down action space while learning the policy distribution through actorcritic (ac) method and supervision signal from the admissible actions. both paradigms leverage admissible actions at different stages at the cost of imposing additional modules and increasing model complexity. in this paper, we take a fresh perspective on leveraging the information available in the tg environment to explore the action space without relying on lms or kgs. we propose a minimal form of utilizing admissibility of actions to constrain the action space during training while allowing the agent to act independently to access the admissible actions during testing. more concretely, our proposed training strategy, ϵ-admissible exploration, leverages the admissible actions via random sampling during training to acquire diverse and useful data from the environment. then, our developed textbased actor-critic (tac) agent learns the policy distribution without any action space constraints. it is noteworthy that our much lighter proposal is under the same condition as other aforementioned methods since all the prior works use admissible actions in training the lm or the agent. our empirical findings, in jericho, illustrate that tac with ϵ-admissible exploration has better or on-par performance in comparison with the stateof-the-art agents that use an lm or kg. through experiments, we observed that while previous methods have their action selections largely dependent on the quality of the lm or kg, sampling admissible actions helps with the action selection and results in acquiring diverse experiences during exploration. while showing a significant success on tgs, we hope our approach encourages alternative perspectives on leveraging action admissibility in other domains of applications where the action space is discrete and combinatorially large. ## basic definitions text-based games. tgs are game simulation environments that take natural language commands and return textual description of the world. they have received significant attention in both nlp and rl communities in recent years. @xcite introduced textworld, a tg framework that automatically generates textual observation through knowledge base in a game engine. it has several hyper-parameters to control the variety and difficulty of the game. @xcite released jericho, an open-sourced interface for human-made tgs, which has become the de-facto testbed for developments in tg. ## related work on tg agents in rl we provide a brief overview of widely known tg agents relevant to the work presented in this paper. we empirically compare these in the section 5.1. contextual action lm (calm)-drrn @xcite uses an lm (calm) to produce a set of actions for a given textual observation from the tgs. it is trained to map a set of textual observations to the admissible actions through causal language modeling. then, deep reinforcement relevance network (drrn) agent was trained on the action candidates from calm. drrn follows ql, estimating the q value per observation-action pair. as a result, calm removes the need for the ground truth while training drrn. 3 knowledge graph advantage actor critic (kg-a2c) @xcite uses the ac method to sequentially sample templates and objects, and kgs for long-term memory and action pruning. throughout the gameplay, kg-a2c organizes knowledge triples from textual observation using stanford openie @xcite to construct a kg. then, the kg is used to build state representation along with encoded game observations and constrain object space with only the entities that the agent can reach within kg, i.e. immediate neighbours. they used admissible actions in the cross entropy supervised loss. kg-a2c inspired agents. @xcite proposed sha-kg that uses stacked hierarchical attention on kg. graph attention network (gat) was applied to sample sub-graphs of kg to enrich the state representation on top of kg-a2c. ammanabrolu et al. (2020) used techniques inspired by question answering (qa) with lm to construct the kg. they introduced q*bert which uses al-bert @xcite fine-tuned on a dataset specific to tgs to perform qa and extract information from textual observations of the game, i.e. "where is my current location?". this improved the quality of kg, and therefore, constituted better state representation. @xcite proposed an exploration technique that injects commonsense directly into action selection. they used log-likelihood score from commonsense transformer @xcite to re-rank actions. @xcite investigated explainable generative agent (hex-rl) and applied hierarchical graph attention to symbolic kg-based state representations. this was to leverage the graph representation based on its significance in action selection. they also employed intrinsic reward signal towards the expansion of kg to motivate the agent for exploration (hex-rl-im) @xcite . all the aforementioned methods utilize admissible actions in training the lm or agent. our proposed method, introduced shortly ( §4), uses admissible actions as action constraints during training without relying on kg or lm. ## text-based actor critic (tac) our agent, text-based actor critic (tac), follows the actor-critic method with template-object decoder. we provide an overview of the system in figure 1 and a detailed description in below. we follow the notation introduced earlier in section 2. encoder. our design consists of text and state encoders. text encoder is a single shared bidirectional gru with different initial hidden state for different input text, (o game , o look , o inv , a n ). the state representation only takes encoded textual observations while the natural language action a n is encoded to be used by the critic (introduced shortly). state encoder embeds game scores into a high dimensional vector and adds it to the encoded observation. this is then, passed through a feed-forward neural network, mapping an instance of observation to state representation without the history of the past information. actor. the actor-critic design is used for our rl component. we describe our generative actor first. our actor network maps from state representation to action representation. then, the action representation is decoded by gru-based template and object decoders @xcite . template decoder takes action representation and produces the template distribution and the context vector. object decoder takes action representation, semi-completed natural language action and the context from template decoder to produce object distribution sequentially. critic. similar to @xcite , we employed two types of critics for practical purpose, state critic for state value function and state-action critic for state-action value function. both critics take the state representation as input, but stateaction critic takes encoded natural language action as an additional input. the textual command produced by the decoder is encoded with text encoder and is passed through state-action critic to predict state-action value, or q value, for a given command. a more detailed diagram for actor and critic is in appendix d. to smooth the training, we introduced target state critic as an exponentially moving average of state critic @xcite . also, the objective function. our objective functions are largely divided into two, rl and sl. rl objectives are for reward maximization l r , state value prediction l v , and state-action value prediction l q . we overload the notation of θ: for instance, v θ (o) signifies parameters from the encoder to the critic, and π θ (a|o) from the encoder to the actor. reward maximization is done as follows, our algorithm is akin to vanilla a2c proposed by @xcite with some changes under our observations. a detailed comparison and qualitative analysis are in appendix e and f. ϵ-admissible exploration. we use a simple exploration technique during training, which samples the next action from admissible actions with ϵ probability threshold. for a given state s, define a a (s) ⊆ a n as an admissible action subset of all natural language actions set. we sample an action directly from admissible action set under uniform distribution, a n ∼ u(a a (s)). formally, we uniformly sample @xmath0, 1] per every step, this collects diverse experiences from altering the world with admissible actions. we also tried a variant where the ϵ is selected adaptively given the game score the agent has achieved. however, this variant under-performed the static ϵ. see appendix i for more details on this and the results. ## experiments in this section, we provide a description of our experimental details and discuss the results. we selected a wide variety of agents (introduced in section 3) utilizing the lm or the kg: calm-drrn @xcite and kg-a2c (ammanabrolu and hausknecht, 2020) as baselines, and sha-kg @xcite , q*bert @xcite , hex-rl and hex-rl-im (peng et al., 2021) as state-of-the-art (sota). experimental setup. similar to kg-a2c, we train our agent on 32 parallel environments with 5 random seeds. we trained tac on games of jericho suite with 100k steps and evaluated with 10 episodes per every 500 training step. during the training, tac uses uniformly sampled admissible action for a probability of ϵ and during the testing, it follows its policy distribution generated from the game observations. we used prioritized experience replay (per) as our replay buffer @xcite . we first fine-tune tac on zork1, then apply the same hyper-parameters for all the games. the details of our hyper-parameters can be found in appendix a. our final score is computed as the average of 30 episodic testing game scores. additionally, our model has a parameter size of less than 2m, allowing us to run the majority of our experiments on cpu (intel xeon gold 6150 2.70 ghz). the full parameter size in zork1 and the training time comparison can be found in appendices b and c. 0 50000 100000 training step 5 10 15 20 score ludicorp 0 50000 100000 training step 4 6 8 score temple tac w/ 1-1 tac w/ 5-3 tac w/ 10-5 tac w/ 15-7 tac w/ 20-10 calm-drrn kg-a2c 0 50000 100000 training step 0 5 10 score reverb 0 50000 100000 training step 0 10 20 30 score zork1 figure 4: the learning curve of tac for stronger supervised signals where 5-3 signifies λ @xmath1. left two plots are with ϵ = 0.3 and right two are with ϵ = 0. ## conclusion text-based games (tgs) offer a unique framework for developing rl agents for goal-driven and contextually-aware natural language generation tasks. in this paper we took a fresh approach in utilizing the information from the tg environment, and in particular the admissibility of actions during the exploration phase of rl agent. we introduced a language-based actor critic method (tac) with a simple ϵ-admissible exploration. the core of our algorithm is the utilization of admissible actions in training phase to guide the agent exploration towards collecting more informed experiences. compared to state-of-the-art approaches with more complex design, our light tac design achieves substantially higher game scores across 10-29 games. we provided insights into the role of action admissibility and supervision signals during training and the implications at test phase for an rl agent. our analysis showed that supervised signals towards admissible actions act as guideline in the absence of reward signal, while serving a regularization role in the presence of such signal. we demonstrated that reasonable ϵ probability threshold is required for high quality unbiased experience collection during the exploration phase.