Policy-based Reinforcement Learning for Generalisation in Interactive Text-based Environments (2023)

Abstract: Text-based environments enable RL agents to learn to converse and perform interactive tasks through natural language. However, previous RL approaches applied to text-based environments show poor performance when evaluated on unseen games. This paper investigates the improvement of generalisation performance through the simple switch from a value-based update method to a policy-based one, within text-based environments. We show that by replacing commonly used value-based methods with REINFORCE with baseline, a far more general agent is produced. The policy-based agent is evaluated on Coin Collector and Question Answering with interactive text (QAit), two text-based environments designed to test zero-shot performance. We see substantial improvements on a variety of zero-shot evaluation experiments, including tripling accuracy on various QAit benchmark configurations. The results indicate that policy-based RL has significantly better generalisation capabilities than value-based methods within such text-based environments, suggesting that RL agents could be applied to more complex natural language environments.

## introduction general domain language comprehension is an important component of the goal to create artificial intelligence agents that can perform real-world activities @xcite . in the pursuit of such systems, classic text-based games are often utilised as a bridge to the generality of the real world @xcite . text-based games are a useful test-bed for exploring interactive dialogue agents in the context of partially observable markov decision processes (pomdps) that have large combinatorial action spaces. additionally, they provide environments that depend on the comprehension and use of natural language. in recent years, there has been a shift in focus from simply performing well in a single text-based environment @xcite to generalising in unseen environments @xcite @xcite . this shift in focus has often relied on complex methods and bespoke solutions @xcite @xcite . a great deal of this research has utilised value-based reinforcement learning (rl) methods which often show high training performance but poor zero-shot generalisation. these methods have been shown to overfit and memorise the training environments, thereby not learning a generalisable policy @xcite . previous work has shown that policy-based methods can work well in non text-based settings where generalization is required @xcite . however, to the best of our knowledge no controlled comparison has been made between the generalisability of value-based and policy-based rl methods in text-based environments. this leads us to investigate policy-based rl in text-based environments. we hypothesise that learning an accurate and general value function in a partially observable environment with a large action space (such as natural language) is too difficult for most value-based methods. we aim to show that policy-based approaches solve this generalisation issue by directly learning a stochastic and general policy. in this paper, we use two text-based environments, coin collector @xcite and qait @xcite . both environments utilise textworld @xcite to generate environments on the fly. agents must acquire natural language skills in order to navigate successfully within the worlds and achieve some specified goal. we compare a policy-based method, trained using the reinforce with baseline algorithm @xcite against coin collector and qait's valuebased approaches on the environments' respective test sets. for comparison, we make use of both coin collector and qait's open-sourced codebases with a simple replacement of the value-based update and action selection method. our results show that the policy-based method leads to substantially improved performance over previous value-based methods on a variety of zeroshot evaluations. in the coin collector task, we see greater average performance and stability on the easy and medium test sets (after minimal hyperparameter tuning). on the qait benchmark, large accuracy improvements are obtained for location questions (28% to 98% for fixed maps) and existence questions (69.2% to 94.8% for fixed maps). the results indicate that policy-based rl has significantly better generalisation capabilities than valuebased methods within these contexts, and are more suited to text-based environments. this paper's main contributions are as follows: ## policy-based text-based agents we propose a policy-based agent that makes only minor changes to previous value-based approaches. for both environments, the neural network architec-ture is kept largely unmodified, in order to focus on comparing the generalisation performance of the value-based baselines and the policy-based method (reinforce with baseline). the only changes are the use of a policy-based update method, an additional mlp to predict state values (to use as b(s) -the baseline function), and the conditioning of future command words on previous command words (shown in figure 2 ). for most value-based methods, the q-values of actions are learnt. this means that the mlp used to predict q-values already has the necessary number of output nodes for a policy-based method. each output layer represents the policy for each word in the command tuple, e.g. (action, modifier, object) for qait and (verb, noun) for coin collector. each output gives a probability distribution over the vocabulary. each word in the command tuple is then sampled from these probability distributions to form the command at each game step. this more closely emulates how a human would speak compared to value-based methods, which traditionally act using greedy action selection at evaluation time (which is deterministic) and epsilon-greedy action selection at training time (which introduces some stochasticity) @xcite . the baseline function approximates the statevalue function to aid in the training of the policy. the baseline network also uses the policy's shared linear layer as input to produce the state value v (s). this is done to regularise the shared linear layer to a common representation to aid the policy and value function in generalisation @xcite . the reinforce with baseline algorithm is used to update model weights. each policy's entropy (for each word in the command tuple) is subtracted from the total loss. this incentivises more stochastic policies to be learnt whilst maximising reward, in order to promote exploration and generalisation. the loss functions used for the policies l a and baseline l c are: the command word conditioning is done as follows: each consecutive word in the command tuple receives the vector of raw outputs from the previous word, concatenated with the shared linear output. a small ablation of the effects of command conditioning indicated only a slight performance increase. conditioning on consecutive words did not lead to any performance gains on the value-based baselines. ## coin collector coin collector @xcite is a text-based deterministic version of the chain experiment @xcite that introduces off-chain nodes (distractor rooms) to distract and confuse the agent. ## qait the question answering with interactive text (qait) @xcite task is a text-based question answering problem in which an agent must interact with a partially observable text-based environment to gather the declarative knowledge required to answer questions. qait aims to test an agent's language comprehension abilities by posing questions about the location, existence and attributes of objects distributed throughout the environment. ## conclusion this work demonstrates the advantages of using policy-based methods in textual environments. more specifically, we investigated the differences in generalisation performance of the reinforce with baseline algorithm compared to value-based rl baselines on two sets of text-based tasks. the results strongly suggest that policy-based rl methods are not only more suited for textual domains due to their training performance, but also possess generalisation capabilities beyond their valuebased counterparts.