A Sequence Modelling Approach to Question Answering in Text-Based Games (2022)

Abstract: Interactive Question Answering (IQA) requires an intelligent agent to interact with a dynamic environment in order to gather information necessary to answer a question. IQA tasks have been proposed as means of training systems to develop language or visual comprehension abilities. To this end, the Question Answering with Interactive Text (QAit) task was created to produce and benchmark interactive agents capable of seeking information and answering questions in unseen environments. While prior work has exclusively focused on IQA as a reinforcement learning problem, such methods suffer from low sample efficiency and poor accuracy in zero-shot evaluation. In this paper, we propose the use of the recently proposed Decision Transformer architecture to provide improvements upon prior baselines. By utilising a causally masked GPT-2 Transformer for command generation and a BERT model for question answer prediction, we show that the Decision Transformer achieves performance greater than or equal to current state-of-the-art RL baselines on the QAit task in a sample efficient manner. In addition, these results are achievable by training on sub-optimal random trajectories, therefore not requiring the use of online agents to gather data.

## introduction traditional methods for question answering (qa) and machine reading comprehension (mrc) are primarily concerned with the retrieval of declarative knowledge, that is, explicitly stated or static descriptions of entities in text documents or within a knowledge base (kb) @xcite . these models tend to answer questions primarily through basic pattern matching skills, further differentiating their abilities from those of humans. conversely, procedural knowledge is the sequence of actions required to perform a task @xcite . to this end, interactive question answering (iqa) has been proposed as a framework for teaching mrc systems to gather the information necessary for question answering @xcite . iqa requires an agent to interact with some dynamic environment in order to gather the required knowledge to answer a question @xcite . as such, the task is well-suited to be approached as a reinforcement learning (rl) problem. @xcite proposed question answering using interactive text (qait) as a means of testing the knowledge gathering capabilities of an agent required to answer a question about its environment. here an agent interacts with a partially observable text-based environment, created using microsoft textworld @xcite , in order to gather information and answer questions about the attributes, location, and existence of objects. the qait task thus aims to benchmark generalisation and provides an environment to train agents capable of gathering information and answering questions. yuan et al.'s proposed baselines (using dqn @xcite , ddqn @xcite , and rainbow @xcite ) all suffered from low sample efficiency and relatively poor performance on all three question types (location, attribute, and existence). these shortcomings suggest that alternative architectures and methodologies are required to improve performance within the qait setting. transformers @xcite have shown success in modelling a diverse range of highdimensional problems @xcite @xcite . additionally, existing language models such as bert (bidirectional encoder representations from transformers) and gpt (generative pre-trained) @xcite have been utilised to reduce the size of datasets required for training downstream language tasks @xcite . these benefits coupled with the demon-strated ability of transformers to model long sequences by utilising the self-attention mechanism makes this architecture ideal for iqa. recent work @xcite have shown the applicability of transformers to sequential decision making problems as an alternative solution to rl problems. these approaches frame rl trajectories as sequences of states, actions, and rewards modelled autoregressively by a transformer. this sequence modelling approach is referred to as the decision transformer (dt) @xcite . in this paper we apply the decision transformer to qait, replacing the online interaction and training methodology of rl approaches with a decision transformer that utilises the gpt-2 @xcite architecture, closely following the methodology outlined by @xcite . we propose an additional qa module that is a fine-tuned bert model, with the aim of leveraging pre-trained language models to provide more accurate answers to questions. we show that by framing the qait task as a sequence modelling problem, a decision transformer matches or exceeds the performance of previous rl-based benchmarks when trained on random episodic rollouts, while using significantly less data. our main contributions are as follows: ## approach training a decision transformer requires offline training data for supervised learning. online reinforcement learning, in contrast, sees an agent continually interacting with the environment to gather experience and update its policy based on observed rewards. ## results we evaluate the decision transformer both where its own answer prediction head is used for questions answering and where this is done with the bert qa model. test set results on the 500 games setting are given in overall, dt-bert outperforms the decision transformer's answer prediction head in location and attribute type questions, while the dt gives a higher accuracy on existence questions. at a high level, these performance differences depend on the state and action space that the model was required to learn and navigate. existence type and attribute type questions may depend on long-range dependencies. for example, existence type questions require the ability to know whether an object has been witnessed or not. when the answer is that an object doesn't exist, the problem is more than just word matching within the last few states. we believe that this is why the decision transformer qa head outperforms the bert model on existence and attribute questions since it has access to the entire state trajectory. on questions whose answers were more likely to be found within the last 512 tokens, dt-bert achieves higher question answering accuracy than the decision transformer. ## conclusion we showed that interactive question answering can be framed as a sequence modelling problem by training transformers for action generation and answer prediction using random roll-outs. results show that the decision transformer approach matches or outperforms current reinforcement learning approaches for qait on most question types and maps type configurations in the 500 game setting. additionally, the approach is more sample efficient than reinforcement learning approaches, reducing the amount of training data required even though the data generated via random rollouts is suboptimal. fine-tuning a bert model for question answering on the same generated dataset improves performance over using the decision transformer directly for question answering in two of the three question types.