An Overview of Natural Language State Representation for Reinforcement Learning
Abstract
A suitable state representation is a fundamental part of the learning process in Reinforcement
Learning.
In various tasks, the state can either be described by natural language or be natural language itself.
This survey outlines the strategies used in the literature to build natural language state representations.
We appeal for more linguistically interpretable and grounded representations, careful justiﬁcation of design decisions and evaluation of the effectiveness of different approaches.
1. Introduction
In any typical Reinforcement Learning (RL) scenario, there is an agent in an environment following a policy to take actions which make it transition through states and receive rewards (Sutton & Barto, 1998).
Each of these elements must be modeled according to the purpose of the task and their representation can inﬂuence the outcome of RL algorithms.
Natural Language Understanding can be integrated into these components, both in language-conditional RL (when natural language is inherent to the task) and in languageassisted RL (when natural language is an additional tool), as discussed in Luketina et al.
The reward function is usually under the spotlight because the agent’s goal is maximizing the expected long-term reward.
But the state representation is no less vital.
Based on it, the agent perceives the environment and comes to decisions on how to act.
While in robotics the agent observes a spatial environment and in arcade games the state may be composed of a sequence of images, natural language is as well a common part of RL states, as illustrated in Figure 1.
The choice of state representation is a problem on its own.
It directly affects the learning process (Jones & Canas, 2010), so if applications neglect this, the agent may be prevented from accessing key information for decision-making. to the
Workshop
Language in Reinforcement Learning , ICML 2020. Copyright 2020 by author( [Figure]
There is ongoing research on the use of natural language to model the reward and the actions (He et al., 2016;
Feng et al., 2018; Goyal et al., 2019b), but we are not aware of any recent systematic survey about the various possibilities of using natural language to model the state representation and how to do that effectively.
We aim at ﬁlling this gap by providing an overview of previous work in which the state is based on text, delineating the main approaches.
We dive into a wide range of NLP papers that apply RL methods and whose state representations have to capture linguistic features that inﬂuence decision-making.
Findings about state representations in this area may potentially be extrapolated to other language-informed RL tasks.
We thus hope this overview aids those seeking the objective of having RL agents capable of understanding natural language.
Since we notice there is no consensus on how to design natural language state representations, we conclude with some concrete recommendations for future research.
2. State Representation in RL
The construction of suitable state representations for RL has been assessed by several works, for example in the ﬁeld of robotics and autonomous cars, where modeling the state is particularly hard due to very high dimensional data coming from multiple sensors (Jonschkowski & Brock, 2013;
De Bruin et al., 2018).
Lesort et al.
(2018), for instance, presented a review on State Representation Learning algorithms and evaluation methods for control scenarios.
By deﬁnition, the state in a Markov Decision Process, which underlies the formalization of RL, needs to be Markovian; i.e. all that needs to be known about the past must be contained in the current state.
Jonschkowski & Brock (2013) summarized other desirable properties of state representations proposed by many authors: It should provide good features for learning the value function; be compact but still allowing the original observations to be reconstructed; change slowly over time; be useful for predicting future observations and rewards given future actions; and ideally be shared by various similar tasks.
Although their focus was robotics, these properties can be extended to textual states.
Many other aspects of state representation have been investigated.
In early works, states were handcrafted by the system designer using features.
Although it remains a common practice, feature engineering has evident drawbacks: it needs an experienced designer, is tedious and time-consuming and does not generalize (B¨ohmer et al., 2015).
The popularization of deep learning can to some extent spare us part of these shortcomings, as raw data can potentially be used as input to end-to-end models that build state representations implicitly.
De Bruin et al.
(2018), for instance, analyzed methods for integrating state representation learning into deep RL.
Still, choosing which information, data structure and model to use can be regarded as a modern form of feature engineering.
Several options have been applied to textual data, as we describe in the next section.
3. State Representation in NLP tasks
When RL methods are adopted to solve NLP tasks, one common characteristic is that the state S is represented as a function of texts T , i.e. S =
f ( T ) .
As we will see, f may take a variety of forms and additional inputs.
We begin by reviewing the development of natural language state representation in dialogue, instruction following, text-based games and text summarization, NLP areas in which RL methods have been used more often.
With the consolidation of deep RL methods, more studies tried to solve several other tasks, which we aggregate in the last part.
Dialogue is likely the NLP area with the most substantial number of papers that adopt RL, with research going on for more than two decades, e.g. Levin et al.
In dialogue, the state representation is usually a mapping from text to an abstract or compact format, such as a template, a belief state or an embedding vector, used by the agent to generate the next utterance.
A Natural Language Understanding component between the text and the state representation is common, as illustrated, for example, in Lipton et al.
Delineating the state space was at ﬁrst mostly a manual task, because considering the entire dialogue used to be impractical (Singh et al., 2000a).
Keeping the state space small was a common concern in order to avoid the curse of dimensionality (Levin et al., 2000; Singh et al., 2000a).
Therefore, sys- tem designers picked a set of variables or features based on their experience, usually resulting in application-dependent representations that did not generalize (Walker, 2000; Levin et al., 2000; Frampton & Lemon, 2005; English & Heeman, 2005; Mitchell et al., 2013; Khouzaimi et al., 2015).
The slots and values of the dialogue state were used to represent the environment state (Papangelis, 2012) or the Information States approach was incorporated (Georgila et al., 2005;
Henderson et al., 2005; Heeman et al., 2012).
It was soon clear that modeling the state space is a fundamental aspect of RL for dialogue, as it has direct impact on the dynamics of the system, and attention was drawn to the fact that the research community had neither established best practices for modeling the state nor agreed upon domain-independent variables (Paek, 2006).
Some studies tried to examine which state representation was more effective for the main task (Schefﬂer & Young, 2002), constructing them with different feature combinations.
There was also discussion on ensuring the Markovian property (Singh et al., 2000b), which means that the “representation must encode everything that the system observed about everything that has happened in the dialogue so far” (Walker, 2000).
Ablation studies and metrics to evaluate the effect of each feature were proposed (Frampton &
Lemon, 2005; Tetreault & Litman, 2008; Heeman, 2009;
Mitchell et al., 2013).
Once neural network models started to be employed, new ways to represent states and to integrate representation learning into the agent’s learning were enabled.
States could be more easily represented by a vector and fed directly into a parametrized value or policy function.
A common approach was building a belief state (a distribution over possible dialogue states, usually represented by a ﬁxed set of slot-value pairs) when dialogue tasks are modeled as POMDPs (Su et al., 2016; Fatemi et al., 2016; Wen et al., 2017b; Weisz et al., 2018).
In Wen et al.
(2017b), a sequence of free form text was mapped into a ﬁxed set of slot-value pairs by an
RNN.
Dhingra et al.
(2017) compared a handcrafted and a neural belief tracker that uses a GRU over turns.
The belief state was sometimes combined with other representations.
Wen et al.
(2017a) modeled the dialogue state vector as a concatenation of user input (encoded by a BiLSTM), a belief vector (probability distributions over domain speciﬁc slot-value pairs, extracted by RNN-CNN belief trackers) and the degree of matching in a knowledge base.
Note that, even when the designer does not have to build the representation manually, the selection of slots and value ranges, the choice of input features and the NLU representations still require human intervention.
Features can still be used as input to deep learning models.
Williams & Zweig (2016) represented the dialogue history with features (such as input and output entities) as input to an LSTM, which in- ferred the state representation and mapped the input directly to actions.
The NLU component played a key role in the state representation e.g. in Manuvinakurike et al.
Cuay´ahuitl et al.
(2016) pointed out that it is typically unclear what features to incorporate in a multi-domain dialogue and proposed applying deep RL directly to texts, so that the agent learns feature representation and the policy together, bypassing NLU components.
Similarly, Li et al.
(2016) used the two previous dialogue turns, transformed into a continuous vector representation by an LSTM encoder, to build representations directly from raw text.
A common strategy has been to use the hidden vector of RNNs to represent and update the environment state (Zhao & Eskenazi, 2016; Williams et al., 2017; Liu & Lane, 2017).
Multimodal state representation has also become frequent, combining encoded text with images (Das et al., 2017), knowledge base queries (Li et al., 2017; Liu & Lane, 2017) or an embedding of a knowledge graph (Yang et al., 2020).
In Instruction Following , deciding the next action means directly interpreting a textual instruction.
In this task, states were ﬁrst designed as tuples of world and linguistic features: words and documents (Branavan et al., 2009; 2010) or cardinal directions and utterances (Vogel & Jurafsky, 2010).
We then observed the use of sentence embeddings (usually by an LSTM or GRU) combined with other sources such as image embeddings (Misra et al., 2017; Hermann et al., 2017; Kaplan et al., 2017; Fu et al., 2019) or instruction memory (Oh et al., 2017). A state processing module was introduced in Chaplot et al. (2018) to create a joint representation of the image and the instruction. Janner et al.
(2018) converted the instruction text into a real-valued vector and used it both to build a global map-level representation and as a kernel in a convolution operation to obtain a local representation of the state. In the related vision-language navigation task, the state was modeled by combining visually grounded textual context and textually grounded visual context (Wang et al., 2019; Wang et al., 2020) or by using an attention mechanism over the representations of the instruction (Anderson et al., 2018). Text-Based Games pose a related, but more general challenge, as here the text describing the current game state must be integrated into the agent state and the best action must be inferred.
The work of Narasimhan et al.
(2015) was a reference for modeling the state representation in text-based games, which is jointly learned together with action policies in their setting.
They used an LSTM over textual data with a mean pooling layer on top, in an attempt to capture the semantics of the game states.
Subsequent works followed the same approach (Ansari et al., 2018; Yuan et al., 2018; Jain et al., 2020) or used variants of continuous vectors built by neural networks (He et al., 2016; Foerster et al., 2016).
Other elaborate ideas appeared subsequently.
Narasimhan et al.
(2018) used a factorized state representation, concatenating an object embedding with its textual speciﬁcation embedding (LSTM or bag-of-words).
Ammanabrolu & Riedl (2019) combined the embedded text (by a sliding BiLSTM) with an embedded knowledge graph built by the agent throughout the game.
Their concatenation served as input to an MLP that outputted state representations.
Murugesan et al.
(2020) used embeddings of a local belief graph and a global common sense graph of entities.
Zhong et al.
(2020) proposed building representations that capture interactions between the goal, a document describing environment dynamics, and environment observations.
Text Summarization is another ﬂourishing area for RL.
In this task, an agent processes a text and either pick key sentences to compose a summary or use Natural Language
Generation to output its own words.
Initial works applying
RL used tuples of features to represent the state, composed of the summary at each time step, a history of actions and a binary variable indicating the terminal state (Ryang &
Abekawa, 2012; Rioux et al., 2014; Henß et al., 2015).
RL was really consolidated for text summarization after deep learning methods became available.
In most cases, the state representation was, as in other tasks, the hidden state of an RNN generating the summary (Ling & Rush, 2017;
Pasunuru & Bansal, 2018), also together with an encoding of a candidate sentence (Lee & Lee, 2017).
Paulus et al.
(2018), a reference for subsequent works (Kry´sci´nski et al., 2018), used context vectors with intra-temporal attention over the hidden states of the encoder (which process the original text) concatenated to the hidden state of the decoder (which generates the summary).
Hierarchical approaches are common.
Wu & Hu (2018) built a document encoding with a CNN operating at word level and a BiGRU at sentence level, whereas Narayan et al.
(2018) and Chen & Bansal (2018) used a CNN at sentence level and an LSTM or BiLSTM at document level to capture global information.
Likewise, Yao et al.
(2018) employed an RNN or CNN sentence encoder, together with a representation of the current summary and the document content.
Other NLP Tasks comprise each a currently smaller number of studies using RL, so we group them here into main general choices of state design.
Some tasks used ordered words (Li et al., 2018, paraphrase generation), predicted words (Grissom II et al., 2014, simultaneous machine translation), a vocabulary set and a taxonomy (Mao et al., 2018, taxonomy induction) or a bag of sentences (Zeng et al., 2018, relation extraction).
Coreference resolution adopted partially formed coreference chains (Stoyanov & Eisner, 2012) or word embeddings and features (Clark & Manning, 2016).
For syntactic or semantic parsing, the parser conﬁguration was used as set of discrete variables describing the state of a parse structure (Zhang & Chan, 2009; Jiang et al., 2012; Lˆe & Fokkens, 2017), a concatenation of their representation built by an LSTM (Naseem et al., 2019) or a query for a knowledge base (Liang et al., 2017).
A usual strategy to represent the state was handcrafting vectors composed of selected variables or metrics, for example, similarity scores and number of words (Ling et al., 2017, text-based clinical diagnosis), conﬁdence scores, tdidf and one-hot encoding of matches (Narasimhan et al., 2016;
Taniguchi et al., 2018, information extraction), situational and linguistic information (Dethlefs & Cuay´ahuitl, 2011, NLG), entitites and relations (Godin et al., 2019, question answering), probability distribution over a set of objects (Hu et al., 2018, question selection) or values generated by a parser (Wang et al., 2018, math word problem).
There has been a myriad of creative efforts to build representations using neural networks, especially RNNs, as one can seamlessly regard the RNN as an agent and its hidden state as the environment state.
The internal state (hidden vector and/or cell vector in LSTMs) was set as the environment state in language modeling (Ranzato et al., 2015), image captioning (Rennie et al., 2017), math word problem (Huang et al., 2018) and NLG (Yasui et al., 2019), sometimes with attention over hidden states of a sequence in grammatical error correction (Sakaguchi et al., 2017) or a concatenation of hidden states further encoded by another neural network in sentence representation (Yogatama et al., 2017).
A Transformer’s hidden states (Vaswani et al., 2017) were also used in machine translation (Wu et al., 2018).
The encoding provided by the output layer was used for semantic parsing (Guu et al., 2017) and also with location-based attention for text anonymization (Mosallanezhad et al., 2019).
Hierarchical network representations combining word level and sentence level encodings were proposed, as in text classiﬁcation (Zhang et al., 2018).
Both the hidden state of the encoder and the decoder comprised the state in sentence simpliﬁcation (Zhang & Lapata, 2017).
CNNs representations combined with predictive marginals were used in active learning for NER (Fang et al., 2017).
4. Concluding Discussion
The proliﬁc literature combining NLP and RL reveals a noticeable synergy between them.
While RL methods have extended frontiers of NLP research, natural language can be a valuable source of information for RL agents, in particular to model the state signal.
An abundance of tailored natural language state representations has been explored and there is a current trend to opt for end-to-end models, encoding or decoding text with the aid of RNNs and for multimodal or hierarchical representations, as shown in Figure 2.
More insight is needed on how natural language state rep-
[Figure]
resentations should be put together. Unfortunately, it is not uncommon to ﬁnd papers with unclear or missing deﬁnitions of their state signal or without much justiﬁcation of their choices, which hinders comparison and reproducibility.
We thus encourage researchers to carefully describe the state representation, provide justiﬁcation for state design modeling and to also report attempts that had negative impact on the agent’s learning process.
Studies to validate the effectiveness of NL state representation used to be common when engineering features was a trend.
Deep learning methods may give us the illusion that almost no human intervention or manual feature selection takes place, but they still play a relevant role in state signal design in terms of choice of architecture, model and input data.
Adjusting evaluation and ablation methods to new approaches is thus necessary.
Goyal et al. (2019a), for instance, experiment with three different natural language instructions representations, to test the effect of language-based reward.
Similar analyses can also be done with languagebased states, as in Narasimhan et al.
There is cutting-edge research being conducted about the interpretability of neural NLP models and their linguistic representational power (Belinkov & Glass, 2019).
Natural language state representation would thus proﬁt from studies about interpretability and also from diagnostic research (Hupkes et al., 2018) on their abilities of distilling, composing and retaining semantic information throughout the agent’s steps, usually expressed by recurrence in the neural networks.
In addition, grounding the meaning of texts to the dynamics of the environment, as discussed in Narasimhan et al.
(2018), is a promising area of future research.
The authors delineate two needs that must be met in state design with language grounding: the representation should fuse different modalities and capture the compositional nature of language in order to map semantics to the agent’s world.
Finally, Narasimhan et al.
(2015; 2018) discuss the beneﬁts of representations that are effective across different games and that enable policy transfer.
Therefore, another desirable property of natural language state representation is crossdomain validity, so that their encoding of world knowledge can be exploited in various RL scenarios. Acknowledgments
We thank the two anonymous reviewers for their feedback and suggestions.