C ASE - BASED R EASONING FOR B ETTER G ENERALIZATION IN T EXT -B ASED G AMES
A BSTRACT
Text-based games (TBG) have emerged as promising environments for driving research in grounded language understanding and studying problems like generalization and sample efﬁciency.
Several deep reinforcement learning (RL) methods with varying architectures and learning schemes have been proposed for
TBGs
However, these methods fail to generalize efﬁciently especially under distributional shifts
In a departure from deep RL approaches , in this paper, we propose a general method inspired by case-based reasoning to train agents and generalize out of the training distribution.
The case-based reasoner collects instances of positive experiences from the agent’s interaction with the world in the past and later reuses the collected experiences to act efﬁciently.
The method can be applied in conjunction with any existing on-policy neural agent in the literature for
TBGs.
Our experiments show that the proposed approach consistently improves existing methods, obtains good out-of-distribution generalization and achieves new state-of-the-art results on widely used environments
I NTRODUCTION
Text-based games (TBGs) have emerged as key benchmarks for studying how reinforcement learning (RL) agents can tackle the challenges of grounded language understanding, partial observability, large action spaces, and out-of-distribution generalization (Hausknecht et al., 2020;
Ammanabrolu & Riedl, 2019).
While we have indeed made some progress on these fronts in recent years (Ammanabrolu & Hausknecht, 2020; Adhikari et al., 2020; Murugesan et al., 2021b;a), these agents are still very inefﬁcient and suffer from insufﬁcient generalization to novel environments.
As an example, state-of-the-art agents require 5 to 10 times more steps than a human to accomplish even simple household tasks (Murugesan et al., 2021b).
As the agents are purely neural architectures requiring signiﬁcant training experience and computation, they fail to efﬁciently adapt to new environments and use their past experiences to reason in novel situations.
This is in stark contrast to human learning which is much more robust, efﬁcient and generalizable (Lake et al., 2017).
Motivated by this fundamental difference in learning, we propose new agents that rely on case-based reasoning (CBR) (Aamodt & Plaza, 1994) to efﬁciently act in the world.
CBR draws its foundations in cognitive science (Schank, 1983; Kolodner, 1983) and mimics the process of solving new tasks based on solutions to previously encountered similar tasks.
Concretely, we design a general CBR framework that enables an agent to collect instances of past situations which led to a positive reward (known as cases).
During decision making, the agent retrieves the case most similar to the current situation and then applies it after appropriately mapping it to the current context.
The CBR agent stores past experiences, along with the actions it performed, in a case memory.
In order to efﬁciently use these stored experiences, the agent should be able to represent relevant contextual information from the state of the game in a compact way, while retaining the property
[Figure]
that contexts that require similar actions receive similar representations.
We represent the state of the game as a knowledge graph (Ammanabrolu & Hausknecht, 2020) and we address these challenges by utilizing (a) a seeded message propagation that focuses only on a subset of relevant nodes and (b) vector quantization (Ballard, 2000) to efﬁciently map similar contexts to similar discrete representations.
Vector quantization allows the model to signiﬁcantly compress the context representations while retaining their semantics; thereby, allowing for a scalable implementation of
CBR in RL settings.
An illustration of the framework is shown in Figure
[Figure]
Our experiments show that CBR can be used to consistently boost the performance of various onpolicy RL agents proposed in the literature for TBGs.
We obtain a new state-of-the-art on the
TextWorld Commonsense (Murugesan et al., 2021b) dataset and we achieve better or comparable scores on 24 of the 33 games in the Jericho suite (Hausknecht et al., 2020)
compared to previous work.
We also show that CBR agents are resilient to domain shifts and suffer only marginal drops in performance (6%) on out-of-distribution settings when compared to their counterparts (35%) .
P RELIMINARIES
Text-based games.
Text-based games (TBGs) provide a challenging environment where an agent can observe the current state of the game and act in the world using only the modality of text.
The state of the game is hidden, so TBGs can be modeled as a Partially Observable Markov Decision
Process (POMDP) ( S , A , O , T , E , r ) , where S is the set of states of the environment of the game, A is the natural language action space, O is the set of observations or sequences of words describing the current state, T are the conditional transition probabilities from one state to another, E are the conditional observation probabilities, r : S ×A → R is the reward function, which maps a state and action to a scalar reward that the agent receives.
Case-based reasoning.
Case-based reasoning (CBR) is the process of solving new problems based on the solution of previously seen similar problems.
Generally, CBR assumes access to a memory that stores past problems (known as cases) and their solutions.
When a new problem is encountered,
CBR will (i) retrieve a similar problem and its solution from memory; (ii) reuse the solution by mapping it to the current problem; (iii) revise the solution by testing it and checking whether it is a viable way to address the new problem; and (iv) retain the solution in memory if the adaptation to the new problem was successful.
C ASE - BASED REASONING IN REINFORCEMENT LEARNING
This section introduces our framework inspired by CBR for improving generalization in TBGs.
Even though we situate our work in TBGs, it serves as a good starting point for applying CBR in more general RL settings.
We consider an on-policy RL agent that, at any given time step t , has access to a memory M t , that can be used to retrieve previous experiences.
The memory contains key-value pairs, where the keys are a context representation of a game state and values are actions that were taken by the agent w.r.t to this context.
As mentioned in Section 2, case-based reasoning can be formalized as a four-step process.
We describe our proposed methodology for each step below.
Algorithm 1 provides a detailed formalization of our approach.
Let C t
= { context ( s t , a t ) | a t ∈
A t } be a set of context selectors for state s t at time step t
A M t ← ∅ for c t ∈ C t do
Let ( c M t , a M t ) =
arg max ( c M t ,a M t ) ∈M t sim ( c t , c M t )
Let δ = sim ( c t , c M t )
if δ > τ then
A M t ← A M t ∪ { ( a M t , δ ) } end end
Build a set of action candidates:
Revise if A t ∩ ˜
A t (cid:54) = ∅ then
Let a (cid:63)t , δ (cid:63) = arg max ˜ a t ,δ ∈ ˜
A t δ else a (cid:63)t = arg max a t ∈A t π ( a t | s t ) end
Let r t = r
( s t , a (cid:63)t ) be the reward obtained at time step t by executing action a (cid:63)t • Retain
Let c (cid:63)t = context ( s t , a (cid:63)t ) be the context of action a (cid:63)t
T = { ( c (cid:63)t , a (cid:63)t ) , . . .
, ( c (cid:63)t − m +1 , a (cid:63)t − m +1 ) } if r t > 0 then
M t +1 ← M t ∪ retain ( T ) end
Retrieve.
Given the state of the game s t and the valid actions A t , we want to retrieve from the memory M t previous experiences that might be useful in decision-making at the current state.
To this end, for each admissible action a t ∈
A t , we deﬁne a context selector c t = context ( s t , a t ) .
The context selector is an action-speciﬁc representation of the state, namely the portion of the state that is relevant to the execution of an action.
We will explain later how the context selector is deﬁned in our implementation.
For each context c t , we retrieve from the memory the context-action pair ( c M t , a M t ) , such that c M t has maximum similarity with c t .
We denote as δ = sim ( c t , c M t ) ∈ [0 , 1] the relevance score given to the retrieved action.
Only actions a M t with a relevance score above a retriever threshold τ are retrieved from M t .
We denote as A M t the ﬁnal set of action-relevance pairs returned by the retriever, as shown in Algorithm 1.
Reuse.
The goal of the reuse step is to adapt the actions retrieved from the memory based on the current state.
This is accomplished by a reuse function, that is applied to each retrieved action to construct a set ˜
A t of candidate actions that should be applicable to the current state, each paired with a conﬁdence level.
Revise.
If any of the action candidates ˜
A t is a valid action, then the one with the highest relevance δ is executed, otherwise a neural agent π is used to select the best action a (cid:63)t .
We denote with r t = r
( s t , a (cid:63)t ) the obtained reward.
Note that π can be an existing agent for TBGs (Murugesan et al.
, 2021c;b; Ammanabrolu &
Hausknecht, 2020).
Retain.
Finally, the retain step stores successful experiences as new cases in the memory, so that they can be retrieved in the future.
In principle, this can be accomplished by storing actions for which the agent obtained positive rewards.
However, we found that storing previous actions as well can result in improved performance.
Therefore, whenever r t > 0 , a retain function is used to select which of the past executed actions and their contexts should be stored in the memory.
In our experiments, the retain function selects the k most recent actions, but other implementations are possible, as discussed in Appendix D.
A CBR POLICY AGENT TO GENERALIZE IN TEXT - BASED GAMES
Designing an agent that can act efﬁciently in TBGs using the described approach poses several challenges.
Above all, efﬁcient memory use is crucial to making the approach practical and scalable.
Since the context selectors are used as keys for accessing values in the memory, their representation needs to be such that contexts where similar actions were taken receive similar representations.
At the same time, as the state space is exponential, context representations need to be focused only on relevant portions of the state and they need to be compressed and compact.
4. R EPRESENTING THE CONTEXT THROUGH SEEDED GRAPH ATTENTION
State space as a knowledge graph.
Following previous works (Ammanabrolu & Riedl, 2019;
Ammanabrolu & Hausknecht, 2020; Murugesan et al., 2021c), we represent the state of the game as a dynamic knowledge graph G t
=
( V t , R t , E t ) , where a node v ∈ V t represents an entity in the game, r ∈ R t is a relation type, and an edge v r −→ v (cid:48)
∈
E t represents a relation of type r ∈ R t between entities v, v (cid:48) ∈ V t .
In TBGs, the space of valid actions A t can be modeled as a templatebased action space, where actions a t are instances of a ﬁnite set of templates with a given set of entities, denoted as V a t ⊆
V t .
As an example, the action “ kill orc with sword ” can be seen as an instance of the template “ kill v 1 with v 2 ”, where v 1 and v 2 are “ orc ” and “ sword ” respectively.
Seeded graph attention.
The state graph G t and the entities V a t are provided as input to the agent for each action a t ∈
A t , in order to build an action-speciﬁc contextualized representation of the state.
A pre-trained BERT model (Devlin et al., 2019) is used to get a representation h (0) v ∈ R d for each node v ∈ V t .
Inspired by Sun et al.
(2018), we propose a seeded graph attention mechanism (GAT), so that the propagation of messages is weighted more for nodes close to the entities V a t .
Let α ( l ) vu denote the attention coefﬁcients given by a graph attention network (Velickovic et al., 2018) at layer l for nodes v, u ∈ V t .
Then, for each node v ∈ V t , we introduce a coefﬁcient β ( l ) v that scales with the amount of messages received by node v at layer l : [Equation] =
[Equation]
if v ∈ V a t otherwise , β ( l +1) v = (
[Equation]
λ )
[Equation] ( l ) [Equation] + λ (cid:88) u ∈N v α ( l ) vu β ( l ) u , where N v denotes the neighbors of v , considering the graph as undirected.
Note that, at layer l = 1 , only the nodes in V a t receive messages, whereas for increasing values of l , β ( l ) v will be non-zero for their ( l − 1)
-hop neighbors as well.
The representation of each v ∈ V t is then updated as:
[Equation]
where FFN ( l +1) is a 2-layer feed-forward network with ReLU non-linearity and W ( l ) ∈
R d × d are learnable parameters.
Finally, we compute a ﬁnal continuous contextualized representation c a t of the state by summing the linear projections of the hidden representations of each v ∈ V a t and passing the result through a further feed-forward network.
4. M EMORY ACCESS THROUGH CONTEXT QUANTIZATION
Given a continuous representation c a t of the context, we need an efﬁcient way to access the memory
M t to retrieve or store actions based on such a context selector.
Storing and retrieving based on the continuous representation c a t would be impractical for scalability reasons.
Additionally, since the parameters of the agent change over the training time, the same context would result in several duplicated entries in the memory even with a pre-trained agent over different episodes.
Discretization of the context.
To address these problems, we propose to use vector quantization (Ballard, 2000) before reading or writing to memory.
Following previous work (Chen et al., 2018;
Sachan, 2020), we learn a discretization function φ : R d → Z DK , that maps the continuous representation c a t into a K -way D -dimensional code c t ∈ Z DK , with | Z K | =
K (we refer to c t as a KD code).
With reference to Section 3, then we will use c t = context ( s t , a t ) =
φ ( c a t ) as the context selector used to access the memory M t .
In order to implement the discretization function, we deﬁne a set of K key vectors k i ∈
R d , i = 1 , . . .
, K , and we divide each vector in D partitions k ji ∈ R d/D , j = 1 , . . .
, D .
Similarly, we divide c a t in D partitions c ja t ∈ R d/D , j = 1 , . . .
, D .
Then, we compute the j -th code
z
j of c t by nearest neighbor search, as z j = arg min i (cid:107) c
ja t − k ji (cid:107) 22 .
We use the straight-through estimator (Bengio et al., 2013) to address the non differentialbility of the argmin operator.
Memory access.
The KD codes introduced above are used to provide a memory-efﬁcient representation of the keys in the memory.
Then, given the KD code representing the current context selector c t , we query the memory by computing a similarity measure sim ( c t , c M t ) between c t and each c M t in M t .
The similarity function is deﬁned as the fraction of codes shared by c t and c M t .
The context-action pair with the highest similarity is returned as a result of the memory access, together with a relevance score δ representing the value of the similarity measure.
4. S YMBOLIC ACTION REUSE AND REVISE POLICY
We use a simple purely symbolic reuse function to adapt the actions retrieved from the memory to the current state.
Let c t be the context selector computed based on state s t and the entities V a t , as explained in Sections 4.1 and 4.2.
Denote with ( c M t , a M t ) the context-action pair retrieved from M t with conﬁdence δ .
Then, the reuse function reuse ( a M t , s t , δ ) constructs the action candidate ˜ a t as the action with the same template as a M t applied to the entities V a t .
If the reuse step cannot generate a valid action, we revert to the neural policy agent π that outputs a probability distribution over the current admissible actions A t .
T RAINING
In Section 3, we have introduced an on-policy RL agent that relies on case-based reasoning to act in the world efﬁciently.
This agent can be trained in principle using any online RL method.
This section discusses the training strategies and learning objectives used in our implementation.
Objective.
Two main portions of the model need to be trained: (a) the retriever , namely the neural network that computes the context representation and accesses the memory through its discretization, and (b) the main neural agent π which is used in the revise step.
All agents π used in our experiments are trained with an Advantage Actor-Critic (A2C) method.
For optimizing the parameters of π , we use the same learning objectives deﬁned by Adolphs & Hofmann (2019), as described in Appendix A. Whenever the executed action a (cid:63)t is not chosen by the model π
but it comes from the symbolic reuse step, then we optimize instead an additional objective for the retriever, namely the following contrastive loss (Hadsell et al., 2006):
[Equation]
where c t denotes the context selector of the action executed at time step t , c M t is the corresponding key entry retrieved from M t , µ is the margin parameter of the contrastive loss, and y t
= 1 if r t > 0 , y t = 0 otherwise.
This objective encourages the retriever to produce similar representations for two contexts where reusing an action yielded a positive reward.
Pretraining.
To make learning more stable and allow the agent to act more efﬁciently, we found it beneﬁcial to pretrain the retriever.
This minimizes large shifts in the context representations over the training time.
We run a baseline agent (Ammanabrolu & Hausknecht, 2020) to collect instances of the state graph and actions that yielded positive rewards.
Then we train the retriever to encode to similar representations the contexts for which similar actions (i.e., actions with the same template) were used.
This is achieved using the same contrastive loss deﬁned above.
E XPERIMENTS
This section provides a detailed evaluation of our approach.
We assess quantitatively the performance of CBR combined with existing RL approaches and we demonstrate its capability to improve sample efﬁciency and generalize out of the training distribution.
Next, we provide qualitative insights and examples of the behavior of the model and we perform an ablation study to understand the role played by the different components of the architecture.
6. E XPERIMENTAL SETUP
Agents.
We consider several agents obtained by plugging existing RL methods in the revise step.
We ﬁrst deﬁne two simple approaches: CBR-only , where we augment a random policy with the
[Table] [Table]
CBR approach, and Text + CBR , which relies on the CBR method combined with a simple GRUbased policy network that consumes as input the textual observation from the game.
Next, we select three recently proposed TBG approaches: Text+Commonsense ( TPC ) (Murugesan et al., 2021b),
KG-A2C (Ammanabrolu & Hausknecht, 2020), and BiKE (Murugesan et al., 2021c), to create the
TPC + CBR , KG-A2C + CBR and BiKE + CBR agents.
We treat the original agents as baselines.
Datasets.
We empirically verify the efﬁcacy of our approach on TextWorld Commonsense ( TWC ) (Murugesan et al., 2021b) and Jericho (Hausknecht et al., 2020).
Jericho is a well-known and challenging learning environment including 33 interactive ﬁction games.
TWC is an environment which builds on TextWorld (Côté et al., 2018) and provides a suite of games requiring commonsense knowledge.
TWC allows agents to be tested on two settings: the in-distribution games , where the objects that the agent encounters in the test set are the same as the objects in the training set, and the out-of-distribution games which have no entity in common with the training set.
For each of these settings, TWC provides three difﬁculty levels: easy , medium , and hard .
Evaluation metrics.
Following Murugesan et al.
(2021b), we evaluate the agents on TWC based on the number of steps ( #Steps ) required to achieve the goal (lower is better) and the normalized cumulative reward ( Norm.
Score ) obtained by the agent (larger is better).
On Jericho , we follow previous work (Hausknecht et al., 2020; Guo et al., 2020; Ammanabrolu & Hausknecht, 2020) and we report the average score achieved over the last 100 training episodes.
6. R ESULTS ON T EXT W ORLD C OMMONSENSE
Table 1 reports the results on TWC for the in-distribution set of games.
Overall, we observe that CBR consistently improves the performance of all the baselines.
The performance boost is large enough that even a simple method as Text + CBR outperforms all considered baselines except BiKE .
Out-of-distribution generalization.
CBR’s ability to retrieve similar cases should allow our method to better generalize to new and unseen problems.
We test this hypothesis on the out-of-
[Figure]
[Figure] [Figure] [Figure] o
[Figure]
li z ed
S c
o r e
[Figure]
r [Figure] N u
[Figure] r [Figure] f
[Figure] t ep [Figure] distribution games in TWC .
The results of this experiment are reported in Table 2.
We notice that all existing approaches fail to generalize out of the training distribution and suffer a substantial drop in performance in this setting.
However, when coupled with CBR, the drop is minor (on average 6% with CBR vs 35% without on the hard level).
Interestingly, even the CBR-only agent achieves competitive results compared to the top-performing baselines.
Sample efﬁciency.
Another key beneﬁt of our approach comes as better sample efﬁciency.
With its ability to explicitly store prior solutions effectively, CBR allows existing algorithms to learn faster.
Figure 2 shows the learning curves for our best agents and the corresponding baselines.
The plots report the performance of the agent over the training episodes, both in terms of the number of steps and the normalized score.
Overall, we observe that the CBR agents obtain faster convergence to their counterparts on all difﬁculty levels.
6. P ERFORMANCE ON THE J ERICHO GAMES
We evaluate our best performing variant from the experiments on TWC ( BiKE + CBR ) against existing approaches on the 33 games in the Jericho environment.
We compare our approach against strong baselines, including TDQN (Hausknecht et al., 2020), DRRN (He et al., 2016), KG-A2C (Ammanabrolu & Hausknecht, 2020), MPRC-DQN (Guo et al., 2020), and RC-DQN (Guo et al., 2020).
The same experimental setting and handicaps as the baselines are used, as we train for 100 000 steps and we assume access to valid actions.
Table 3 summarizes the results of the Jericho games.
We observe that our CBR agent achieves comparable or better performance than any baseline on 24 (73%) of the games, strictly outperforming all the other agents in 18 games.
6. Q UALITATIVE ANALYSIS AND ABLATION STUDIES
Insights on the model.
Figure 3 provides two examples showing the BiKE + CBR agent interacting with the zork1 game.
In the example on top, the agent retrieves an experience that can be successfully reused and turned into a valid action at the current time step.
The heat maps visualize the value of the context similarity function deﬁned in Section 4 for the top entries in the memory.
In the negative example at the bottom instead, the agent retrieves an action that is not useful and needs to fall back to the neural policy π .
Figure 4 (top) shows the fraction of times that actions retrieved from the memory are reused successfully in the TWC games.
We observe that, both for in-distribution and out-of-distribution games, the trained agent relies on CBR from 60% to approximately 70% of the times.
Figure 4 (bottom) further shows the fraction of times that the neural agent would have been able to select a rewarded action as well, when the CBR reuses a successful action.
The plot shows that, for the out-of-distribution games, the neural agent would struggle to select good actions when the CBR is used.
[Table]
[Table]
in case put jewels in case take diamond take emerald take jewels put diamond in case take jewlels take emerald take diamond go north take coal take gold put emerald in case take egg take diamond take emerald take jewels put jewels in case 0.0 0.
put emerald in case put jewels in case take diamond take emerald take jewels put diamond in case take jewlels take emerald take diamond go north take coal take gold put emerald in case take egg take diamond take emerald take jewels put jewels in case 0.0 0.
Take emerald
Take diamond
Put jewels in case
Observations & Actions > Take diamond […]
Living Room Above the trophy case hangs an elvish sword of great antiquity.
Memory
Put emerald in case
Action reuse : Put emerald in case
Put diamond in case put emerald in case put jewels in case take diamond take emerald take jewels put diamond in case take jewlels take emerald take diamond go north take coal take gold put emerald in case take egg take diamond take emerald take jewels put jewels in case 0.0 0. put emerald in case put jewels in case take diamond take emerald take jewels put diamond in case take jewlels take emerald take diamond go north take coal take gold put emerald in case take egg take diamond take emerald take jewels put jewels in case 0.0 0.
Take emerald
Go north
Take jewels
Observations & Actions > Climb tree […]
Beside you on the branch is a small bird's nest.
In the bird's nest is a large egg encrusted with precious jewels…
Memory
Go west
Action reuse : No valid action, fallback to neural policy agent is able to reuse a previous experience and in a case where the revise step is needed.
Non-rewarded action
OUT
IN
OUT
IN
OUT
IN
H a r d
M e d i u m
E asy
Action reuse
Neural Agent
OUT
IN
OUT
IN
OUT
IN
H a r d
M e d i u m
E asy
Action reuse
Neural Agent
OUT
IN
OUT
IN
OUT
H a r d
M e d i u m
E a
Action reuse
Neural Agent
OUT
IN
OUT
IN
OUT
H a r d
M e d i u m
Action reuse
Neural Agent
OUT
IN
OUT
IN
[Figure]
i
[Figure]
Action
[Figure]
Neural Agent
OUT
IN
OUT
IN
[Figure]
Action
[Figure]
Neural Agent
IN
OUT
IN
OUT
IN OUT [Figure] asy
[Figure]
OUT IN OUT IN
[Figure]
Agent
IN
OUT
IN
OUT
IN
OUT R
[Figure]
N
[Figure]
OUT
IN
OUT
IN
[Figure]
Ablation studies.
In order to understand the role of the main modules of our CBR agent, we designed some ablation studies.
First, instead of using the seeded GAT, we deﬁne the context of a state-action pair context ( s t , a t ) as just one of the entities that a t is applied to.
This deﬁnition suits well the TWC games because rewarded actions are always applied to one target object and a location for that object (see Appendix G for details).
Note that, since the set of entities is discrete, no context quantization is needed.
We report the performance of the resulting BiKE + CBR (w/o GAT) agent in
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Table 4.
The results show that CBR on TWC is effective even with this simple context deﬁnition, but the lower performance of the agent demonstrates the advantage of incorporating additional context information.
Finally, we investigate the role played by vector quantization, by experimenting with an agent ( BiKE + CBR w/o VQ ) that stores the continuous context representations.
In general, this poses scalability challenges, but since TWC has only 5 games per difﬁculty level, each with a small number of objects, we were able to evaluate the performance of this agent on the three levels separately.
The results, reported in Table 4, show that this agent performs much worse than the other
CBR implementations.
This happens because storing continuous representations over the training results in duplicated entries in the memory and makes it harder to retrieve meaningful experiences.
Figure 5 demonstrates how the size (number of entries) in the memory grows over the training time.
In this experiment, we trained the agent on all difﬁculty levels at the same time, resulting in the implementation running out of memory (OOM) on the GPU.
More ablation studies are reported in
Appendix C, D, E, F, and G.
R ELATED WORK
Text-based RL.
TBGs are a rich domain for studying grounded language understanding and how text information can be utilized in control.
Prior work has explored text-based RL to learn strategies for multi-user dungeon games (Narasimhan et al., 2015) and other environments (Branavan et al., 2012).
Zahavy et al.
(2018) proposed the Action-Elimination Deep Q-Network (AE-DQN), which learns to predict invalid actions in the text-adventure game Zork .
Recently, Côté et al.
(2018) introduced TextWorld , a sandbox learning environment for training and evaluating RL agents on text-based games.
On the same line, Murugesan et al.
(2021b) introduced TWC that requires agents with commonsense knowledge (Murugesan et al., 2020; Basu et al., 2021).
The LeDeepChef system (Adolphs & Hofmann, 2019) achieved good results on the First TextWord Problems (Trischler et al., 2019) by supervising the model with entities from FreeBase , allowing the agent to generalize to unseen objects.
A recent line of work learns symbolic (typically graph-structured) representations of the agent’s belief.
Notably, Ammanabrolu & Riedl (2019) proposed KG-DQN and Adhikari et al.
(2020) proposed GATA .
We also use graphs to model the state of the game.
Case-based reasoning in RL.
In the context of RL, CBR has been used to speed up and improve transfer learning in heuristic-based RL.
Celiberto Jr
et al.
(2011) and Bianchi et al.
(2018) have shown that cases collected from one domain can be used as heuristics to achieve faster convergence when learning an RL algorithm on a different domain.
In contrast to these works, we present a scalable way of using CBR alongside deep RL methods in settings with very large state spaces.
More recently, CBR has been successfully applied in the ﬁeld of knowledge-based reasoning.
Das et al.
(2020) and Das et al.
(2021) show that CBR can effectively learn to generate new logical reasoning chains from prior cases, to answer questions on knowledge graphs.
C ONCLUSION AND FUTURE WORK
In this work, we proposed new agents for TBGs using case-based reasoning.
In contrast to expensive deep RL approaches, CBR simply builds a collection of its past experiences and uses the ones relevant to the current situation to decide upon its next action in the game.
Our experiments showed that CBR when combined with existing RL agents can make them more efﬁcient and aid generalization in out-of-distribution settings.
Even though CBR was quite successful in the TBGs explored in our work, future work is needed to understand the limitations of CBR in such settings. A CKNOWLEDGEMENTS
This work was funded in part by an IBM fellowship to SD and in part by a small project grant to MS from the Hasler Foundation.
A T RAINING DETAILS
All agents used in our experiments are trained with an Advantage Actor Critic (A2C) method according to the general scheme deﬁned below.
We use the n -step temporal difference method (Sutton & Barto, 2018) to compute the return R ( s t , a t ) of a single time step t in a session of length
T as:
[Equation]
where V ( s T ) denotes the value of s T computed by the critic network, γ is the discount factor, and r is the reward function.
We then compute the advantage
A ( s t , a t ) =
R ( s t , a t ) − V ( s t ) .
The ﬁnal objective term consists of four separate objectives.
First, we have L ( t ) π that denotes the objective of the policy, which tries to maximize the the advantage A :
[Equation]
We then add the objective of the critic as:
[Equation]
L ( t ) v encourages the value of the critic V to better estimate the reward R by reducing the mean squared error between them.
To prevent the policy from assigning a large weight on a single action, we perform entropy regularization by introducing an additional term:
[Equation]
The η parameter helps balance the exploration-exploitation trade-off for the policy.
The sum of the above objectives deﬁnes the loss when the neural agent π is used to select the action a (cid:63)t .
In case actions are reused from the memory, then we use the contrastive loss as discussed in Section 5, namely we compute the loss as:
[Equation]
B E NHANCING BASELINE AGENTS WITH CBR ON J ERICHO
In this section
[Table]
we report additional experimental results on a subset of the Jericho games, in order to show the performance improvement obtained by different baseline agents when enhanced with case-based reasoning.
Table 5 shows the results obtained when coupling the agents described in
Section 6.1 with CBR.
Similarly to what we discussed for TWC in Section 6.
[Table]
we observe that
CBR consistently improves the performance of all the agents.
The best performing agent is BiKE +
CBR , which is the agent that we evaluated on the complete set of games in Section 6.
[Table]
C A BLATION STUDY ON MEMORY ACCESS
In this section, we investigate the effectiveness of our approach based on vector quantization for efﬁcient memory access.
We consider several variants, where VQ is either dropped completely or replaced with other techniques.
C. A LTERNATIVES TO VECTOR QUANTIZATION
All the agents we consider are variants of the best-performing agent on Jericho and TWC , namely the BiKE + CBR agent.
We experiment with the following techniques.
• BiKE + CBR (w/o VQ) completely removes the vector quantization and stores the continuous context representations as keys in the case memory.
In order to make this feasible, for the experiments on Jericho , we limit the size of the memory to the 5000 most recent entries.
• BiKE + CBR ( RP ) relies on random projection (RP) in order to reduce the dimensionality of the context representations stored by the CBR approach.
In this case, each context representation is projected into a p -dimensional space using a random matrix R ∈ R p × d , with components drawn from a normal distribution N (0 , 1 p ) with mean 0 and standard deviation 1 p .
• BiKE + CBR (SRP) employs sign random projection (SRP) in order to obtain a discrete representation of the context.
In this variant, after projection to a p -dimensional space, the context representations are discretized by applying an element-wise sign function.
• BiKE + CBR (LSH) replaces vector quantization with locality sensitive hashing (LSH).
In this case, context representations are converted into h -bit hash codes for l different hash tables.
The retrieve step selects the representation with the highest cosine similarity to the query context encoding, among the vectors falling in the same bucket.
C. R ESULTS AND DISCUSSION
Table 6 shows the scores achieved by each variant of the BiKE + CBR method on a subset of the
Jericho games.
We notice that the approaches relying on continuous context representations ( w/o
VQ and RP ) perform poorly compared to the others and to the plain BiKE agent.
This conﬁrms our hypothesis that CBR needs discrete context representations to make the retrieve step more stable.
The BiKE + CBR (LSH) agent, which relies on locality sensitive hashing, achieves competitive results and consistently outperforms the BiKE agent.
This shows that locality sensitive hashing can be a viable alternative to vector quantization.
However, we observe that our BiKE + CBR agent, based on vector quantization, achieves better results on almost all the games, conﬁrming the beneﬁt of learning the discrete representation as well.
Note also that LSH requires storing the complete continuous representations to be able to detect false positives, namely contexts with the same hash code as the query vector, but low cosine similarity.
This makes this alternative less memory efﬁcient compared to our implementation based on VQ.
[Table]
D A BLATION STUDY ON THE RETAIN MODULE
In this section, we evaluate different alternatives to select which actions should be retained in the memory of the agent
. D. A LTERNATIVES FOR THE RETAIN MODULE
In our main experiments described in Section 6, the retain module has been implemented to store the last k context-action pairs in the memory, whenever the reward obtained by the agent is positive.
For Jericho , we set k = 3 , as speciﬁed in Appendix J. However, other design choices are possible.
We consider the following variants of the BiKE + CBR agent.
• BiKE + CBR (rewarded action only) retains only the rewarded context-action pair, without sampling any of the previous actions.
This variant is a simple implementation that works well in practice, but may fail to identify useful actions that were not rewarded.
• BiKE + CBR (TD error) samples previous context-action pairs based on the temporal difference (TD) error.
In details, the agent still retains k context-action pairs: whenever the reward r t at time step t is positive, the rewarded context-action pair ( c (cid:63)t , a (cid:63)t ) is retained, together with k − 1 additional pairs sampled from the current trajectory, with a probability proportional to the TD error.
D. R ESULTS AND DISCUSSION
Table 7 shows the scores obtained by the agents on a subset of the Jericho games.
Our main implementation storing the last k = 3 actions achieves the best results, whereas the BiKE +
CBR (rewarded action only) agent performs slightly worse than the other two implementations.
The agent based on the TD error achieves competitive results and a state-of-the-art score on the snacktime game.
We observe that all variants perform well in practice and achieve overall better results than the baselines in Table 3.
[Table]
E T RAINING THE AGENT AND THE RETRIEVER JOINTLY
In this experiment, we try to understand if training the neural agent and the CBR retriever jointly would improve upon our choice of just training the two networks separately.
Our implementation works by training the neural agent π and the CBR retriever separately with different objectives, as described in Appendix A.
However, we can make the neural agent aware of the CBR retriever and train the two networks jointly.
In this case, we need to modify the architecture of the neural agent π in order to take into account the action candidates ˜
A t produced by the CBR.
In details, we compute an action selector by attention between the valid actions A t and the candidates ˜
A t and we concatenate this action selector to the vector used by the neural agent to score admissible actions.
Then, the agent and the retriever can be trained jointly, optimizing an objective given by the sum of all the losses in Section A.
Table 8 and 9 show the results obtained by the baseline agents when they are trained jointly with the CBR retriever.
On TWC , we observe that the joint variants of the agents achieve comparable results with their counterparts in Table 1 and 2.
On Jericho , the agents trained jointly with the retriever achieve strong results, but they perform slightly worse than our main approach that keeps the retriever separate from the neural agent.
This shows that the joint version is slightly harder to train.
Also, it brings the disadvantage that the architecture of the neural agent has to be changed to take the CBR into account, whereas our main approach that keeps the retriever separate allows readily plugging any on-policy agent for TBGs.
[Table] [Table]
F M ULTI - PARAGRAPH TEXT - BASED RETRIEVER
Knowledge graphs have been used extensively in text-based games and other areas of natural language understanding (Ammanabrolu & Hausknecht, 2020; Atzeni & Atzori, 2018; Kapanipathi et al., 2020).
This section describes an alternative to our graph-based retriever, which only relies on the textual observations without modeling the state of the game as a graph.
Recent work (Guo et al., 2020) has shown that enriching the current observation with relevant observations retrieved from the history of interactions with the environment can achieve competitive results on Jericho .
Therefore, in order to assess the effectiveness of our graph-based implementation, we compare to a multi-paragraph text-based retriever (MTPR) inspired by the work of Guo et al. [Table]
In this case
[Table]
we do not model the state as a graph and, subsequently, we remove the seeded graph attention mechanism from the retriever.
Instead, given the current natural language observation o t , we compute an action-speciﬁc representation following Guo et al.
(2020), concatenating o t with the n most recent observations that share objects with it or with the given action.
The encoded observation is then discretized using vector quantization as in our main architecture.
We evaluated the text-based retriever on Jericho , integrating it in the same baseline agents described in Section 6.1.
Table 10 shows the scores obtained by the agents.
We observe that, overall, the graph-based retriever performs better on the vast majority of the games.
This result conﬁrms the ability of our approach based on seeded graph attention to extract relevant information from the state of the game, compared to a retriever that only relies on text information.
G A DDITIONAL RESULTS USING ENTITIES AS CONTEXT SELECTORS
As mentioned in Section 6.4, we performed an ablation study where the seeded graph attention and the state graph where not used in the retriever.
Instead, we represent the context as just a single focus entity.
This choice suits very well the TWC games, as the goal of each game is to tidy up a house by putting objects in their commonsensical locations.
Hence, each rewarded action in TWC is of the form “ put o on s ” or “ insert o in c ”, where o is an entity of type object , s is a supporter , and c is a
[Table]
container (Murugesan et al., 2021c; Côté et al., 2018).
As an example, a rewarded action could be “ insert dirty singlet in washing machine ”, where dirty singlet is the object and washing machine is the container.
Representing the context as just single entities of type object , means that the memory of the CBR agent is storing what action to apply to each object in the game, and therefore the agent is in practice constructing a registry where each object is paired with its commonsensical location.
Let c v , c u be two context entities.
The retriever then computes the similarity between the contexts as:
[Equation]
where cosine denotes the cosine similarity, FFN is a 2-layer feed-forward network and h v , h u are the BERT encodings of the [CLS] token for objects v and u respectively.
The retriever is therefore encouraged to map objects that should be placed in the same location (either a supporter or a container ) to similar representations.
Figure 6 depicts a t -SNE (van der Maaten & Hinton, 2008) visualization of the representations
FFN ( h v ) learned by the retriever of the BiKE + CBR (w/o GAT) agent.
The plot shows that entities that belong to the same location are mapped to similar representations.
This holds both for objects in the in-distribution games and for objects in the out-of-distribution games.
[Figure]
We evaluated all the CBR agents with the simple retriever deﬁned above.
Table 11 reports the results for the in-distribution and out-of-distribution games.
The results conﬁrm that the entity-based context selection performs well and achieves good out-of-distribution generalization.
However, we remark that the complete retriever described in Section 4 consistently achieves better results, showing the importance of incorporating additional structured information. [Table]
H C ASE - BASED REASONING AND OUT - OF - DISTRIBUTION GENERALIZATION
Out-of-distribution generalization has recently fueled signiﬁcant research effort and several datasets and approaches have been proposed in the past few years (Bahdanau et al., 2019; Keysers et al., 2020;
Atzeni et al., 2021).
Our experiments on TWC allowed assessing the hypothesis that case-based reasoning can be used to tackle out-of-distribution (OOD) generalization in text-based games.
Table 12 shows the absolute OOD generalization gap of the different agents evaluated in our experiments.
Note that this table does not report any new result, but it simply provides the absolute difference between the values in Table 1 and 2.
We observe that the agents relying on CBR achieve a considerably better generalization performance out of the training distribution, almost comparable to the results obtained on the same distribution as the training data.
In some cases, the normalized score achieved by the CBR agents in the out-of-distribution games equals the score obtained in the in-distribution games.
This happens because case-based reasoning forces the agent to map contexts including entities that were not seen at training time to the most similar contexts in the CBR memory.
CBR allows the agent to solve completely new problems and generalize by effectively retrieving past cases and mapping the retrieved actions from the training distribution to the most similar options in the OOD setting.
Note that the only good OOD generalization gap for the agents that are not relying on CBR (the #Steps of the Text agent on the Hard level) is an artifact of the experiment, as all agents were limited to a maximum of 50 steps.
Figure 6 shows a nice example of the capability of the agent to generalize OOD.
In this case, entity embeddings were used as the context representations [Table] and we observe that entities that are not included in the training distribution are correctly mapped to the right cluster.
This shows that the
CBR approach learns effective and generalizable context representations based on the objective of the games.
These representations are then used by the agent to select relevant experiences and map the actions used at training time to the most viable alternative in the OOD test set.
[Table]
I A DDITIONAL BASELINES ON J ERICHO
Several methods have been proposed recently for text-based games.
In order to keep the results in the main paper more compact, we only included in Table 3 well-known and top-performing baselines that were evaluated on the full (or almost) set of Jericho games.
For completeness, this section compares our best agent ( BiKE + CBR ) with the following additional methods:
• Q*BERT (Ammanabrolu et al., 2020) is a deep reinforcement learning agent that plays text games by building a knowledge graph of the world and answering questions about it; • Trans-v-DRRN (Xu et al., 2020) relies on a lightweight transformer encoder to model the state of the game; • DBERT-DRRN (Singh et al., 2021) makes use of DistilBERT (Sanh et al., 2019) ﬁne-tuned on an independent set of human gameplay transcripts.
Table 13 shows the scores obtained by these baselines compared to our agent enhanced with CBR.
Overall
[Table]
we observe that the BiKE + CBR agent outperforms the baselines on the majority of the games, conﬁrming the effectiveness of case-based reasoning as a viable approach to boost the performance of text-based RL agents.
[Table]
J H YPERPARAMETERS AND REPRODUCIBILITY
All CBR agents are trained using the same hyperparameter settings and the same hardware/software conﬁguration.
As mentioned in Section 4, we use a pre-trained BERT model (Devlin et al., 2019) to represent initial node features in the state graph.
BERT is only used to compute the initial representations of the entities and is not ﬁne-tuned.
We use the following hyperparameters for our experiments
• We set the hidden dimensionality of the model to d = 768 and we use 12 attention heads for the graph attention network, each applied to 64 -dimensional inputs.
• We use n l = 2 seeded GAT layers for TWC and n l = 3 for Jericho.
• On both datasets, we apply a dropout regularization on the seeded GAT with probability of 0 .
1 at each layer.
• Similarly, for the experiments on TWC , we only sample the most recent context-action pair from T , whereas we sample k = 3 pairs for Jericho.
We used k = 2 for the scalability analysis depicted in Figure 5.
• The retriever threshold is kept constant to τ = 0 .
7 across all experiments.
• On TWC , we train the agents for 100 episodes and a maximum of 50 steps for each episode.
On Jericho, as mentioned, we follow previous work and we train for 100 000 valid steps, starting a new episode every 100 steps or whenever the games ends.
• We set the discount factor γ to 0.9 on all experiments.
• For the ablation study on memory access, we set the output dimensionality of the RP and SRP methods to p = 64 .
For LSH, we set the number of hash tables to l = 16 and the length of the hash codes of h = 8 bits.
We artiﬁcially limit the size of each bucket to themost recent entries.
Experiments were parallelized on a cluster where each node was dedicated to a separate run.
The conﬁguration of the execution nodes is as reported in Table 14.
[Table]