M ONTE -C ARLO P LANNING AND L EARNING WITH L ANGUAGE A CTION V ALUE E STIMATES
Anonymous authors
Paper under double-blind review
A BSTRACT
Interactive Fiction (IF) games provide a useful testbed for language-based reinforcement learning agents, posing signiﬁcant challenges of natural language understanding, commonsense reasoning, and non-myopic planning in the combinatorial search space.
Agents based on standard planning algorithms struggle to play IF games due to the massive search space of language actions.
Thus, language-grounded planning is a key ability of such agents, since inferring the consequence of language action based on semantic understanding can drastically improve search.
In this paper, we introduce Monte-Carlo planning with Language
Action Value Estimates (MC-LAVE) that combines a Monte-Carlo tree search with language-driven exploration.
MC-LAVE invests more search effort into semantically promising language actions using locally optimistic language value estimates, yielding a signiﬁcant reduction in the effective search space of language actions.
We then present a reinforcement learning approach via MC-LAVE, which alternates between MC-LAVE planning and supervised learning of the selfgenerated language actions.
In the experiments, we demonstrate that our method achieves new high scores in various IF games.
I NTRODUCTION
Building an intelligent goal-oriented agent that can perceive and react via natural language is one of the promising real-world applications of artiﬁcial intelligence.
As a means of pursuing this, we consider Interactive Fiction (IF) games
(Nelson, 2001; Montfort, 2005), text-based simulation environments where the agent interacts with the environment only through natural language.
They serve as a useful testbed for developing language-based goal-oriented agents, posing important challenges such as natural language understanding, commonsense reasoning, and non-myopic planning in the combinatorial search space of language actions.
IF games naturally have a large branching factor, possessing at least hundreds of natural language actions that can affect the simulation at game states.
This renders naive exhaustive search infeasible and raises the strong need for language-grounded planning ability, i.e. effective search space is too large to choose an optimal action without inferring the future impact of language action by understanding the environment state described in natural language.
Still, standard planning methods such as Monte-Carlo tree search (MCTS) are language-agnostic and rely only on uncertainty-driven exploration that encourages to search more on less-visited states and actions.
This simple uncertainty-based strategy is not sufﬁcient to ﬁnd an optimal language action within a limited search time, especially when each language action is treated as a discrete token.
On the other hand, recent reinforcement learning agents for IF games have started to leverage pre-trained word embeddings for language understanding (He et al., 2016; Fulda et al., 2017; Hausknecht et al., 2019) or knowledge graphs for commonsense reasoning (Ammanabrolu & Hausknecht, 2020), but their exploration strategies are still limited to the (cid:15) -greedy or the softmax policies, lacking more structured and non-myopic planning ability.
As a consequence, current state-of-the-art agents for IF games still have not yet been up to the human-level play.
In this paper, we introduce Monte-Carlo planning with Language Action Value Estimates (MCLAVE), a planning algorithm for the environments with text-based interactions.
MC-LAVE combines Monte-Carlo tree search with language-driven exploration, addressing signiﬁcant search inefﬁciency attributed to the lack of language understanding.
It starts with credit assignment to language actions via Q-learning of the experiences collected from the past searches.
Then, MC-LAVE assigns non-uniform search priorities to each language action based on the optimistically aggregated Qestimates of the past actions that share similar meanings with the candidate action, so as to focus more on the semantically promising actions.
This is in contrast to the previous methods that involve language understanding in the form of a knowledge graph, where the insigniﬁcant language actions are uniformly ﬁltered out by graph mask (Ammanabrolu & Hausknecht, 2020; Ammanabrolu et al., 2020).
The non-uniform search empowered by language understanding in MC-LAVE yields better search efﬁciency while not hurting the asymptotic guarantee of MCTS.
We then present our reinforcement learning approach that uses MC-LAVE as a strong policy improvement operator.
Since MCTS explores the combinatorial space of action sequences, its search results can be far better than the simple greedy improvement, as demonstrated in the game of Go (Silver et al., 2017).
This ﬁnal algorithm, MC-LAVE-RL, alternates between planning via MC-LAVE and supervised learning of self-generated language actions.
Experimental results demonstrate that MC-LAVE-RL achieves new high scores in various IF games deﬁned in the Jericho framework (Hausknecht et al., 2019), showing the effectiveness of language-grounded MC-LAVE planning.
B ACKGROUND 2. I NTERACTIVE F ICTION G AME
Interactive Fiction (IF) games are fully text-based environments where the observation and the action spaces are deﬁned as natural language.
The game-playing agent observes textual descriptions of the world, selects an language-based action, and receives the associated reward.
IF games can be modeled as a special case of partially observable Markov decision processes (POMDPs) deﬁned by tuple (cid:104) S, A, Ω , T, O, R, γ (cid:105) , where S is the set of environment states s , A is the set of language actions a , Ω is the set of text observations o , T ( s (cid:48) | s, a ) =
Pr( s t +1 =
s (cid:48) | s t = s, a t = a ) is the transition function, R ( s, a ) is the reward function for taking action a in state s , O ( s )
= o is the deterministic observation function in state s , and γ ∈ (0 , 1) is the discount factor.
The history at time step t , h t = { o 0 , a 0 , . . .
, o t − 1 , a t − 1 , o t } , is a sequence of observations and actions.
The goal is to ﬁnd an optimal policy π ∗ that maximizes the expected cumulative rewards, i.e. π ∗ = arg max π E π [ (cid:80) ∞ t =
γ t R ( s t , a t )] .
We use the same deﬁnition of observation and action space as Hausknecht et al.
; Ammanabrolu & Hausknecht (2020); Cˆot´e et al.
(2018), i.e. An observation is deﬁned by o t =
( o t desc , o t game , o t inv , a t − 1 ) where o t desc is textual description of the current location of agent, o t game is simulator response to the previous action taken by the agent, o t inv is information of agent’s inventory, and a t − 1 is the previous action taken by the agent.
An action is denoted by a sequence of words a t =
( a 1 t , a 2 t , . .
. , a | a t | t ) .
Finally, we denote A valid ( o t ) ⊆
A as a set of valid actions with respect to the given observation o t , which is provided by the Jericho environment interface.
Figure 1 shows an example of observation and action in Z ORK 1 which is one of the most popular IF games.
2. C HALLENGES IN I NTERACTIVE F ICTION G AME
IF games possess important challenges for language-based reinforcement learning agents, requiring natural language understanding, commonsense reasoning, and non-myopic language-grounded planning ability in combinatorial search space of language actions (Hausknecht et al., 2019).
More concretely, consider the particular game state of Z ORK 1 described in Figure 1.
In this example situation, a human player would be capable of performing strategic planning based on language understanding and commonsense reasoning as follows: (1) the ‘ closed trap door ’ will have to be opened and be explored to proceed with the game, (2) however, acquiring the ‘ lantern ’ should precede entering the trap door since the cellar, which is expected to exist below the trap door, could be likely pitch-dark.
Without such language-grounded reasoning and planning, the agent may explore the entire action space uniformly, making it vulnerable to being eaten by a monster in the cellar who always appear when there is no light source.
As a result, any agent that lacks the ability of long-term planning with language reasoning is prone to be stuck in a suboptimal policy, which enters the cellar to obtain an immediate reward and does nothing further to avoid encountering the monster that kills the agent immediately.
𝒐 𝒕 𝐝𝐞𝐬𝐜 : Living Room.
You are in the living room.
There is a doorway to the east, a wooden door with strange gothic lettering to the west, which appears to be nailed shut, a trophy case, and a closed trap door at your feet.
Above the trophy case hangs an elvish sword of great antiquity.
A battery-powered brass lantern is on the trophy case.
𝒐 𝒕 𝐠𝐚𝐦𝐞 : With a great effort, the rug is moved to one side of the room, revealing the dusty cover of a closed trap door.
𝒐 𝒕 𝐢𝐧𝐯 : You are empty-handed.
𝒂 𝒕−𝟏 : push rug 𝒂 𝒕 : open trap door 𝑨 𝐯𝐚𝐥𝐢𝐝 𝒐 𝒕 : { north, turn on lantern, open trap door, open case, take sword, take lantern, east, take all }
Observation
Action
Valid actions
To overcome the bottleneck, the agent should be able to infer that ‘ take lantern ’ is worthy enough to be chosen in preference to other actions even though it gives no immediate reward.
In this work, we will precisely address such failures of myopic planning that arise from the lack of language-grounded exploration strategy.
To this end, we ﬁrst address the nonmyopic-planning with MCTS-based long-term planning. Then we further improve the search efﬁciency of MCTS-based planning by incorporating our novel language-grounded exploration strategy based on Language Action Value Estimates, which will be detailed in the subsequent section.
2. M ONTE -C ARLO T REE S EARCH
Monte-Carlo Tree Search (MCTS) (Kocsis & Szepesv ´ ari, 2006; Coulom, 2006;
Browne et al., 2012) is a generic online planning algorithm that combines random sampling and tree search, and it has been the de-facto standard method for large sequential decision-making problems.
Starting from an empty tree, MCTS repeatedly performs the four phases of selection, expansion, rollout, and backpropagation to evaluate the nodes of the search tree more and more accurately.
UCT (Kocsis &
Szepesv ´ ari, 2006) is the standard MCTS method, which adopts UCB (Auer et al., 2002) as an action selection rule at each internal node of the search tree:
[Equation]
where h is the history of past observations and actions, Q ( h, a ) is the average value of sampled returns with taking action a in h , N ( h ) is the number of simulations performed in h , N ( h, a ) is the number of times action a is selected in h , and c is a constant that balances exploration and exploitation.
However, UCT suffers from severe search inefﬁciency in the problems with a large action space such as IF games, since it requires to take every action at least once and relies only on uncertainty-driven (or visit counts-based) exploration.
PUCT (Silver et al., 2017; 2018) partially addresses the challenges of IF games by adopting PUCB (Rosin, 2011), which involves a prior action distribution π ( ·| h ) and eliminates the need for choosing every action at least once.
The action selection rule is given by:
[Equation]
The prior distribution π ( ·| h ) in Eq.
(2) is typically trained by behavioral cloning of ( h root , a ∗ root ) samples obtained by the result of tree search (Anthony et al., 2017; Silver et al., 2018).
However, this procedure not only discards other search information such as abundant ( h t , a t , r t , h t +1 ) samples obtained during the search but also hardly encourages information sharing across the semantically similar actions.
Therefore, PUCT is still limited in search efﬁciency for the tasks with language action space, raising the need for a language-driven exploration strategy.
M ONTE -C ARLO P LANNING WITH L ANGUAGE A CTION V ALUE E STIMATES
Language-grounded planning ability is essential to address the challenges of IF games.
To this end, we need a mechanism to incorporate language understanding and commonsense reasoning into the exploration strategy of search.
In this section, we introduce Monte-Carlo planning with Language Action Value Estimates (MC-LAVE), a novel planning algorithm that combines MCTS with language-driven exploration.
MC-LAVE does not rely on any handcrafted heuristic evaluation but starts tabula rasa, based only upon pre-trained word embeddings that can well deﬁne the semantic similarities between two language actions.
In addition, it only requires a black-box simulator ( r, s (cid:48) , o (cid:48) )
∼ G ( s, a ) , which yields ( o, a, r, o (cid:48) )
samples at each simulator query.
3. C REDIT A SSIGNMENT WITH S EARCH E XPERIENCE
Language-driven exploration in MC-LAVE starts by identifying which word sequences of language action are likely to give high long-term rewards.
This credit assignment of language action is done by
Q-learning using all the experiences D
= { ( o, a, r, o (cid:48) ) } obtained during the past searches.
Following the previous work (Hausknecht et al., 2019; Ammanabrolu & Hausknecht, 2020; Ammanabrolu et al., 2020; Madotto et al., 2020), we assume that the current observation captures all the information regarding the latent state, as it is just a text description of the game state.
Therefore, we represent the Q-function as a function of the current observation and action, rather than a function of full history and action.
Our parameterization of Q φ ( o, a ) basically follows that of Deep Reinforcement Learning Relevance Network (DRRN)
(He et al., 2016; Hausknecht et al., 2019), and more details can be found in
Appendix A. Finally, the Q φ is trained by minimizing the temporal difference error using D :
[Equation]
where ¯ φ is an exponential moving average of φ , i.e. soft target update, and A valid ( o (cid:48) ) is a set of valid actions in the observation o (cid:48) provided by Jericho environment.
While PUCT of Eq.
(2) also uses previous search information to construct a prior policy π , it is typically trained only on planning trajectories ( h root , a ∗ root ) that generally cover very limited regions.
Therefore, it can hardly provide informative search guides for areas outside the planning trajectories.
In contrast, Q φ of
Eq.
(3) is trained by using every search experience encountered during search processes, thus it can potentially provide more information over much broader set of states and actions.
However, the credit assignment provided by Q φ still struggles to provide useful information for novel language observation-action regions that have never been experienced, making its direct adoption as an exploration strategy inappropriate: the search process necessarily entails visits to novel areas where Q φ does not work properly, and thus Q φ may instead promote exploitation.
3. L ANGUAGE - DRIVEN E XPLORATION VIA L ANGUAGE A CTION V ALUE E STIMATES
The key to generalizing existing experiences to novel situations is context-agnostic information sharing between language actions based on their semantic similarities.
For example, if an action of ‘ take something ’ has ever been useful in the past, we may expect that the action of ‘ take something else ’ can also be worthy in other situations.
Motivated by such instances, we introduce Language
Action Value Estimates (LAVE), which assigns non-uniform search priorities to each language action by examining whether its semantic neighborhood actions have been beneﬁcial in the past.
As the ﬁrst step for LAVE, we deﬁne a semantic neighborhood of action a , a set of the past experiences whose actions share similar semantics with action a by:
[Equation]
where δ is a hyperparameter that controls a proximity threshold, and d ( a, ¯ a ) is a distance measure between two actions a and ¯ a in language embedding space represented by the pre-trained word embeddings e ( · ) :
[Equation]
For simplicity, we use the average of the word embeddings ψ ( a ) as the language action (or sentence) embedding, where the semantic similarity can be well-measured by the cosine similarity in the embedding space.
[Figure]
Transition tuples
[Figure]
⋯
Monte-Carlo planning with Language Action Value Estimates 𝑜 !
(#) 𝑎 !
∗(#) 𝑜 &(#) 𝑜 ’(#) 𝑜 ((#) 𝑎 &∗(#) 𝑎 ()!
∗(#) 𝑎 !
∗(#) 𝑎 &∗(#) 𝑎 ’∗(#) 𝑎 (∗(#)
Then, the LAVE exploration bonus for the action a is estimated through a soft maximum over the
Q-values of a ’s semantic neighborhood actions in order to encourage optimistic exploration.
[Equation]
Note that L φ ( a ) in Eq.
(5) relies only on the Q-values of (¯ o, ¯ a ) samples experienced in the past, thus can be reliably estimated.
Also, it is context-agnostic in which the action a is taken, thus circumvents the challenge of measuring the similarity between the very long textual observations, while promoting active information sharing between affordable language actions.
Finally, MC-LAVE encourages the language-driven exploration in the selection phase of MCTS via
LAVE L φ ( a ) , which selects actions at each intermediate nodes by the following rule:
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
(6) where π θ ( a | o ) is a prior policy trained by behavior cloning of past planning results, L φ ( a ) is a language-driven exploration bonus, c LAVE is an exploration constant.
MC-LAVE uses the policy π θ for incorporating prior knowledge of the past planning trajectories, as well as the language-driven bonus L φ ( a ) for facilitating information sharing between semantic neighborhood actions.
MCLAVE initially encourages to explore the actions with high language-driven exploration bonuses and low visit count, but it does not hurt the asymptotic behavior of MCTS (Theorem 2 in Appendix C).
The full pseudocode for the MC-LAVE is provided in Appendix D.
R EINFORCEMENT L EARNING VIA MC-LAVE
As shown in (Silver et al., 2017), MCTS can be used as a strong policy improvement operator for reinforcement learning.
Similarly, we use MC-LAVE as a policy improvement operator for learning the IF game.
As a result, our reinforcement learning algorithm, MC-LAVE-RL, alternates between
MC-LAVE planning and supervised learning of self-generated actions:
Planning Use the MC-LAVE with current policy π θ and Q-function Q φ to collect the n planning trajectories T
= (cid:83) i τ ( i ) and experience replay D :
[Equation]
where a ∗ t ( i ) is selected action at i -th trajectory t -th timestep by MCTS and T denotes a episode length.
(For the ﬁrst iteration, MC-LAVE uses uniform policy and randomly initialized Q-function.)
Algorithms
DRRN †
TDQN †
KG-A2C ‡
MC!Q*BERT §
PUCT-RL
MC-LAVE-RL
R e qu i r e m e n t s
Valid Action
Hard
Soft Soft
Soft
Hard Hard
Resettable - - - (cid:51) (cid:51) (cid:51)
External Info - - -
ALBERT & Jericho-QA -
[Table]
D i f
ﬁ c u lt
G a m e s
[Table]
ORK
[Table]
P o ss i b l e
G a m e s
[Table]
et al.
The results of other algorithms are from Hausknecht et al.
(2019) † , Ammanabrolu & Hausknecht (2020) ‡ , and Ammanabrolu et al.
§ respectively.
All our results (PUCT-RL, MC-LAVE-RL) indicate averages over 5 independent runs, and their standard errors are provided in Appendix E due to space limit.
Policy training Update the policy network parameter θ by minimizing the negative loglikelihood with using collected planning trajectories T ,
i
e. supervised learning with crossentropy loss:
[Equation]
Q-learning Update the Q-function parameter φ by minimizing the temporal difference error from experience replay D :
[Equation]
We continue to update the policy through the above policy iteration with MC-LAVE operator until the policy performance converges.
E XPERIMENTS
In this section, we show experimental results of our approach on IF games included in Jericho environment (Hausknecht et al., 2019).
The action spaces of IF games in Jericho environment are deﬁned by the sequences of the game-dependent input vocabulary whose size is roughly fromto 2000.
While the size of the vocabulary is limited compared to large-scale open domain dialogues, it still serves as a useful testbed to demonstrate the effectiveness of semantic-based information sharing of MC-LAVE 1 .
First, we evaluate performances of MC-LAVE-RL comparing with baseline methods.
Second, we compare intermediate scores of MC-LAVE-RL on Z ORK 1 with a baseline, PUCT-RL that uses PUCT as a policy improvement operator, to show that MC-LAVE is the more effective for IF games than PUCT.
Third, we investigate the effectiveness of semantic neighborhood by comparing the planning performance of MC-LAVE with varying the size of semantic neighborhood. Finally, we give a qualitative analysis of our approach by providing a representative example,
[Table]
which shows how MC-LAVE focuses on promising actions during search.
Detailed conﬁgurations of our experiments are described in Appendix B.
5. E VALUATION ON I NTERACTIVE F ICTION G AMES
First, we compare the performance of MC-LAVE-RL with the following algorithms: (1) DRRN (Hausknecht et al., 2019), a variant of DQN algorithm (Mnih et al., 2013) for natural language action space, (2) TDQN (Hausknecht et al., 2019), an extension of LSTM-DQN algorithm (Narasimhan et al., 2015) incorporating with template-based action generation, (3) KG-A2C (Ammanabrolu & Hausknecht, 2020), actor-critic method with knowledge graph state representation, (4)
MC!Q*BERT (Ammanabrolu et al., 2020), an extension of KG-A2C with BERT-based knowledge graph construction and knowledge-graph-based intrinsic reward.
In addition, we also compare MCLAVE-RL with our baseline called PUCT-RL, which uses PUCT as a policy improvement operator.
Table 1 summarizes handicaps leveraged in each algorithm and the performance of MC-LAVE-RL and baseline algorithms across 9 IF games included in Jericho environment.
The results show that
MC-LAVE-RL outperforms or matches the state-of-the-art results on 8 out of 9 games.
Although
MC-LAVE-RL requires more handicap or assumption, it performs the same or better than strong baseline MC!Q*BERT which requires similar assumptions and more requirements.
In addition,
MC-LAVE-RL achieves higher game scores on overall games compared to PUCT-RL, which is a baseline algorithm that only excludes language-driven exploration strategy from MC-LAVE-RL.
Furthermore, MC-LAVE-RL performs signiﬁcantly better than other methods on difﬁcult games such as Z ORK 1, D EEPHOME , and L UDICORP , which are categorized by Hausknecht et al.
as a relatively challenging game due to the large action space and sparse rewards.
In the case of Z ORK 1, as described in Section 2.2, other algorithms (except MC!Q*BERT, which is concurrent work by Ammanabrolu et al.
fail to overcome the bottleneck and are stuck into the suboptimal policy, which leaves the agent being eaten by a monster in the dark without a light source.
On the other hand, MC-LAVE-RL successfully ﬁnds the optimal action (i.e.
‘ take lantern ’) on the bottleneck state by using the exploration strategy suitable for natural language spaces and obtains additional rewards by discovering novel states that cannot be reached with other methods.
For instance, trajectories of MC-LAVE-RL agent frequently show that the agent overcomes bottlenecks and behaves crucial actions to complete the game, such as ‘ killing the monster with sword ’ and ‘ putting the painting in the trophy case ’.
Further illustrative examples of trajectories are provided in Appendix F.
5. R EINFORCEMENT L EARNING VIA MC-LAVE
In order to understand the effectiveness of MC-LAVE as a policy improvement operator, we compare the performances of PUCT-RL and MC-LAVE-RL in Z ORK 1.
Table 2 reports the intermediate results of planning and supervised learning in each iteration of the policy iteration.
In each iteration, the policy and the Q-function are trained using planning trajectories and experience replay collected from 25 independent planning agents.
As can be seen in Table 2, the performance of MC-LAVE-RL is improved more consistently than PUCT-RL, both in planning and learning.
At the beginning of the policy iteration, PUCT-RL improves the performance, but it fails to overcome bottleneck and converges to a suboptimal policy: PUCT utilizes the prior policy learned by imitating the planning results of the previous iteration to estimate the exploration bonus, but this uncertainty-based method is not much effective to encourage the agent to explore the action space that is not sufﬁciently covered.
On the other hand, MC-LAVE-RL not only uses the prior policy, but also uses Q-Network [Table] for credit assignment to language actions.
This allows a more focused exploration on semantically promising actions and consequently overcomes the bottleneck to further improve the performance
5. T HE E FFECT OF THE S IZE OF S EMANTIC N EIGHBORHOOD
[Figure] The size of semantic neighborhood is particularly crucial for the performance of MC-LAVE planning. To see this, we conducted a simple ablation experiment in Z ORK 1 on the effect of varying proximity threshold δ , which controls the size of the semantic neighborhood of action N ( a ) in
Eq.
Moreover, to demonstrate that the LAVE exploration bonus works as a meaningful exploration bonus beyond simple random noise injection, we also present the result of a simple baseline PUCT+RANDOM, which uses
MC-LAVE action selection rule of Eq.
(6) with the neighborhood given by 10 randomly sampled actions.
This baseline can be understood as performing the search with noise injection via the randomly chosen neighborhood, while not considering information sharing between semantically similar actions.
As shown in Figure 3, the performance of PUCT+RANDOM is almost identical to PUCT, which supports that random noise injection alone does not lead to improving search efﬁciency.
Moreover, the results of MC-LAVE shows that too small or too large δ are not effective, which can be naturally explained as follows: (1) when δ = 0 , no sample in the replay buffer is considered as neighborhood, leading LAVE exploration bonus to always zero, (2) when δ = 2 (the largest possible value), every sample in the replay buffer is considered as neighborhood, yielding LAVE exploration bonus to a constant value for all a .
Therefore, with either two extremes of δ , the MC-LAVE action selection rule of Eq.
(6) is reduced to the PUCT action selection rule of Eq.
Finally, the moderately small δ = 0 .
3 works the best, which highlights the importance of information sharing between meaningfully deﬁned semantic neighborhoods to solve
IF games efﬁciently.
5. Q UALITATIVE A NALYSIS
In this section, we provide a qualitative analysis on the effectiveness of language-driven exploration in MC-LAVE.
We show an illustrative example in Table 3 that demonstrates how MC-LAVE focuses more on the semantically promising actions.
Table 3 shows search results of PUCT and MC-LAVE in the bottleneck state of Z ORK 1, described in Figure 1.
In this example, the ‘ open trap door ’ is a suboptimal action that obtains an immediate reward, while ‘ take lantern ’ gives no immediate reward, but it is an optimal action when considering long-term dependency.
As can be seen in
Table 3, PUCT cannot effectively explore because of an evenly search based on uncertainty, and converges into suboptimal action (i.e. ‘ open trap door ’) that obtains an immediate reward.
On the other hand, MC-LAVE assigns search priorities to actions based on the optimistically aggregated
Q-estimates of the past actions.
As a result of focusing on semantically promising actions, MCLAVE selects the optimal action (i.e. ‘ take lantern ’) even though it does not give any immediate reward.
PUCT also performs non-uniform search based on prior policy, but this only encourages to explore the actions of previous behavior and does not focus on semantically promising actions.
R ELATED W ORK
Variants of Monte-Carlo tree search.
Leveraging prior knowledge for MCTS has been one of the most important direction for improving MCTS (Gelly & Silver, 2007; Silver et al., 2016; 2017; 2018; Gelly & Silver, 2011).
PUCT (Rosin, 2011; Silver et al., 2017; 2018) is one of the most popular MCTS algorithms, which incorporates prior knowledge as a policy.
The prior policy in PUCT is typically trained by imitation learning of past behavior; other search information, such as transition tuples and inferred Q-values, is discarded.
To address this limitation, Hamrick et al.
(2019) presents Search with Amortized Value Estimates (SAVE), which incorporates Q-based prior knowledge into MCTS.
In SAVE, Q-function is ﬁt by inferred Q-values and transition tuples from search, then is used as a prior into MCTS.
Although SAVE improves sample efﬁciency in MCTS, it lacks a language-grounded exploration strategy.
Rapid Action Value Estimation (RAVE) (Gelly & Silver, 2011) is another notable method that addresses information sharing, which allows sharing action values in the search tree.
However, RAVE is designed especially for the game of Go and cannot be straightforwardly applied to IF games.
Furthermore, to leverage the prior knowledge given as textual information, Branavan et al.
(2011) present a method which extracts the most relevant information to current state and incorporates linguistic information into the Monte-Carlo search framework.
However, they assume natural language documents that contain domain knowledge of the task.
Reinforcement Learning via Monte-Carlo tree search.
There are a number of recent work on using MCTS for reinforcement learning.
Silver et al.
(2017) achieved successful results with MCTS as a policy improvement operator in the game of Go.
Also, in the negotiation dialogue domain,
Jang et al.
(2020) used Bayes-adaptive Monte-Carlo planning as a policy improvement operator to prevent the issue of diverging from human language.
Inspired by these successes, we use MC-LAVE algorithm as a policy improvement operator which effectively works on IF games.
Deep Reinforcement Learning for Interactive Fiction Games.
IF games have attracted a lot of interest from the RL research since the introduction of Jericho (Hausknecht et al., 2019), an OpenAI
Gym-like (Brockman et al., 2016) learning environment for IF games.
Jericho includes baseline RL agents such as DRRN and TDQN.
DRRN is a variant of deep Q-Network architecture for handling natural language actions in text-based decision making tasks.
TDQN is an extension of LSTMDQN (Narasimhan et al., 2015) utilizing predeﬁned templates in order to reduce effective search spaces of combinatorial language action space.
More recently, knowledge-graph-based approaches have been introduced to address the issues of partial observability and commonsense reasoning (Ammanabrolu & Riedl, 2019; Ammanabrolu & Hausknecht, 2020; Ammanabrolu et al., 2020).
KG-A2C (Ammanabrolu & Hausknecht, 2020) is an actor-critic algorithm that leverages knowledge graph state representation for commonsense reasoning.
Ammanabrolu et al.
(2020) introduce
Q*BERT, an extension of KG-A2C, which updates the knowledge graph by using pre-trained language model ALBERT (Lan et al., 2020), a variant of BERT (Devlin et al., 2019).
To encourage better exploration and overcome the bottleneck state in IF games, they introduce MC!Q*BERT that additionally uses intrinsic motivation based on changes of the knowledge graphs.
C ONCLUSION
We presented Monte-Carlo planning with Language Action Value Estimates (MC-LAVE), an
MCTS-based algorithm for language action space, which combines MCTS with language-driven exploration strategy.
MC-LAVE assigns search priorities to each language action based on Q-values of previously executed actions that share similar meanings, then invests more search effort into semantically promising actions.
We then incorporated MC-LAVE into reinforcement learning by using MC-LAVE as a policy improvement operator.
Our approach achieved remarkable results in difﬁcult games that feature large action space and sparse reward and outperforms or matches the state-of-the-art on the various IF games.
A I MPLEMENTATION DETAILS
As discussed in section
[Figure]
MC-LAVE-RL alternates (1) planning
[Figure]
(2) supervised learning and Qlearning.
In the planning phase
[Figure]
multiple planning workers are run in parallel on a distributed computing system to collect various planning trajectories and experience replay.
After planning
[Figure]
we merge collected trajectories and experience replay and train the policy network and Q-Network using merged trajectories and experience replay respectively.
For Q-Network
[Figure]
we exploit the same architecture used in DRRN (He et al., 2016) illustrated as Figure
[Figure]
which takes an observation o t , an action a t as an input and outputs Q φ ( o t , a t ) .
For the policy network, we use a network architecture similar with DRRN as described in Figure
[Figure]
which directly outputs f θ ( a t | o t ) , an unnormalized probability of an action a t given an observation o t , i.e. π θ ( a t | o t )
∝ f θ ( a t | o t ) .
[Figure] [Figure]
B E XPERIMENTS DETAILS
[Figure]
[Table]
depth
[Table] [Table]
[Table] [Table]
c PUCT
C A SYMPTOTIC O PTIMALITY OF MC-LAVE
In this section, we show that the asymptotic optimality of PUCB (Rosin, 2011) is still held after the introduction of the language-driven exploration bonus in Eq.
We start by restating the theoretical result for PUCB.
Lemma 1.
(Theorem 1 in (Rosin, 2011))
For any stochastic policy π ( a ) > 0 with (cid:80) a π ( a ) = 1 , c 1 > 0 , and c 2 > 0 , PUCB action selection rule at trial t is deﬁned as:
[Equation]
where N ( a ) is the number of simulations action a is taken in the past, r ( t, a ) is an empirical average reward of action a at the start of trial t .
, Then, expected regret of PUCB action selection rule is bounded by O ( √ n log n ) ∀ n > 1 .
Note that the prior policy π in the PUCB rule encourages the actions that have a high value of π ( a ) to be selected more often.
In this work, we use a variant of PUCB as in (Silver et al., 2017).
Theorem 2.
MC-LAVE action selection rule at trial t is deﬁned as:
[Equation]
where N ( a ) is the number of simulations action a is taken in the past, r ( t, a ) is an empirical average reward of action a at the start of trial t , π ( a ) > 0 is a stochastic policy with (cid:80) a π ( a ) = 1 , c 1 > 0 , c 2 > 0 , c 3 > 0 and L ( a ) > 0 is a bonus of action a .
Then MC-LAVE action selection rule also achieves expected regret bounded by O ( √ n log n ) ∀ n > 1 .
Proof.
The theorem can be proved by showing that MC-LAVE action selection rule can be represented as PUCB action selection rule with another policy.
MC-LAVE action selection rule is deﬁned as
[Equation]
Deﬁne p ( a ) , π (cid:48) ( a ) , c (cid:48) 2 as follows:
[Equation]
Then MC-LAVE action selection rule can be represented as follows:
[Equation]
which induces PUCB action selection rule with another policy π (cid:48)
and coefﬁcient c (cid:48) 2 .
Hence, by
Lemma 1, MC-LAVE achieves expected regret bounded by the same order with PUCB, which is
O ( (cid:112) n log( n )) ∀ n > 1 .
Thus, in the same manner we obtain the analysis of UCT from that of UCB, i.e. induction with a mild assumption on the drift-condition of the inherent non-stationary bandit problem, we can obtain the analysis of MCTS using MC-LAVE using the above result.
D P SEUDOCODE OF MC-LAVE
procedure S EARCH ( s 0 ) o 0 ← O ( s 0 )
h 0 ← o 0 repeat
S IMULATE ( s 0 , h 0 , 0 ) until T IMEOUT () return arg max a ∈
A valid ( o 0 ) Q ( h 0 , a ) end procedure procedure S IMULATE ( s, h, t )
if t = ( planning horizon T ) then return 0 end if [ a, rollout ] ← S ELECT A CTION ( h )
[ r, s (cid:48) , o (cid:48) ] ← G ( s, a ) h (cid:48)
← hao (cid:48) if rollout then R (cid:48)
←− R OLLOUT ( s (cid:48) , t + 1) else R (cid:48)
←− S IMULATE ( s (cid:48) , h (cid:48) , t + 1) end if N ( h ) ← N ( h )
+
N ( h, a ) ← N ( h, a ) +
Q ( h, a ) ← Q ( h, a ) + R − Q ( h,a ) N ( h,a ) return r + R (cid:48)
end procedure procedure S ELECT
A CTION ( h ) a ←− arg max a ∈
A valid ( o ) (cid:34)
Q ( h, a ) + c PUCT π θ ( a | o ) √ N ( h ) N
( h,a )
+ + c LAVE L φ ( a ) √ N ( h ) N ( h,a )
+ (cid:35) if N ( h, a ) = 0 then rollout ←− true else rollout ←− false end if return [ a, rollout ] end procedure procedure R OLLOUT ( s, t )
if t = ( planning horizon T ) then return 0 end if o ← O ( s ) a ∼ π θ ( ·| o )
[ r, s (cid:48) , o (cid:48)
[Table]
E C OMPARISON R ESULTS OF S OFT AND H ARD C ONSTRAINTS
[Table]
For an equivalent comparison in terms of valid action handicap
[Table]
we also re-implemented KG-A2C (published state-of-the-art) in both ways that the valid action handicap are used as a soft constraint via entropy loss over valid actions and as a hard constraint for the effective action space.
The results show MC-LAVE-RL still signiﬁcantly outperforms or matches the both of KG-A2C with using soft and hard constraints on the 8 out of 9 games.
F I LLUSTRATIVE B EHAVIOR E XAMPLES
To understand how MC-LAVE-RL discovers more novel states than other reinforcement learning agents, we provide illustrative examples that represent the behavior of each agent playing Z ORK 1.
[Figure]
G I LLUSTRATIVE E XAMPLES FOR L ANGUAGE - ACTION E MBEDDING
We present the t-SNE visualization for the language-action embedding and the (semantic) neighborhood of the action ‘ take lantern ’ on Z ORK 1.
In the following ﬁgures, the color of points indicates the observation-marginalized Q-value of each action with respect to the replay buffer, which is deﬁned as:
[Equation]
Only a subset of the entire language actions is presented in the ﬁgures to prevent the cluttered presentation.
The bold font with cyan and yellow indicate the (semantic) neighborhood of the action ‘ take lantern ’ and ‘ open trap door ’, and the bold font with magenta indicates the common neighborhood of them.
As shown in Figure 7, the semantic neighborhood of language actions deﬁned by the pre-trained word embedding is composed of semantically similar actions.
However, as shown in Figure 7, when using the word embedding ﬁne-tuned by Q φ training, semantic neighborhood includes a number of irrelevant actions.
This result highlights the importance of word embedding for effective information sharing in MC-LAVE-RL. [Figure] t- [Figure]
[Figure]
word embedding
[Figure]
H
I LLUSTRATIVE E XAMPLE FOR S EMANTIC - BASED I NFORMATION S HARING
[Table] In order to show the effectiveness of language-action embedding, we further investigate MC-LAVE planning on the bottleneck state presented in Figure 1 and Table 3. Speciﬁcally, in the bottleneck state, we artiﬁcially modiﬁed the action ‘ take lantern ’ to ‘ get lantern ’ within the valid action set which is used for MC-LAVE planning. As shown in Table 6, MC-LAVE still can successfully select the modiﬁed ‘ get lantern ’ over ‘ open trap door ’ as a ﬁnal action, even though ‘ get lantern ’ had never seen by the agent before.
This generalization property of MC-LAVE is due to the effect of pre-trained embedding, which shows the importance of language-action embedding for information sharing across semantic space.