Neuro-Symbolic Approaches for Text-Based Policy Learning (2021)

Abstract: Text-Based Games (TBGs) have emerged as important testbeds for reinforcement learning (RL) in the natural language domain. Previous methods using LSTM-based action policies are uninterpretable and often overfit the training games showing poor performance to unseen test games. We present SymboLic Action policy for Textual Environments (SLATE), that learns interpretable action policy rules from symbolic abstractions of textual observations for improved generalization. We outline a method for end-to-end differentiable symbolic rule learning and show that such symbolic policies outperform previous state-of-the-art methods in text-based RL for the coin collector environment from 5-10x fewer training games. Additionally, our method provides human-understandable policy rules that can be readily verified for their logical consistency and can be easily debugged.

## introduction text-based games are increasingly being used as a benchmark for progressing the state of the art in natural language rl. these games typically require solving goals that are defined by natural language descriptions such as "retrieve the coin in the cellar". the agent receives a textual description of the scene and can interact with the environment using only textual commands such as "go north", "take knife" upon which it receives a reward signal for completing the goal (or sub-goal). the action policy model is trained using such reward signals. previous methods in text-based rl typically use memory-based recurrent systems for feature extraction from textual observations @xcite or knowledge graph extraction @xcite for better 1 reproducibility: we will release our code at https://github.com/subhajit1411/ slate-text-based-rl observation: you find yourself in a bedroom. an usual one. i guess you better just go and list everything you see here. there is an exit to the north. don't worry, it is unguarded. there is an exit to the south. don't worry, it is unblocked. you don't like doors? why not try going west, that entranceway is unguarded. symbolic facts: hasexit(bedroom, north), hasexit(bedroom, south), hasexit(bedroom, west), hasvisited(bedroom, south) figure 1 : overview of our neuro-symbolic rule learning. slate learns interpretable action policy for each action verb, go and take, from first-order symbolic states. we show the raw weights for each and gate (∧) which are thresholded to obtain the logical rules. in this paper, we bridge the gap between gradient-based weight learning and symbolic reasoning applied to text-based rl. we introduce symbolic action policy for textual environments (slate), a method for interpretable policy learning in text-based games from symbolic state representation. our goal is to learn symbolic rules as logical connectives for generating action commands by gradient-based training. we present a symbolic rule learning framework using both mlp with symbolic inputs and logical neural network (lnn) @xcite , a recent symbolic reasoning-based approach, to learn lifted rules from the first-order symbolic abstraction of textual observations. we show that our symbolic action policy learning framework outperforms previous text-based rl methods in terms of generalization to unseen games even when the previous methods use symbolic state representation. ## related work most previous works on text-based rl handle the problems of partial observability and large action space. lstm-dqn @xcite ) is an early work on text-based rl that used an lstmbased encoder for feature extraction from textual observations and q-learning (watkins and dayan, 1992) for learning the action policy model. lstm-drqn @xcite used memory units in the action score to handle the issue of partial observability. crest @xcite showed that previous methods overfit the training data and improved generalization by training a bootstrapped model on context-relevant observation text only. @xcite presents the winning strategies in the first-textworld competition, which uses recurrent feature extraction along with the a2c @xcite rl algorithm for training the policy. previous works that extract knowledge graphs from observations (ammanabrolu and riedl, 2019; ammanabrolu and hausknecht, 2020) showed improved performance compared to processing raw textual observations. the recent method of @xcite use a dynamic belief graph learning for better generalization than text-based policy learning. ## symbolic action policy for textual environments (slate) to improve generalization in tbgs, we propose slate that learns logical rules based on firstorder symbolic inputs from the environment. the symbolic inputs are fed into the logical rule learner to obtain the likelihood of action commands. ## results for baseline methods we use three previous agents: lstm-dqn @xcite , lstm-drqn @xcite and crest @xcite . the results show that our proposed slate successfully generalizes to unseen test games learned from 5 -10x fewer games and performs better zero-shot generalization. learned interpretable rules: figure 2 shows how we extract the learned rules for easy n25 games case. we use a weight threshold of 1 n input , and all connections greater than the threshold contribute to the learned rule. the rule for go-lnn action verb looks at exits that are not visited from the current room given the coin is not present in the room. take-lnn takes coin only if the coin is present in the room. these rules match with the logic a human agent would typically use for navigation in such an environment. comparison to text-based agents: we also compare slate with lstm-dqn trained/tested on symbolic inputs. lstm-dqn agent is given the symbolic input as textual facts like "has exit east has exit south has visited south". since slate does not have a recurrent action policy and makes single-step decisions, for a fair comparison we do not compare it with lstm-drqn that has a recurrent action scorer unit. the results in zero-shot transfer: similar to the experiments in @xcite , figure 3 shows the zeroshot performance of slate compared to lstm-drqn and crest when trained on games having 15 rooms (l15) and tested on 20 (l20) and 25 (l25) rooms. slate (with lnn) shows better generalization compared to previous text-based agents due to compact and logically consistent rule learning. this is because slate learns lifted rules that inherently provide the structure to generalize to unseen configurations that pure deep learning approaches struggle to achieve. results on cooking world: we also show results on cooking games @xcite , where the agent has to navigate to the kitchen, gather items that a present in the recipe, and prepare and eat the meal. for this game, we use 6 predicates: inroom(x) (if item x is in the current room), isingredient(x) (if item x is a required ingredient), ininventory(x) (if item x is currently in the inventory), closed(x) (if item x is in closed state), hasexit(x) and hasvisited(x). the actions verbs to learn rules for are: take(x), open(x), and go(x). we use the game walkthrough to learn lnn rules in the teacher imitation mode. the learned rules for three verbs are: open(x) ← inroom(x) ∧ closed(x), take(x) ← inroom(x) ∧ isingredient(x), go(x) ← ¬inroom(x) ∧ hasexit(x). we assume that learning rules with first-order quantifiers with forall (∀) and exists (∃) are out-of-scope of this paper. therefore, the rule for "prepare meal" is specified as ∀ z (isingredient(z) ∧ ininventory(z)) and for "eat meal" is ininventory(@xmath0. we present the normalized score on test games for slate for difficulty level 3 with training 20 at 66.67% which outperforms the scores of 41.7% for gata, 41.7% for gata-gtp, and 46.7% for gata-gtf from @xcite , showing slate's efficiency for complicated cases. however, there is still room for improvement for slate on the cooking games because the rule for go(x) ← ¬inroom(x) ∧ hasexit(x) is inconsistent and the term ¬inroom(x) is not required on the right-hand side for the correct rule. ## conclusions we present slate, a neuro-symbolic approach for action policy learning in text-based games using a differentiable rule learner from first-order symbolic inputs. our method outperforms previous text-based state-of-the-art methods on textworld coin collector games from 5 -10x fewer training games and shows zero-shot generalization to unseen test configurations since it learns compact, interpretable, and logically correct rules. in this paper, we presented neuro-symbolic rule learning for the case, where the list of possible predicates require knowledge of the domain and are known before the training. for a more generalized rule learning on an apriori unknown set of predicates, we plan to use information extraction techniques like abstract meaning representation (amr) and openie for obtaining a domain-agnostic graphical state representation for rule induction as a future extension of this work.