Learning Symbolic Rules over A bstract M eaning R epresentations for Textual Reinforcement Learning (2023)

Abstract: Text-based reinforcement learning agents have predominantly been neural network-based models with embeddings-based representation, learning uninterpretable policies that often do not generalize well to unseen games. On the other hand, neuro-symbolic methods, specifically those that leverage an intermediate formal representation, are gaining significant attention in language understanding tasks. This is because of their advantages ranging from inherent interpretability, the lesser requirement of training data, and being generalizable in scenarios with unseen data. Therefore, in this paper, we propose a modular, NEuro-Symbolic Textual Agent (NESTA) that combines a generic semantic parser with a rule induction system to learn abstract interpretable rules as policies. Our experiments on established text-based game benchmarks show that the proposed NESTA method outperforms deep reinforcement learning-based techniques by achieving better generalization to unseen test games and learning from fewer training interactions.

## introduction text-based games (tbgs) @xcite serve as popular sandbox environments for evaluating natural language-based reinforcement learning. the agent observes the state of the game in pure text and issues a textual command to interact with the environment. tbgs are partially observable where the full state of the world is hidden and action commands facilitate the agent to explore the unobserved parts of the environment. the reward signal from the environment is used to improve the agent's policy and make progress in the game. text-based games sit at the intersection of two research areas, i.e., language understanding and reinforcement learning. existing rl agents for tbgs primarily use embeddings for observation as representations and are fed to an action scorer for predicting the next action @xcite 2015a; @xcite , ignoring the advances in language understanding. on the other hand, there has been a recent surge in neurosymbolic techniques, particularly those that use symbolic representations, for better language understanding @xcite through reasoning. in light of exploring such advances for text-based reinforcement learning, this work proposes a neuro-symbolic approach. our approach, named nesta (neuro symbolic textual agent) is a modular approach comprising a generic semantic parser in combination with a symbolic rule induction system as shown in figure 1 . the semantic parser translates text into the form of symbolic triples. nesta uses abstract meaning representation @xcite as the initial parse which is then transformed into triples. this symbolic representation is used by an adaptation of the inductive logic programming (ilp) sys-tem using logical neural networks @xcite for learning horn clauses as action rules. nesta, in comparison to other end-to-end learning approaches, has the following advantages: (a) modular language understanding using pre-trained large language models enabling our system to leverage the advances in semantic parsing. while such modular semantic parsing-based techniques have been around for other nlp tasks such as reading comprehension @xcite , knowledge base question answering @xcite , and natural language inference @xcite , this work is the first to demonstrate the application for tbgs ; (b) learning symbolic rules for model-free rl using a neuro-symbolic framework facilitates inherent interpretability and generalizability to unseen situations @xcite @xcite . the rules learned by nesta are abstract and not specific to entities in the training data. these abstract action rules in policies for tbgs enable reasoning over unseen entities during training. our main contributions in this work are: (1) we propose a novel and modular neuro-symbolic agent named nesta. to the best of our knowledge, nesta is the first to use a generic semantic parser with a rule learning system for tbgs, (2) our empirical analysis of commonsense-aware textworld games shows that nesta outperforms deep rl methods by a significant margin. we also show that nesta has better sample efficiency compared to traditional text-based rl agents obtaining better test performance with up to 5× lesser training interactions, and (3) our method produces interpretable abstract rules from the rule induction system. ## neuro-symbolic textual agent text-based rl agents for tbgs interact with the environment using text-only action commands and obtain feedback solely as textual observations. as the agent does not have access to global state information, it is modeled as a partially observable markov decision process (pomdp) @xcite represented as (s, a, t , r, ω, o), where (s, a, t , r) represent a markov decision process. ω represents the finite set of all observations, and o represents the observation function representing the conditional distribution over observations for a given action and next state. the goal of the agent is to learn optimal action probabilities at each step such that the expected future reward is maximized. we present neuro-symbolic textual agent (nesta), a modular approach for tbgs. figure 1 illustrates the overview of nesta which comprises of three primary components: (a) semantic parser, which extracts symbolic representation of the text using amr as the generic semantic representation, (b) rule learner, an ilp-based rule induction module, which learns logical rules that abstract out the entities in the games, making these rules generally applicable to test games containing unseen entities, and (c) pruner, that reduces the amount of branching factor at each step by pruning actions that do not contribute to the expected future reward. below, we describe these components in detail. ## outlier rejection in policy training the training samples that nesta collects from interacting with the environment can be noisy and this can affect learning a good policy. there exists two sources of noise: (a) amr noise, where amr incorrectly parses the surface text resulting in erroneous identification of entity extraction or relationships between entities, and (b) rl credit assignment noise, where discounted reward gives reward to a suboptimal action taken right before a correct action. although symbolic reasoners have the advantages of learning from fewer data and better generalization, they are not robust to noise. we mitigate the effect of noise in lnn policy training by using a consensus-based noise rejection method. our noise rejection method trains the lnn policy on multiple subsets of training data and selects the model with the smallest training error as the best model. the multiple subsets of training data are prepared as follows -for each training subset, a particular predicate p from the predicate list p is given priority. we only choose state transitions that contain the predicate p ensuring that this predicate will be part of the final learned rule, thus elimi-nating the source of amr noise for this predicate (such a subset is rejected if the number of such transitions is less than some threshold percentage). subsequently, the resulting transitions are sorted by the discounted reward g t and we only retain the top first k% of this sorted data as training data. this encourages action transition with more immediate average reward gains to constitute the training data. ## experimental results our experiments are designed to answer these questions that analyze if nesta can overcome the common drawbacks of deep rl methods: (i) can nesta enable better generalization in test environments? (ii) does nesta improve upon sample efficiency while still maintaining good reward performance, (iii) are the rules learned by nesta, human interpretable? for comparing the performance of various methods, we use the metrics of normalized score (total reward from the games normalized by maximum reward) and number of steps to reach the goal (lower is better). our experiments were conducted on ubuntu 18.04 operating system with nvidia v100 gpus. ## conclusion in this paper, we present nesta, a neuro-symbolic policy learning method that modularizes language understanding using an amr-based semantic parsing module and rl policy optimization using an ilp rule learner. nesta benefits from prior advances in amr-based generic parsers for symbolic fact extraction allowing the ilp symbolic learner to solely learn interpretable action rules. nesta outperforms sota models on tbgs by showing better generalization while learning from a fewer number of training interactions. we believe our model is one of the first works combining advances in neural semantic parsing and efficient symbolic planning for text-based rl. we hope this work will encourage future research in this direction.