EXPLORER : Exploration-guided Reasoning for Textual Reinforcement Learning (2024)

Abstract: Text-based games (TBGs) have emerged as an important collection of NLP tasks, requiring reinforcement learning (RL) agents to combine natural language understanding with reasoning. A key challenge for agents attempting to solve such tasks is to generalize across multiple games and demonstrate good performance on both seen and unseen objects. Purely deep-RL-based approaches may perform well on seen objects; however, they fail to showcase the same performance on unseen objects. Commonsense-infused deep-RL agents may work better on unseen data; unfortunately, their policies are often not interpretable or easily transferable. To tackle these issues, in this paper, we present EXPLORER which is an exploration-guided reasoning agent for textual reinforcement learning. EXPLORER is neuro-symbolic in nature, as it relies on a neural module for exploration and a symbolic module for exploitation. It can also learn generalized symbolic policies and perform well over unseen data. Our experiments show that EXPLORER outperforms the baseline agents on Text-World cooking (TW-Cooking) and Text-World Commonsense (TWC) games.

## introduction natural language plays a crucial role in human intelligence and cognition. to study and evaluate the process of language-informed sequential decision-making in ai agents, text-based games (tbgs) have emerged as important simulation environments, where the states and actions are usually described in natural language. to solve game instances, an agent needs to master both natural language processing (nlp) and reinforcement learning (rl). at a high level, existing rl agents for tbgs can be classified into two classes: (a) rulebased agents, and (b) neural agents. rule-based agents such as nail @xcite 1 code available at: https://github.com/kinjalbasu/explorer ## symbolic policy learner deep reinforcement learning (drl) has experienced great success by learning directly from highdimensional sensory inputs, yet it suffers from a lack of interpretability. interpretability of an agent's action is of utmost importance in sequential decision-making problems, as it increases the transparency of black-box-style agents; it also helps rl researchers understand the high-level behavior of the system better. to make a system interpretable, one of the most widely used approaches is learning the agent's policies symbolically. in our work, explorer learns these symbolic policies in the form of logical rules represented in the asp. an example of such a rule is insert(x, fridge) :-apple(x) which can be translated as "x is insertable into a fridge if x is an apple" foot_0 . these learned asp rules not only provide a better understanding of the system's functionality but can also be used to predict the agent's next action using an asp solver. explorer learns the rules iteratively (in an online manner) and applies the rules to predict an action in collaboration with the neural module. our results show that this approach is very effective in terms of performance and interpretability. partial observability and non-monotonic reasoning: explorer works in a partially observable environment, where it needs to predict an action based on its prior knowledge. if explorer fails, then it learns something new that will be applied in the next episode. the reasoning approach of explorer is non-monotonic in nature: that is, what it believes currently may become false in the future with new evidence. we can model this using a non-monotonic logic programming paradigm that supports default rules and exception to defaults @xcite . in this work, the belief of explorer has been represented as an answer set program in the form of default rules with exceptions. with the help of inductive logic programming (ilp) (see section 3.1) and exception learner (see section 3.2), these rules are learned by explorer after each episode and then applied in the following episode. the agent uses an asp solver to predict actions by utilizing the observation and the rules. based on the outcome after applying the rules, the learned policies are updated with the exception (if needed), and new rules are learned as needed. ## rule generalization importance of rule generalization: an ideal rl agent should not only perform well on entities it has seen but also on unseen entities or outof-distribution (ood) data. to accomplish this, policy generalization is a crucial feature that an ideal rl agent should have. to verify this, we used explorer without generalization on the tw-cooking domain, where it performs well, however, it struggles on the twc games. twc games are designed to test agents on ood entities that were not seen during training but are similar to the training data. as a result, the policies learned as logic rules will not work on unseen objects. for example, the rule for apple (e.g., insert(x, fridge) <-apple(x). ) cannot work on another fruit such as orange. to tackle this, we lift the learned policies using wordnet's @xcite hypernym-hyponym relations to get the generalized rules (illustration in figure 5 ). motivation comes from the way humans perform tasks. for example, if we know a dirty shirt goes to the washing machine and we have seen a dirty pant, we would put the dirty pant into the washing machine as both are of type clothes and dirty. excessive generalization is bad: on one hand, generalization results in better policies to work with unseen entities; however, too much generalization leads to a drastic increment in false-positive results. to keep the balance, explorer should know how much generalization is good. for an example, "apple is a fruit", "fruits are part of a plant", and "plants are living thing". now, if we apply the same rule that explains a property of an apple to all living things, the generalization will have gone too far. so, to solve this, we have proposed a novel approach described in section -4.1. ## future work and conclusion in this paper, we propose a neuro-symbolic agent explorer that demonstrates how symbolic and neural modules can collaborate in a text-based rl environment. also, we present a novel information gain-based rule generalization algorithm. our approach not only achieves promising results in the tw-cooking and twc games but also generates interpretable and transferable policies. our current research has shown that excessive reliance on the symbolic module and heavy generalization may not always be beneficial, so our next objective is to develop an optimal strategy for switching between the neural and symbolic modules to enhance performance.