Learning Neuro-Symbolic World Models with Conversational Proprioception (2023)

Abstract: The recent emergence of Neuro-Symbolic Agent (NeSA) approaches to natural language-based interactions calls for the investigation of model-based approaches. In contrast to model-free approaches, which existing NeSAs take, learning an explicit world model has an interesting potential especially in the explainability, which is one of the key selling points of NeSA. To learn useful world models, we leverage one of the recent neuro-symbolic architectures, Logical Neural Networks (LNN). Here, we describe a method that can learn neuro-symbolic world models on the TextWorld-Commonsense set of games. We then show how this can be improved further by taking inspiration from the concept of proprioception, but for conversation. This is done by enhancing the internal logic state with a memory of previous actions while also guiding future actions by augmenting the learned model with constraints based on this memory. This greatly improves the game-solving agents performance in a TextWorld setting, where the advantage over the baseline is an 85% average steps reduction and x2.3 average score.

## introduction recent emergence of neuro-symbolic (ns) approaches include natural language-based sequential decision making @xcite @xcite . they propose a model-free approach of learning a logical policy, and tested with interactive-text games @xcite @xcite , which have become an interesting benchmark in the intersection of natural language processing and sequential decision making. ns approaches give the direct explainability of what is learned and allow natural integration of external knowledge as logic. despite that, existing ns approaches are of model-free reinforcement learning (rl) but it would be useful if we figure 1 : simplified overview of the problem setting and agent. the observation from the environment describes the state in a natural language form. our agent first uses a semantic parser to produce an estimate of the logical state. we then use this in a model-based rl module that learns a logical world model to plan actions to be taken. we augment the rl agent with a specific memory component inspired from proprioception. in this paper, we focus on the problem of learn-ing logical world models in ns methods. the main research question to be addressed is then how we can learn such models for text-based games using a general semantic parser. as a state-of-the-art interactive-text agent, gata @xcite constructs belief graphs used to enhance deep rl methods. in contrast to understanding the world state in a latent space, we want to explicitly use the logical world models to plan optimal action sequences and to provide direct explainability of the decision making policy. for the explainability purpose, we leverage general semantic parsing, following one of the early work constructing knowledge graphs (ammanabrolu and riedl, 2019 ). an overview of our proposed method is depicted by figure 1 . the left side depicts that the environment state can be sufficiently approximated as a set of logical facts. continuing in the top right, the agent can get textual observations of the environment. we assume that we have a semantic parser @xcite that converts these observations into a logical form. in the real situation the semantic parsing is good, but won't be perfect, hence we require that our agent should be capable of handling noisy logical states. from such states, our agent should produce suitable actions for accomplishing its tasks in the environment. the main contributions of this paper are: the proposal of a novel world model-learning method with a neuro-symbolic approach, and its experimental results with twc. ## problem definition text-based games are often modelled with the rl problem setting in mind as partially observable -markov decision processes (po-mdp) @xcite . as a first approach, we add an assumption -that the semantic parser can remove partial observability and that we are dealing with an mdp. at each time step the agent uses the information in a state, s, to take an action, a, which transitions the state to the new state, s ′ according to the state transition function t such that s ′ = t (s, a). while acting in this environment the agent also gets rewards, r, according to an unknown reward function, r, such that @xmath0, a). the training loop consists of exploring the environment by taking actions while keeping track of the experience in the form of {s, a, r, s ′ }. an agent then uses this set of experiences to learn something that enables it to take better actions. in the model-free rl setting, the agent learns a policy or value function which can directly govern the actions. here, we are interested in the model-based rl setting where the agent learns a model of the world which usually consists of both t and r. this model can then be used with planning methods to find the optimal actions. based on the classical model-based rl setting, our problem has two more important specifications. first, we assume that our environment is relational, similar to @xcite . this means that all actions and states are composed of relational logic. they may be in the propositional form but there must be a corresponding lifted form that has a consistent meaning. for example, the propositional state, on(book,table) can be abstracted or lifted into on(x,y) with predicate, on, and the variables, (x, y). the first assumption is that all states and actions handled by the agent are in this relational lifted form. this assumption can be handled as a design specification of the semantic parser. the second assumption is that the goal state is given. this is a weaker assumption that is already used in current rl research, the so-called goal-conditioned rl. here, it allows us to concentrate only on learning t since r is no longer required for planning when we are given the goal state. ## learning logical world models the problem of learning logical rules that explain a given set of logical examples can be cast into the general problem called inductive logic programming (ilp) @xcite . what needs to be done is then to cast our relational model-based rl problem into ilp form. but before going into that detail, it is important to note that relying on classical ilp has significant failings. in particular, it is not well suited to noisy data to the extent that a single erroneous data point may cause the whole system to fail. however, newer methods that leverage neural networks have shown great promise on working even with noisy data @xcite . these are sometimes called neural ilp, differentiable ilp or neuro-symbolic ilp. these advances are the main impetus for us to research on the learning of logical world models. we may use any such ilp method that is noiseresistant but here we use the logical neural network (lnn) @xcite as a neuro-symbolic ai framework. it is an end-to-end dif-ferentiable system that enables scalable gradientbased learning and it has a real-valued logic representation of each neuron that enables logical reasoning @xcite . ## experiment and discussion for evaluating the quality of the world model learned, we first qualitatively analyze the learned action models. then we measure the interactivetext agent performance against a quantitative textworld benchmark. in this paper, we experiment on the textworld commonsense (twc) set of games @xcite with the same experimental settings. once we have a logical world model, we can use it with a planner. here, we use the fast-downward system @xcite . for convenience, we convert the learned logical transition model into the pddl (planning domain definition language) format by combining (α, β, ϵ) into the preconditions and (γ, σ) into the effects. we also augment the state with ϵ(s, a). ## conclusion we proposed a model-based rl agent for textbased games which comprises of a semantic parser producing logical states, a neuro-symbolic ilp module for learning logical world models, and an off-the-shelf planning system to produce optimal actions in the game world. we augment this with a proprioception-inspired module comprising both the memory and memory-based constraints. our results and experiments show that each of the components are essential and our model-based nesa agent outperforms previous benchmarks on the textworld commonsense set of games.