Neuro-Symbolic Reinforcement Learning with First-Order Logic (2021)

Abstract: Deep reinforcement learning (RL) methods often require many trials before convergence, and no direct interpretability of trained policies is provided. In order to achieve fast convergence and interpretability for the policy in RL, we propose a novel RL method for text-based games with a recent neuro-symbolic framework called Logical Neural Network, which can learn symbolic and interpretable rules in their differentiable network. The method is first to extract first-order logical facts from text observation and external word meaning network (ConceptNet), then train a policy in the network with directly interpretable logical operators. Our experimental results show RL training with the proposed method converges significantly faster than other state-of-the-art neuro-symbolic methods in a TextWorld benchmark.

## introduction deep reinforcement learning (rl) has been successfully applied to many applications, such as computer games, text-based games, and robot control applications @xcite @xcite @xcite . however, these methods require many training trials for converging to the optimal action policy, and the trained action policy is not understandable for human operators. this is because, although the training results are sufficient, the policy is stored in a black-box deep neural network. these issues become critical problems when the human operator wants to solve a real-world problem and verify the trained rules. if the trained rules are understandable and modifiable, the human operator can control them and design an action restriction. while using a symbolic (logical) format as representation for stored rules is suitable for achieving interpretability and quick training, it is difficult to train the logical rules with a traditional training approach. in this paper, we propose an action knowledge acquisition method featuring a neuro-symbolic lnn framework for the rl algorithm. through experiments, we demonstrate the advantages of the proposed method for real-world problems which is not logically grounded games such as blocks world. since natural language observation is easier to convert into logical information than visual or audio, we tackle text-based interaction games for verifying the proposed method. figure 1 shows an overview of our method. the observation text is input to a semantic parser to extract the logical values of each propositional logic. in this case, the semantic parser finds there are two exits (north and south). the method then converts first-order logical (predicates) facts from the propositional logics and categories of each word, such as ∃@xmath0, north}, ⟨find x⟩ = t rue and ∃@xmath1, west}, ⟨find x⟩ = f alse. these extracted predicated logics are fed into lnn which has some conjunction gates and one disjunction gate. the lnn trains the weights for these connections by the reward value to obtain the action policy. the contributions of this paper are as follows. various prior works have examined rl for textbased games. lstm-dqn @xcite is an early study on an lstm-based encoder for feature extraction from observation and q-learning for action policy. lstm-dqn++ @xcite extended the exploration and lstm-drqn @xcite was proposed for adding memory units in the action scorer. kg-dqn (ammanabrolu and riedl, 2019) and gata @xcite extended the language understanding. ledeepchef @xcite used recurrent feature extraction along with the a2c @xcite . crest @xcite was proposed for pruning observation information. twc @xcite was proposed for utilizing common sense reasoning. however, none of these studies used the neuro-symbolic approach. for recent neuro-symbolic rl work, the neural logic machine (nlm) @xcite was proposed as a method for combination of deep neural network and symbolic logic reasoning. it uses a sequence of multi-layer perceptron layers to deduct symbolic logics. rules are combined or separated during forward propagation, and an output of the entire architecture represents complicated rules. in this paper, we compare our method with this nlm. ## conclusion in this paper, we proposed a novel neuro-symbolic method for rl on text-based games. according to the evaluation on the natural language text-based game with several difficulties, our method can converge extremely faster than other state-of-theart neuro-only and neuro-symbolic methods, and extract trained logical rules for improving interpretability of the model.