Deeply Embedded Knowledge Representation & Reasoning For Natural Language Question Answering: A Practitioner’s Perspective (2020)

Abstract: Successful application of Knowledge Representation and Reasoning (KR) in Natural Language Understanding (NLU) is largely limited by the availability of a robust and general purpose natural language parser. Even though several projects have been launched in the pursuit of developing a universal meaning representation language, the existence of an accurate universal parser is far from reality. This has severely limited the application of knowledge representation and reasoning (KR) in the field of NLP and also prevented a proper evaluation of KR based NLU systems. Our goal is to build KR based systems for Natural Language Understanding without relying on a parser. Towards this we propose a method named Deeply Embedded Knowledge Representation & Reasoning (DeepEKR) where we replace the parser by a neural network, soften the symbolic representation so that a deterministic mapping exists between the parser neural network and the interpretable logical form, and finally replace the symbolic solver by an equivalent neural network, so the model can be trained end-to-end. We evaluate our method with respect to the task of Qualitative Word Problem Solving on the two available datasets (QuaRTz and QuaRel). Our system achieves same accuracy as that of the state-of-the-art accuracy on QuaRTz, outperforms the state-of-the-art on QuaRel and severely outperforms a traditional KR based system. The results show that the bias introduced by a KR solution does not prevent it from doing a better job at the end task. Moreover, our method is interpretable due to the bias introduced by the KR approach.

## introduction developing agents that understand natural language is a long standing challenge in ai. towards this, several question answering challenges have been proposed, namely squad @xcite containing reading comprehension problems, obqa @xcite , qasc @xcite containing science questions requiring inference over multiple facts, propara @xcite , socialiqa @xcite , recipeqa @xcite requiring understanding of events and effects, quartz @xcite , quarel @xcite requiring qualitative reasoning and babi @xcite containing a broad set of synthetic tasks. for most of these challenges there exists a kr based methodology which typically says, if "the problem and the associated knowledge is represented as 'r', then there exists an algorithm 'a' which can compute the answer". however, almost no end-to-end system that executes such a solution exists (except for babi and quarel), as obtaining the desired representation 'r' with precision is a challenging task. for the dataset babi, which contains synthetically generated simple sentences, existing semantic parsers work well and thus several kr systems @xcite @xcite have been implemented for it. but for other datasets, researchers have had to build their own semantic parser when implementing a kr solution. for e.g., the work in @xcite has developed the quasp + translation system for quarel. data collection for training a semantic parser is a costly process and often parser error becomes a bottleneck to the final system performance. our goal is eliminate reliance on a semantic parser and to allow rapid implementation of kr based solutions so that the gap between "there is a kr solution" and "there is a system implementing a kr solution" diminishes. roughly speaking, our proposed approach takes a kr solution and simulates it in a neural network. there are three design choices that are involved in the construction of the simulator neural net-work. the first design process aims to answer the following question: "how to encode the symbolic representation 'r' in terms of vectors so that a deterministic process can convert the vectors back to the original symbolic form?". the second design process aims to construct a neural network which is responsible for computing the desired vector encoding of 'r'. the third process, implements the reasoning algorithm 'a' in a neural network which takes as input the vector encoding of the symbolic representation 'r'. the parameters of the networks are learned jointly in an end-to-end fashion. we call this approach, deeply embedded knowledge representation & reasoning (deepekr). in this work, we describe a deepekr solution for the task of qualitative problem solving ( ## qualitative word problem solving a noticeable portion of textual knowledge, particularly in science, economics, and medicine, are qualitative in nature, i.e. they describe how changing one entity (e.g., diesel car) affects another (e.g., air pollution). to help nlu systems become better at understanding such sentences, recently two datasets, quarel and quartz, containing qualitative word problems ( ## a kr solution a kr solution typically describes a high level language where a parser translates the natural lan-k1 bigger stars produce more energy, so their surfaces are hotter. q1 jan is comparing stars, specifically a small star and the larger sun. given the size of each, jan can tell that the sun puts out heat that is (a) greater (b) lesser k2 an object with greater mass or greater velocity has more kinetic energy. q2 milo threw both a basketball and a baseball through the air. if the basketball has more mass then the baseball, which ball has more kinetic energy (a) basketball (b) baseball k3 a sunscreen with a higher spf protects the skin longer. q3 billy is wearing sunscreen with a higher spf than lucy. who will be protected from the sun for longer? (a) lucy (b) billy ## encoding the symbolic representation with vectors in this section we describe, how we encode the symbolic representation in terms of vectors. we model each term t whether it is a concept description (e.g., "spf of sunscreen") , a concept value (e.g., "more") or a frame of reference (e.g., "billy") in terms of two vectors, namely the term surface vector, a t and the term content vector, v t . the term surface vector, a t captures the attention over the natural language input and surrogates for the symbolic description (in our case, phrases like "spf of sunscreen"). the term content vector v t surrogates for its meaning. for the terms in st , such as concept 1 value, which take values from a close set, the dimension of the term content vector v t is equal to the size of that close set, essentially describing a distribution over the members of the set. in the symbolic form, each qualitative word problem is represented in terms of 12 terms. in its vector form, each problem is thus represented as 12 pair of vectors. let m be the length of the input sequence tokens (words or sub-words) containing both the knowledge sentence and the multiple choice question (see figure 1 ). each term surface vectors a t is then a member of the set [0, 1] n (figure 1 ). ideally, we want a t to @xmath0, 1} n , however we don't put such an hard constraint to keep the algorithm differentiable and expect that the learned model will exhibit such behavior. ## model in this section we provide the complete detail about how the term vectors, the indicator variables and the correct answer choice is calculated using the tokenized input containing the qualitative knowledge and the multiple choice question. ## related work our work is related to all the works in neuro-symbolic reasoning @xcite @xcite @xcite @xcite ) that aims at implementing a symbolic theorem prover with neural networks. these works provides proof that more complicated symbolic reasoning algorithms than the one used in this work, can be implemented using neural nets. however the algorithms proposed in these work operates over symbolic input, which again calls for a parser. on the other hand several neural systems have been developed for constituency parsing @xcite , dependency parsing @xcite , semantic role labelling @xcite , parsing to the language of abstract meaning representation @xcite or task specific semantic parsing @xcite . these works also provide useful knowledge while constructing a deepekr solution. in this work, the input problem is translated to a set of fixed number of terms. however, depending on the end application the representation format could be a graph, stack, table. thus the work in graph neural networks @xcite , which operates over graphs or the neural state machine @xcite that operates over automata is also related to our work. in this work we have proposed to replace the symbolic representation by vectors so that dependency over an accurate parser can be avoided. with a similar goal, the work in @xcite proposes to use textual entailment to replace the parser. the central idea behind the proposal is, if the input is supposed to be translated to a predicate e.g., claima("protection","more","billy"), instead of asking the parser to translate it to the symbolic form, generate a textual description for the predicate e.g., "protection is more for billy" and use a textual entailment system to check if the input string entails it. a drawback of this approach is that generation of the textual description of a symbolic term currently requires handwritten templates. a system, namely gvqps @xcite following this approach has been built for the quarel dataset. our work is directly related to the quasp + system @xcite for quarel that trains a parser to obtain a symbolic representation of a qualitative word problem and uses a symbolic reasoner implemented in prolog to obtain the answer. our work is also related to the bert @xcite based multiple choice question solver that takes as input "[cls] knowledge [sep] question [sep] option x [sep]" and computes the score for option x. ## experiments we evaluate our system on the quartz and quarel dataset. the quartz dataset contains a total of 3864 problems. the train, dev and test split respectively contain 2696, 384 and 784 problems. the quarel dataset contains a total of 2771 problems. the train, dev and test split respectively contain 1941, 278 and 552 problems. we have used the bert-large-uncased-whole-word-masking model in our experimentation. performance on quartz ## conclusion knowledge representation and reasoning (kr) based solutions are interesting for natural language understanding as they are interpretable and can work with declarative knowledge. however, systems that implement kr solution with traditional parser and symbolic solvers normally fall short on performance when compared to neural systems. these observations and issues related to parser and symbolic reasoning have resulted in less interest towards kr solutions. however, we show that we can take a kr solution and implement in a way that is competitive with neural systems and is also explainable. for the qualitative word problems, the reasoning is fairly simple. our future work includes applying this method to other areas requiring more complex reasoning.