A Neural-Symbolic Approach to Natural Language Understanding (2022)

Abstract: Deep neural networks, empowered by pre-trained language models, have achieved remarkable results in natural language understanding (NLU) tasks. However, their performances can drastically deteriorate when logical reasoning is needed. This is because NLU in principle depends on not only analogical reasoning, which deep neural networks are good at, but also logical reasoning. According to the dual-process theory, analogical reasoning and logical reasoning are respectively carried out by System 1 and System 2 in the human brain. Inspired by the theory, we present a novel framework for NLU called Neural-Symbolic Processor (NSP), which performs analogical reasoning based on neural processing and logical reasoning based on both neural and symbolic processing. As a case study, we conduct experiments on two NLU tasks, question answering (QA) and natural language inference (NLI), when numerical reasoning (a type of logical reasoning) is necessary. The experimental results show that our method significantly outperforms state-of-the-art methods in both tasks.

## introduction natural language understanding (nlu) has made remarkable progress recently, when pre-trained language models such as bert @xcite , roberta @xcite , and bart @xcite are exploited. the deep neural networks based on pre-trained language models even exhibit performances superior to humans in the tasks of question answering (qa) @xcite and natural language inference (nli) @xcite . however, language understanding requires not only analogical reasoning, which deep neural networks are good at @xcite , but also logical reasoning, including numerical reasoning. for example, to answer the questions in according to the dual-process theory developed by the psychologist @xcite , human thinking is carried out by two different systems in the brain. system 1 is a fast, unconscious, and effortless mode of thinking, often associated with analogical reasoning. in contrast, system 2 is a slow, conscious, and effortful mode of thinking, also evoking logical reasoning. the theory should also hold for human language understanding, a specific case of thinking. inspired by the theory, we propose a new framework for language understanding, which performs both analogical reasoning and logical reasoning, corresponding to system 1 and system 2 respectively. in fact, designing ai systems containing system 1 and system 2 is a popular research topic recently (e.g., @xcite ). our key idea is to employ a neural network to conduct analogical reasoning on the text input as usual and in the meantime, to employ another neural network to translate the text input into a program and a symbolic system to execute the program to perform logical reasoning. the new framework for natural language understanding, called neural-symbolic processor (nsp), is shown in figure 1 . first, the encoder transforms the input texts (the question and text in qa, the two texts in nli) into a text embedding (an intermediate representation). then, the predictors take the text embedding as input and generate neural predictions. in parallel, the decoders transform the text embedding into programs. note that the encoder and the decoders form sequence-to-sequence models. moreover, the executors take the programs as input and generate symbolic predictions. finally, the mixture-of-experts (with a gating network) takes all the neural predictions and symbolic predictions as input and selects one of the predictions to make the final prediction. we evaluate the nsp framework in qa and nli. for the qa task, we sample a subset of drop (about 32k instances), referred to as drop-subset, and annotate a program for each question-answer pair requiring numerical reasoning. the experiments on the drop-susbset show that our approach outperforms the baselines by 2.40% and 2.51% in terms of f1 score and exact match. in particular, for question-answer pairs requiring multiplication, division, and averaging, our approach improves f1 by 56.42%. for question-answer pairs requiring addition and subtraction, our approach improves f1 by 3.91%. for the nli task, we use the awpnli dataset @xcite and also annotate programs for each text pair. the experiments on awpnli show that our approach exceeds the baseline with a large margin of 20.7% in terms of f1 score. the contribution of our work is as follows: ## neural-symbolic processor we describe the proposed framework neural-symbolic processor (nsp) in this section. ## conclusion this paper proposes a novel framework for natural language understanding (nlu), referred to as neural-symbolic processor (nsp). nsp employs two types of reasoning, analogical reasoning and logical reasoning. to 'understand' language, ana-passage & question prediction complicated multi-step computation passage: ... in the second quarter@n4, new orleans regained the lead as qb drew brees ( a former charger ) completed a 12 -yard@n5 td pass to wr devery henderson ( with a failed pat ) and rb deuce mcallister getting a 1 -yard@n6 td run. san diego answered as qb philip rivers completed a 12 -yard@n7 td pass to rb ladainian tomlinson, but the saints replied with brees completing a 30 -yard@n8 td pass to wr lance moore. the chargers closed out the half with rivers completing a 12 -yard@n9 td pass to te antonio gates. in the third quarter@n10, new orleans increased its lead brees completing a 1 -yard@n11 td pass to te mark campbell ... san diego tried to rally as kaeding nailed a 31 -yard@n16 field goal, rivers completed a 14 -yard@n17 td pass to wr vincent jackson question: how many more yards of touchdown passes did drew brees make than philip rivers? prediction: diff(add(n5,n8,n11,n17),add( n7,n9)) result: 33 manual: diff(add(n5,n11,n8),add(n7,n9,n17)) ground truth: 5 counting long strings passage: ... they explain that the gap may persist due to the crack epidemic, the degradation of african -american family structure, the rise of fraud in the educational system ( especially with respect to no child left behind ), the decrease in unskilled real wages and employment among african -americans due to globalization and minimum wage increases, differences in parental practices ( such as breastfeeding or reading to children ), and " environmental conditions shaped by [ african -americans ] themselves. ... question: how many reasons are cited as causing the persistent gap between white and black iqs? prediction: questions with complex conditions passage: ... yet the raiders would answer with kicker sebastian janikowski getting a 33 -yard@n5 and a 30yard@n6 field goal. houston would tie the game in the second quarter@n7 with kicker kris brown getting a 53yard@n8 and a 24 -yard@n9 field goal. ... question: how many field goals did both teams kick in the first@q0 half ? prediction: count(n5,n6,n8,n9) result: 4 manual: count(n5,n6) ground truth: 2