Teaching Neural Module Networks to Do Arithmetic (2022)

Abstract: Answering complex questions that require multi-step multi-type reasoning over raw text is challenging, especially when conducting numerical reasoning. Neural Module Networks (NMNs), follow the programmer-interpreter framework and design trainable modules to learn different reasoning skills. However, NMNs only have limited reasoning abilities, and lack numerical reasoning capability. We upgrade NMNs by: (a) bridging the gap between its interpreter and the complex questions; (b) introducing addition and subtraction modules that perform numerical reasoning over numbers. On a subset of DROP, experimental results show that our proposed methods enhance NMNs’ numerical reasoning skills by 17.7% improvement of F1 score and significantly outperform previous state-of-the-art models.

## introduction complex question answering (cqa) over text is a challenging task in natural language understanding (nlu). based on the programmer-interpreter paradigm, neural module networks (nmns) @xcite learn to first parse complex questions as executable programs composed of various predefined trainable modules, and then execute such programs (implemented by modules) over the given paragraph to predict answers of all kinds. nmns achieve competitive reasoning performance on a subset of drop @xcite , and possess remarkable interpretability that is also important for cqa. however, nmns' numerical reasoning capability is insufficient: it is incapable of handling arithmetic operations such as addition and subtraction between numbers, which make up nearly 40% questions of the drop dataset. moreover, a gap exists between the interpreter and the complex question since there is no interaction between them. motivated by these, we propose two methods to improve nmns' numerical reasoning skills. first, we incorporate the original question in the interpreter, aiming to directly provide question information in the "execution" process, especially number-related questions. the intuition behind is that, in the original nmns, questions participate in the process only through the programmer. this can cause a distance between queries and returns. for example, in figure 1 , the first row shows that the original nmns found the wrong event (i.e., 'besieged sinj') solely based on the paragraph information. in contrast, our model @xmath0.e., 'sinj finally fell') with the help of question information. second, we introduce new modules to support addition and subtraction of up to three numbers. endowing nmns with the ability to support arithmetic can greatly boost its overall performance on drop and beyond. for instance, in figure 1 , the second row shows that the original nmns improperly adopt the find-num module for the addition question because the module set does not cover such an arithmetic ability. to facilitate the learning of the add/sub modules, we extract qa pairs related to addition and subtraction from the original drop dataset to construct a new dataset for training and evaluation. experimental results show that our methods significantly enhance nmns' numerical reasoning capability. on a subset of drop, our methods improve f1 score by 17.7% absolute points, and on add-sub questions by 65.7% absolute points. compared to @xcite , which is specifically designed for numerical reasoning, our method outperforms it by 2.9% absolute f1 points. question type paragraph question answers date compare in the morean war, the republic of venice besieged sinj in october 1684 and then again march and april 1685, but both times without success. … with the help of the local population of poljica as well as the morlachs, the fortress of sinj finally fell to the venetian army on 30 september 1686. on 1 september 1687 the siege of herceg novi started, and ended with a venetian victory on 30 september. … which happened first, the fell of sinj or the siege of herceg novi? original nmns: besieged sinj 𝑵𝑵𝑵𝑵𝑵𝑵 ± : sinj finally fell ground-truth: the fell of sinj / sinj finally fell original nmns' program: (span(compare-date-lt(find find))) 𝑵𝑵𝑵𝑵𝑵𝑵 ± : (span(compare-date-lt(find find))) addition / subtraction (2 numbers) … in the first quarter, the niners struck first as kicker joe nedney got a 47-yard field goal. in the second quarter, the saints took the lead with qb drew brees completing a 5-yard and a 33-yard td pass to wr lance moore. san francisco would answer with nedney's 49-yard field goal, yet new orleans replied with brees' 47-yard td pass to wr robert meachem. … how many yards was nedney's combined field goal yards in the first and second quarters? original nmns: 47 𝑵𝑵𝑵𝑵𝑵𝑵 ± : 96 ground-truth: 96 original nmns' program: (find-num (filter (find)) 𝑵𝑵𝑵𝑵𝑵𝑵 ± : (addition(find-num(find))(find-num(find))) addition / subtraction (3 numbers) the greek census 2011 recorded 9,903,268 greek citizens (91.56%), 480,824 albanian citizens (4.44%), 75,915 bulgarian citizens (0.7%), 46,523 romanian citizenship (0.43%), 34,177 pakistani citizens (0.32%), 27,400 georgia (country) citizens (0.25%) and 247,090 people had other or unidentified citizenship (2.3%). … how many more people were greek citizens compared to albanian and bulgarian citizens combined? original nmns: 9903268 𝑵𝑵𝑵𝑵𝑵𝑵 ± : 9346529 ground-truth: 9346529 original nmns' program: (find-num (find-max-num (find))) 𝑵𝑵𝑵𝑵𝑵𝑵 ± : (subtraction (find-num(find)) (addition(find-num(find))(find-num(find)))) approach, a programmer maps natural-language questions into machine-readable representations (logical forms), which are executed by an interpreter to yield the final answer. for instance, wnsmn @xcite uses a generalized framework of dependency parsing inspired by the stanford dependency parse tree @xcite to parse queries into noisy heuristic programs. neural module networks @xcite extend semantic parsing by making interpreter a learnable function with specified modules and executing the logical forms from the programmer in a step-wise manner. neural module networks initially is proposed to overcome the visual question answering (vqa) problem @xcite , where questions are often compositional. @xcite employs the programmer-interpreter framework with attention @xcite to tackle the cqa task. specifically, the programmer parses each question into an executable program. the interpreter takes the program as input and perform various symbolic reasoning functions. the modules are defined in a differentiable way, aiming to maintain the uncertainty about each intermediate decision output and propagate them through layers. for instance, the predicted program of the first example in figure 1 is span(compare-date-lt(find,find)). the interpreter would first calls the find module twice to find events queried by the question (e.g. ## model in this section, we tend to illustrate our proposed methods. basically, we will show the incorporation of questions in section 3.1. in section 3.2, the newly extended module: addition and subtraction will be described. ## conclusion in this work, we extend nmns' numerical reasoning capability to 2-/and 3-number addition and subtraction, and incorporate the influence of question information to the interpreter on number related questions. experimental results show that our methods significantly enhance nmns' numerical reasoning ability, with an increase of 17.7% absolute f1 points on a newly constructed drop subset that includes arithmetic questions. moreover, our approach also outperforms numnet, a sota numerical reasoning model, by 2.9% f1 points.