Improving Numerical Reasoning Skills in the Modular Approach for Complex Question Answering on Text (2021)

Abstract: Numerical reasoning skills are essential for complex question answering (CQA) over text. It requires opertaions including counting, comparison, addition and subtraction. A successful approach to CQA on text, Neural Module Networks (NMNs), follows the programmer-interpreter paradigm and leverages specialised modules to perform compositional reasoning. However, the NMNs framework does not consider the relationship between numbers and entities in both questions and paragraphs. We propose effective techniques to improve NMNs’ numerical reasoning capabilities by making the interpreter question-aware and capturing the relationship between entities and numbers. On the same subset of the DROP dataset for CQA on text, experimental results show that our additions outperform the original NMNs by 3.0 points for the overall F1 score.

## introduction complex question answering (cqa) is a challenging task, requiring a model to perform compositional and numerical reasoning. originally proposed for the visual question answering (vqa) task, neural module networks (nmns) @xcite have recently been adopted to tackle the cqa problem over text @xcite . the nmns is an end-to-end differentiable model in the programmer-interpreter paradigm @xcite . briefly, the programmer learns to map each question into a program, i.e. a sequence of neural modules, and the interpreter then "executes" the program, operationalized by modules, on the paragraph to yield the answer for different types of complex questions. nmns achieves the best performance on a subset of the challenging drop dataset @xcite and is interpertable by nature. however, nmns' performance advantage is not consistent, as it underperforms in some types of questions that require numerical reasoning. for instance, for date-compare questions, mtmsn @xcite achieves an f1 score of 85.2 foot_0 , whereas nmns' performance is 82.6. similarly, for count questions, the f1 score is 61.6 for mtmsn and 55.7 for nmns. this performance gap stems from two deficiencies of nmns, which we describe below with the help of two examples in figure 1 . firstly, nmns' interpreter is oblivious to the question when executing number-related modules. for executing number-related modules, the interpreter only receives the paragraph as input, but not the question. such a lack of direct interactions with the question impairs model performance: the entities in the question, which may also occur in the paragraph, can help locate significant and relevant numbers to produce the final answer. in the first example in figure 1 , if the interpreter is aware of the correct event mentioned in the question (i.e. "the constituent assembly being elected"), it can easily find the same event in the paragraph and further locate its date ("12 november") precisely. without this knowledge, the original nmns found the wrong event (i.e. "dissolved the constituent assembly"), thus the wrong date ("january 1918"), leading to an incorrect answer. secondly, nmns disregards the relative positioning of entities and their related numbers in the paragraph. although nmns can learn separate distributions over numbers extracted from a paragraph, it does not have an effective mechanism to identify the number that connects to a given entity. such an ability to recognise the association among numbers and entities is vital for learning numerical reasoning skills: the operation between numbers is meaningful only when they refer to the same entity or the same type of entities. the second example in figure 1 illustrates the positioning of entities and their related numbers. with only a constraint on a window around an entity, the nmns' interpreter tends to identify the nearest number as the related one to a given entity ("august 1996 to december 1997" for entity "puk and kdp later co-operated"), resulting in wrong predictions. question paragraph nmns answer our answer which event happened first, the constituent assembly being elected, or the elimination of hierarchy in the army? … on 12 november , a constituent assembly was elected. in these elections, 26 mandatory delegates were proposed by the bolshevik central committee and 58 were proposed by the socialist revolutionaries. of these mandatory candidates, only one bolshevik and seven socialist revolutionary delegates were women. ... the bolsheviks dissolved the constituent assembly in january 1918, when it came into conflict with the soviets. on 16 december 1917 , the government ventured to eliminate hierarchy in the army, removing all titles, ranks, and uniform decorations. … hierarchy in the army (incorrect) constituent assembly was elected (correct) what happened first: the u.s.-mediated washington agreement or puk and kdp later co-operated? in september 1998, barzani and talabani signed the u.s.-mediated washington agreement establishing a formal peace treaty. in the agreement, …, including the puk and kdp. the kdp estimated that 58,000 of its supporters had been expelled from puk-controlled regions from october 1996 to october 1997. the puk says 49,000 of its supporters were expelled from kdp-controlled regions from august 1996 to december 1997. the puk and kdp later co-operated with american forces during the 2003 invasion of iraq, … puk and kdp later cooperated (incorrect) the u.s.mediated washington agreement (correct) we propose three simple and effective mechanisms to improve nmns' numerical reasoning capabilities. firstly, we improve the interpreter to make it questionaware. by explicitly conditioning the execution on the question, the interpreter can exploit the information contained in the question. secondly, we propose an intuitive constraint to better relate numbers and their corresponding entities in the paragraph. finally, we strengthen the auxiliary loss to increase attention values of entities in closer vicinity within a sentence. experimental results show that our modifications significantly improve nmns' numerical reasoning performance by up to 3.0 absolute f1 points. with minor modification, these mechanisms are simple enough to be applied to other modular approaches. ## proposed model in this section, we will discuss the deficiencies of nmns described in section 1 and propose three techniques to overcome these problems. considering the importance of questions while executing programs, we incorporate a question-to-paragraph alignment matrix to form a question-aware interpreter in section 3.1. in section 3.2, the correspondence between numbers and their related entities is enhanced with a simple and effective constraint on number-related modules. in section 3.3, we strengthen the auxiliary loss function in nmns to further concentrate attention in the same sentence. ## conclusion neural moudule networks (nmns) represent an interpretable state-of-the-art approach to complex question answering over text. in this paper, we further improve nmns' numerical reasoning capabilities, by making the interpreter question-aware and placing stronger constraints on the relative positioning of entities and their related numbers. experimental results show that our approach significantly improves nmns' numerical reasoning ability, with an increase in f1 of 3.0 absolute points.