Improving compositional generalization for multi-step quantitative reasoning in question answering (2022)

Abstract: Quantitative reasoning is an important aspect of question answering, especially when numeric and verbal cues interact to indicate sophisticated, multi-step programs. In this paper, we demonstrate how modeling the compositional nature of quantitative text can enhance the performance and robustness of QA models, allowing them to capture arithmetic logic that is expressed verbally. Borrowing from the literature on semantic parsing, we propose a method that encourages the QA models to adjust their attention patterns and capture input/output alignments that are meaningful to the reasoning task. We show how this strategy improves program accuracy and renders the models more robust against overfitting as the number of reasoning steps grows. Our approach is designed as a standalone module which can be prepended to many existing models and trained in an end-to-end fashion without the need for additional supervisory signal. As part of this exercise, we also create a unified dataset building on four previously released numerical QA datasets over tabular data.

## introduction any natural language system that processes or interacts with numeric data requires quantitative reasoning to function. this has inspired research in several nlp domains, including reading comprehension @xcite , textual entailment @xcite , data-to-text generation @xcite , and question answering @xcite . a major challenge in quantitative reasoning is the interplay between numeric expressions and natural language @xcite . standard neural approaches rely heavily on lexical matching, leading to overfitting over spurious verbal patterns. in contrast, a purely symbolic approach excels at numerical reasoning, but struggles when sophisticated verbal reasoning is required @xcite . in this paper we introduce a novel attention strategy that captures the interplay between numeric and verbal modalities, which improves program accuracy and renders models more robust to overfitting. focusing on the question answering task, we show how our proposed method, named com-paqt (compositional attention for quantitative reasoning), enables the model to attend to relevant parts of text at each reasoning step. compaqt enhances the performance of state of the art models on several recently released qa datasets, especially for multi-step programs. it is implemented as a plug-and-play module that can be added to existing models with minimal effort and without the need for any additional supervision. concretely, we offer the following contributions: ## related work studies that have tackled quantitative reasoning in qa tasks fall into two categories. some studies have explored quantitative reasoning for answering questions over real-world data such as statistical records @xcite , wiki entries @xcite , enterprise documents @xcite , and financial reports @xcite . since numeric data is very often expressed in tabular structures, this category often involves question answering over tabular data, or hybrid table/text passages. other studies have explored math word problems @xcite , which require modeling abstractions and mapping arithmetic logic between language and math symbols @xcite . ## problem statement in the retriever-generator configuration of a qa model, our goal is to improve quantitative reasoning in the generator component. figure 1 illustrates the typical architecture of a generator as an encoder-decoder model. the encoder uses a contextual representation model such as roberta (liu al., 2019 ). the decoder combines a recurrent module with one or more cross-attention heads between the natural language input and the program output. as the output is generated step by step, it is crucial for the cross-attention module(s) to capture relevant components of the input, otherwise they can simply memorize spurious verbal patterns and fail to generalize, especially as the number of steps grows in the output. many neural models struggle to exhibit the same behavior, due to overfitting to spurious patterns in natural language, or in the output. as we will later discuss in section 5.1, quantitative reasoning datasets can exhibit a long-tail distribution, biased towards simpler patterns. figure 2 illustrates how this phenomenon takes place in the training split of one such dataset. the figure shows the prominence of the most common sequences of arithmetic operators in the finqa training set @xcite . as the number of steps grows, the tail grows longer and the sample size smaller, thus providing less information to the model and forcing it to rely on repetitively encountered patterns in the past. our goal is to encourage the model to focus its attention on relevant components of the input during generation. figure 3 illustrates the expected attention patterns for the fourth example from using this insight, we encourage the model to adjust its attention patterns accordingly. ideally, we would like to implement a cross-attention mechanism that mimics the alignments shown in figure 3 . however, doing so would require additional training data with gold alignments between the input and the output, akin to @xcite . to avoid this, instead of using a cross-attention module, we propose a self-attention module applied to the input alone. the output of this self-attention module will be served as input to any downstream cross-attention modules. the self-attention module primes the input in such a way that downstream cross-attention can achieve the desired alignments. this means that the self-attention module attends most to tokens in the input that are non-repetitive (in figure 3 these would be non-bold tokens such as "percent" or "change"). we hypothesize that this would make it easier for any downstream cross-attention module to distinguish between primed tokens and unprimed tokens, and use this information to distribute its attention mass more accurately, ignoring non-operative tokens. the remaining sections describe our methodology and experimental results in detail. ## methodology let q be a question made up of a sequence of tokens q 1 , • • • , q n . let f be the evidence obtained by the retriever, made up of a sequence of tokens f 1 , • • • , f m . note that the evidence can be composed of one or more concatenated facts, as illustrated in given a retriever-generator model, we prepend a self-attention module to the generator, as illustrated by the red dotted box in figure 1 . first, we encode q||f using a contextual embedding model such as roberta @xcite with embedding size d enc . this results in an embedding matrix @xmath0. at each generation step i, we apply scaled dot-product self-attention @xcite to u , resulting in the attention grid n+m) and the attention output n+m) . our goal is to encourage a (i) to focus its alignments properly, such that x (i) supplies relevant information to the generator. we follow a similar strategy to @xcite , but in the absence of gold alignments, use the heuristics described in section 3. concretely, we add the below term to the loss: for instance after the model attends to the word "change" in order to generate subtract, it learns to shift its focus away. the linear decay term helps the model assign a higher penalty to earlier generation steps. suppose the gold program is: in the absence of linear decay, both predictions would receive the same penalty. the decay term assigns a lower penalty to the first prediction, since it gets the first operation correct. finally, the alignment loss can be added to the default loss of the generator: in this section, we describe our experimental set up, including the datasets and the baseline models. ## results and discussion in this section, we investigate the effectiveness of compaqt through four questions: additionally, we examine whether the combined dataset offers an advantage over the largest constituent dataset. ## conclusion in this study, we proposed a method to improve multi-step quantitative reasoning for question answering. our method facilitates compositional generalization by encouraging the model to attend to relevant components of the input at each generation step. we demonstrated the effectiveness of our approach over four recently released tabular qa datasets. our method, named compaqt, was able to significantly improve program accuracy on three of the datasets, especially for multi-step programs. we also created a collection of qa samples for multi-step quantitative reasoning, by combining the datasets and unifying their format. in future studies, we hope to explore data-native approaches to quantitative reasoning such as augmentation and synthesis, as well as approaches embedded in symbolic reasoning @xcite .