Exploiting Numerical-Contextual Knowledge to Improve Numerical Reasoning in Question Answering (2022)

Abstract: Numerical reasoning over text is a challenging subtask in question answering (QA) that requires both the understanding of texts and numbers. However, existing language models in these numerical reasoning QA models tend to overly rely on the pre-existing parametric knowledge at inference time, which commonly causes hallucination in interpreting numbers. Our work proposes a novel attention masked reasoning model, the NC-BERT, that learns to leverage the number-related contextual knowledge to alleviate the over-reliance on parametric knowledge and enhance the numerical reasoning capabilities of the QA model. The empirical results suggest that understanding of numbers in their context by reducing the parametric knowledge influence, and refining numerical information in the number embeddings lead to improved numerical reasoning accuracy and performance in DROP, a numerical QA dataset.

## introduction understanding numbers in text is critical when dealing with numerical reasoning problems over text. most previous works @xcite @xcite @xcite @xcite on such numerical reasoning over text have shown substantial amounts of performance gain in numerical question answering (qa) tasks such as drop @xcite . while these models display stellar performance, previous studies that evaluate model's numerical reasoning robustness @xcite @xcite suggest that these qa models that typically depend on large language models (lm) such as bert @xcite and roberta @xcite suffer from a limitation: the disregard of context information at inference time for numerical reasoning. question : "which age group makes up the largest percentage of population?" passage : "14.3% … are in their 20s, … 14.9% in their 40s and 13.0% are in their 50s. ## approach in this work, we propose an attention masked question answering (qa) model, namely the numerical-contextual bert (nc-bert), that learns to rely on the contextual knowledge and exhibits improved numerical reasoning capability in qa. first, we construct a tripartite attention mask (i.e., nc-mask) over a self-attention layer on top of the encoder to direct the flow of necessary contextual information to the number embeddings, so that number-related contextual knowledge such as entity-number and type-number relations are effectively leveraged during model training. we then adopt a complementary numeracy-inducing regularizer to counteract the numeracy dilution issue caused by our masking strategy and to further enhance the accuracy of numerical calculation. ## experiments to validate the proposed approach, we establish the two following research questions: q1. "does number-related contextual knowledge help improve numerical reasoning?" q2. "does compensating for the diluted numeracy give more accurate numerical calculation?" q1 is assessed by the channel-wise application of the nc-mask on our model encoder. by evaluating each channel and its effect on the model's reasoning capability, we reveal how the numberrelated contextual knowledge influence the baseline's numerical reasoning. we also determine the optimal placement of the nc-mask by exploring where in the encoder should the mask be applied for better numerical reasoning. q2 is evaluated by applying dice-loss and thereby replenishing the diluted numeracy of the number embeddings. in addition, we experiment with the least significant digit first (lsdf) generation scheme on our encoder-decoder architecture. the goal is to explore the idea of taking the "carry" into account to bring about an additional benefit to the overall numerical reasoning capability of our model (details in the appendix). ## results in this section, we explain the results of our nc-mask scheme and complementary numeracy regularizer by comparing the per answer-type exact match (em) and f1 scores. the results also include head-and layer-wise probing done to determine the optimal masking position in the model. ## conclusion this work proposes a novel attention masking scheme, nc-mask, to relieve question answering (qa) models of the language model (lm) encoder's over-reliance on parametric knowledge and improve the numerical reasoning accuracy and robustness. our analyses and empirical results provide strong evidence that bert, a commonly used encoder in qa models, needs to employ the extra attention channels to leverage the number-related contextual knowledge for robust numerical reasoning instead of entirely relying on the inherent selfattention mechanism. by additionally adopting a numeracy-inducing regularization term, our work also shows that the proposed masking scheme and regularization are complementary, and retaining numeracy is essential for accurate numerical calculation. future efforts should focus on increasing the scale of models with the masking scheme, since masked, attention-constrained layers appear to more positively contribute to model's reasoning capability than the addition of fully self-attentive layers.