Have You Seen That Number? Investigating Extrapolation in Question Answering Models (2021)

Abstract: Numerical reasoning in machine reading comprehension (MRC) has shown drastic improvements over the past few years. While the previous models for numerical MRC are able to interpolate the learned numerical reasoning capabilities, it is not clear whether they can perform just as well on numbers unseen in the training dataset. Our work rigorously tests state-of-the-art models on DROP, a numerical MRC dataset, to see if they can handle passages that contain out-of-range numbers. One of the key findings is that the models fail to extrapolate to unseen numbers. Presenting numbers as digit-by-digit input to the model, we also propose the E-digit number form that alleviates the lack of extrapolation in models and reveals the need to treat numbers differently from regular words in the text. Our work provides a valuable insight into the numerical MRC models and the way to represent number forms in MRC.

## introduction the research in question-answering (qa) models that are able to perform reading comprehension and discrete reasoning over numbers in the passage has seen significant progress, like the models in drop @xcite @xcite . despite their ability to understand the complex context and numbers within, none of them deal with the notion of whether these models can robustly handle "unseen" numbers during testing. the ability to extend discrete, symbolic rules such as addition and subtraction on numbers outside what we already know is called extrapolation, and this is an essential part of human intelligence. for example, if one can reason on numbers over text that range between 0 and 100, it is logically reasonable to infer that it should be able to handle numbers larger than as of the census of 2000, there were 49,927 people, 18,009 households, and 12,192 families residing in the county. the population density was 48 people per square mile (19/km²). as of the census of 2000, there were 4,992,700 people, 1,800,900 households, and 1,219,200 families residing in the county. the population density was 4,800 people per square mile (19/km²). answer: 80,128 answer: 8,012,800 figure 1 : an example of drop data instance perturbation to test the numerical qa models for extrapolation capabilities. this figure shows cardinal-type numbers factored by 100. we also note that the models for drop typically use transformer models as the encoder for context understanding. as shown in @xcite ; @xcite , subword tokenization methods arbitrarily subdivide the numbers and cause two very similar numbers to be in two disparate forms. this observation is in line with @xcite 's conclusion that how the numbers are presented to the model, or their surface form, influences the modeling of numbers. the surface forms proposed in @xcite provide digitplace information with a special set of tokens (the first three surface forms in figure 2 ), to increase model's accuracy in a simple addition task. however, they fail to imbue the extrapolation capability in their tested models, observing that the addition rules could not be extended beyond the length of numbers seen during training. therefore, we propose a new surface form called e-digit (figure 2 ) that addresses the lack of extrapolation capability in the models. our e-digit method successfully generalizes to out-of-distribution numbers and outperforms all the other surface forms by a significant margin. ## related work previous works like numnet @xcite attempt to tackle the drop task by using graphs to imbue the model with the relative magnitude information. genbert @xcite pre-trains bert @xcite with synthetic number and text data. qdgat @xcite designs a graph neural network with fully-connected number nodes of same entity type. while there are many other related works on this topic @xcite @xcite @xcite , none of them address the problem of extrapolation in drop. although @xcite reveals that naqanet @xcite struggles to deal with numbers outside the training range, showing a drop in performance in the extrapolation setting, they simply treat it as one of the failure modes in naqanet and provide no further analysis on this alarming issue on model reliability. a survey on numerical representations @xcite "while 2015 estimates place the median household income for cooke county at $53,552, past estimates showed the median household income … " a recent study by @xcite on the changes in how a number is presented to a model shows that different surface forms have significant influence on t5 @xcite in solving a simple arithmetic task. however, their proposed surface forms fail to extrapolate. they also explicitly provide the arithmetic operators and do not require complicated textual understanding for discrete reasoning. this begs the question of whether the same approach is viable in drop, where reasoning is done across multiple sentences, requires dealing with heterogeneous type numbers, and the operations should be inferred from the text. ## empirical investigation on extrapolation we first seek to determine whether the state-ofthe-art models on drop @xcite can extrapolate their numerical reasoning capabilities to unseen numbers during inference. dataset drop is a reading comprehension benchmark that requires models to perform a set of discrete reasoning such as counting, sorting and basic arithmetic operations. in this work, we construct the extrapolated version of drop evaluation set by perturbing numbers with addition and multiplication of pre-defined numbers as in figure 1 . then, we test the existing models for their extrapolation capabilities with these variant datasets. data perturbation prior to constructing the evaluation datasets, we use a named entity recognition (ner) system 1 to extract and identify seven different entity types among the numbers in the text, namely: ordinal, date, quantity, cardinal, money, time, percent. among these seven entity types, we apply the aforementioned extrapolation perturbation to quantity, cardinal and money only, because date, percent, time and ordinal require typespecific, handcrafted perturbations. for instance, if we were to perturb, "king james was born in may 25, 1926", it is not possible to simply change the range of "25" and "1926" by multiplying a 100, which neglects the entity-specific characteristics and requires question-level adjustment. since we are probing the models to evaluate their extrapolation capability on unseen numbers, changing the range of the three types suffices. we use the four versions of extrapolated drop evaluation set to observe the changes in performance along with the magnitude of changes in number range: add(10), add(100), factor(10), factor(100). add(n) means adding n and factor(n) means multiplying n to the numbers that appear in the passage. the numbers from the passage, question and answer are perturbed with one of the four perturbation schemes above. naturally, by the distributive law, the validity of the perturbed answer value holds. for example (see figure 1 ), applying factor(100) 1 stanford's stanza toolkit for nlp @xcite figure 3 : money type number distribution in the drop train and evaluation datasets. bin width is set to 50 with the numbers shown up to 80th percentile for visibility. the numbers are highly skewed to right. to a sequence, 49,927 + 18,009 + 12,182 = 80,128, results in 100 * (49,927 + 18,009 + 12,182) = 100 * (80,128). the same rule applies to other perturbation methods. as for the count-type answers that consist of numbers, we apply a heuristic where we consider both the number answers within the range of 0 to 9 and their extrapolated variants as answers. this prevents accidental perturbation of count-type answers and also considers arithmetic-type answers that have been extrapolated. ## injecting inductive bias on numbers with surface form representations after revealing the lack of extrapolation capability in the models, we gauge the influence of different surface forms of numbers as input to the mrc models. based on the observation on the importance of surface forms in arithmetic word problems (awp) @xcite , we evaluate if altering the surface form representation of numbers in drop alleviates the performance discrepancy shown in ## conclusion in this work, we investigated the extrapolation problem in complex numerical reasoning over text. our probing results shed light on the significant lack of drop models' capabilities by simulating a more realistic and ultimately needed benchmark (i.e., extrapolation). one of the key findings is that treating numbers as words inevitably requires a vast coverage of numbers, leading to sample inefficiency. this motivated us to adopt a more generalizable surface form representation, proposing the e-digit method that successfully generalizes to unseen numbers. empirical results highlight simple surface representations benefit the model with digit information for extrapolation, and our e-digit method effectively generalizes it further. our work opens up a new research direction in numerical reasoning over text on how to reduce the discrepancy between the original and extrapolated settings.