Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning (2022)

Abstract: Pretrained Language Models (LMs) have demonstrated ability to perform numerical reasoning by extrapolating from a few examples in few-shot settings. However, the extent to which this extrapolation relies on robust reasoning is unclear. In this paper, we investigate how well these models reason with terms that are less frequent in the pretraining data. In particular, we examine the correlations between the model performance on test instances and the frequency of terms from those instances in the pretraining data. We measure the strength of this correlation for a number of GPT-based language models (pretrained on the Pile dataset) on various numerical deduction tasks (e.g., arithmetic and unit conversion). Our results consistently demonstrate that models are more accurate on instances whose terms are more prevalent, in some cases above 70% (absolute) more accurate on the top 10% frequent terms in comparison to the bottom 10%. Overall, although LMs appear successful at few-shot numerical reasoning, our results raise the question of how much models actually generalize beyond pretraining data, and we encourage researchers to take the pretraining data into account when interpreting evaluation results.

## introduction large language models have demonstrated outstanding zero-and few-shot performance on various reasoning benchmarks @xcite . in particular, their high performance on numerical tasks, such as addition and multiplication, suggests that models may have learned the ability to perform the underlying reasoning operations simply through a combination of pretraining and model size @xcite . as these numerical reasoning tasks become increasingly prevalent for evaluating the performance of work done while robert logan was at uci. q: what is 24 times 18? q: what is 23 times 18? a: figure 1 : multiplication performance: plot of gpt-j-6b's 2-shot accuracy on multiplication (averaged over multiple multiplicands and training instances) against the frequency of the equation's term in the pretraining corpus. each point represents the average performance for that term (e.g., 24) multiplied by numbers 0-99 and 5 choices of random seeds. as in the example, the performance difference for the numbers 24 and 23 is more than 20%. we find a strong correlation between the accuracy for a number and its frequency in pretraining. for illustration, consider the arithmetic task of multiplying two integers (shown in figure 1 ). a model that has learned proper arithmetic skills should be able to answer the queries irrespective of the frequencies of the operands in the pretraining data. therefore, it should have roughly equivalent performance when answering the queries q: what is 24 times x? and q: what is 23 times x? despite the fact that 24 appears more frequently in the pretraining data. this is not the case with current lms and we will study the effect of frequency terms in details through this paper. to show the effect of frequency, in this example, we plot the average accuracy of gpt-j-6b @xcite on the numbers 0-99 (averaged over 0-99 as the other operand) against the frequency of the number in the pretraining data in figure 1 . we find a strong correlation between the term frequency and the model performance indicating that the model reasoning is not robust to these frequencies. note that even "rare" terms still appear on the order of millions of times in the pretraining data. in this work, we investigate this impact of the frequency of test instance terms in a model's pretraining data on the model's performance. we experiment on numerical reasoning tasks of addition, multiplication, and unit conversion. we count occurrences of the numbers and units in instances of these tasks in the pretraining data, including cooccurrences of term pairs or triples within a fixed window. this procedure allows us to aggregate over instances in which these terms appear and observe the relationship between term frequency and model accuracy on instances that include those terms. we summarize this behavior through the performance gap between instances that have the most frequent terms and instances that have the least frequent terms. intuitively, models that exhibit a high performance gap are much more accurate on instances that are more common in the pretraining data, suggesting that the model does not generalize appropriately and is affected by dataset overlap. we present analysis on these numerical reasoning tasks for three sizes of the eleutherai/gpt models pretrained on the pile dataset @xcite , which has been publicly released and thus permits this kind of analysis (in contrast to the data that, e.g., @xcite ) was trained on). our results consistently show a large performance gap between highest-and lowest-frequency terms; in some cases there is a more than 70% average accuracy gap between the top and bottom 10% terms. we also investigate whether this performance gap can be explained by strong memorization effects, i.e. by instances that are memorized by the language model. to achieve this, we remove instances that contain frequent combinations of numbers from our analysis, and study the performance on the remaining instances. even in this case, we still find a strong correlation between frequency of terms and average performance, indicating that our results cannot be explained solely by direct memorization. these observations suggest that any evaluation of reasoning that does not take the pretraining data into account is difficult to interpret, and that we need to revisit evaluation of language models with respect to their pretraining data before making any conclusion about the models generalization abilities beyond the pretraining data. ## background and methodology numerical reasoning has been essential part of complex multi-step reasoning tasks for natural language understanding @xcite . recently, large language models have exhibited an ability to perform numerical reasoning tasks in few-shot settings without requiring any modifications to their parameters through a method called in-context learning @xcite . our goal is to evaluate this reasoning skill in-depth and with respect to the pretraining data. this section provides background information on in-context learning and introduces our method for measuring the performance gap of the models on numerical reasoning tasks based on differences in pretraining term frequency. ## experiment setup in this section, we describe our setup to measure the effect of pretraining data on the few-shot evaluation of a number of numerical reasoning tasks for different language models. language models we experiment on the following models from eleutherai: gpt-j-6b @xcite , and gpt-neo-1.3b, gpt-neo-2.7b @xcite . these models are publicly available, but more importantly, they are among the few models that their pretraining corpus has also been released. these language models are trained on the pile dataset @xcite , a large-scale lan-guage modeling dataset consisting of english documents in 22 academic or other professional data sources. we count the frequency of all integers with less than seven digits using a slightly modified version of spacy english tokenizer @xcite . to calculate the frequencies of the numbers we use amazon elastic map reduce (emr) platform. we use the huggingface foot_1 transformer integration of the models for experiments. numerical reasoning tasks we create three types of datasets that target the mathematical capabilities of language models since solving mathematical questions is a useful reasoning capability of the models (brown et al., 2020). figure 3 : the gpt-j-6b accuracy on arithmetic and operator inference, with k shots. the average accuracy of the binned instances is highly correlated with their term frequencies ω {x1} in the pretraining corpus (x-axis). ## results with the three types of numerical reasoning tasks (consisting of 11 total datasets), we present an evaluation of the effect of pretraining term frequency on the performance of the language models. for each dataset, we measure the performance gap on instances that consist of rarer (relatively) terms, for a few different choices of what to compute frequency over (different combinations of the instance terms). we also investigate the effect of the model size on this performance gap and do a case study to further clarify if all this impact is due to memorization. arithmetic we first study the performance on simple addition and multiplication of numbers. the results for the gpt-j-6b model is provided in operation inference these tasks aim to assess the model capability to both infer the math operation and to perform the actual computation. as we see in time-unit conversion the performance gap evaluated on the time unit conversion experiments is in these results suggest that even though the model gets more accurate, the improvements focus on more frequent instances of the task. (example figures for time units experiments is provided in figures 4, 5) decades to years: as we observe in ## related work a large and growing body of literature has investigated a number of related concerns with large language models (for discussion of more tangentially related work see appendix a.1). ## discussion in this work, we consider how to conduct few-shot evaluations in light of the analysis with the pretraining data. prior work has attempted to control for overlap between pretraining data and the test instances, but as we have seen, those methods are insufficient. for example, brown et al. (2020) measure the impact of removing instances from evaluation datasets that share 13-gram overlap with their pretraining data on gpt-3's accuracy, and also argue that the low occurrence of exact phrases such as "num1 + @xmath3. however, we show that lm performance is impacted by much simpler statistical patterns, as small as unigram overlap with the pretraining data. for these reasons, we strongly recommend that evaluation of reasoning capabilities should take the pretraining corpus into account, and any claims of reasoning can only be made after demonstrating robustness to the effect of pretraining. current lm benchmarks, that are dissociated from the model's pretraining data, make it impossible to interpret few-shot reasoning performance results. it is worth mentioning that, even a performance gap of 0 is likely not sufficient to claim reasoning capabilities-what exactly constitutes "reasoning" remains ill-defined-but it may be a necessary condition, and one that current models do not meet. in this study, we are not making a causal claim, and in general, there may be confounders that we have not eliminated in our setting. recently, @xcite introduced a causal framework based on pretraining data statistics for understanding language model's factual predictions. to be able to use the causal inference techniques they construct and assume a causal graph for the task of extracting factual knowledge from pretrained language models. we recommend further research in the proposed direction for other nlp tasks such as reasoning and interventions during training to provide finer-grained analysis of the effect of pretraining. one potential concern is that our experiments do not distinguish whether incorrect answers are due to lack of reasoning or lack of recognition, i.e. it is possible that the model has the ability to multiply but the embeddings for rare terms are not adapted to that algorithm. however, recognizing numbers is a prerequisite to numerical reasoning, thus if the models lack the ability to identify numbers, this still means that they lack numerical reasoning skills. that said, we also suspect that the errors are not due to recognition. even the most infrequent terms in our experiments have been seen millions of times-they are not unknown tokens. ## conclusion we show that in-context language model performance on numerical reasoning tasks can be impacted significantly by low-order co-occurrence statistics in the pretraining data, raising questions on the extent to which these models are actually reasoning to solve these tasks. these observations suggest the necessity for reconsidering and redefining the reasoning evaluation schemes for the large language models. further characterizing the impacting factors on the models reasoning capacities is also an important tasks for the community. most importantly, we suggest that the community should not treat the pretraining data of the large language models as unknown black boxes. overlooking the impact of the pretraining data can be misleading in evaluating the model reasoning skills.