Numerical Correlation in Text (2022)

Abstract: Evaluation of quantitative reasoning of large language models is an important step towards understanding their current capabilities and limitations. We propose a new task, Numerical Correlation in Text, which requires models to identify the correlation between two numbers in a sentence. To this end, we introduce a new dataset, which contains over 2,000 Wikipedia sentences with two numbers and their correlation labels. Using this dataset we are able to show that recent numerically aware pretraining methods for language models do not help generalization on this task posing a challenge for future work in this area.

## introduction numerical reasoning tasks are one area where the performance of large language models (llms) has not improved as drastically @xcite as on other tasks. good performance is critical for many downstream applications in areas such as fact checking, question-answering, or search. different tasks have been proposed to evaluate the numerical reasoning capabilities of llms @xcite . we can analyze these tasks along two dimensions: diversity of knowledge required and how solvable the task is. higher diversity ensures better coverage across different domains while higher solvability yields more interpretable metrics. mathematical word problems (mwps) are written in a way that the text of the problem is always sufficient to determine the exact unique answer and are therefore highly solvable. however, they lack in diversity since many mwp datasets are constructed from templates or are even fully synthetic. in contrast, numerical cloze-style problems requires highly diverse knowledge since they can be easily formed from any text that includes numbers. 1 work completed during internship at salesforce research. please direct correspondence to: dspokoyn@cs.cmu.edu figure 1 : an illustrative plot of certain numerical evaluation tasks along the two dimensions of diversity and solvability. our aim with numerical correlation is for the task to be both diverse and solvable. a consequence of formulating cloze-style problems is that many texts do not provide sufficient information to determine the correct answer and have inherent uncertainty which results in a lower solvability. as an example from the numersense dataset @xcite , "some plant varieties can grow up @xmath0." in figure 1 , we show an illustrative plot of tasks along these two dimensions. a good numeracy evaluation task should be both diverse and solvable. in this work we propose numerical correlation in text, a new task that aims to retain both high diversity and high solvability. given two numbers in text the task is to predict whether the numbers are positively, negatively or not correlated. for example: "some plant varieties can grow up to 6 feet tall and require 20 liters of water a month". we expect a positive correlation between the height of the plant and the amount of water it would need. this shows the key insight that predicting the correlation relationship between two numbers is possible with- the president travels on average **30** times a year on air force one a boeing **747**. neutral 2. a **2** bedroom, **1800** square feet house is hard to find in this neighborhood. positive 3. to cook a 20 lb turkey place in the oven for **2** hours at **435** degrees. negative although correlation between two numbers can involve incredibly complex functions, we approximate the correlation to be linear and treat it as a three-way classification. we use a qualification task to select a group of amazon mechanical turk (amt) labelers and construct a dataset of wikipedia sentences which contain two numbers and their correlation relationship. we investigate the performance of four models: two general pretrained language transformers and two numerically aware models on our new dataset in a few-shot setting. when probed on the numerical correlation task we see that all models exhibit a plateau in their performance with only 6% of the training data. further all models underperform the human baseline in both the finetuning and linear probing setting. surprisingly, our results also indicate that existing numerically pretraining methods do not result in better performance on the numerical correlation task. ## experiments given a sentence x and two numbers y 1 and y 2 in the text, we define the task of predicting the correlation between the two numbers as a classification task with the label set @xmath1, n egative, n eutral}. we compare four models, two general pretrained language models (bart @xcite and roberta @xcite ) and two numerically aware models (gemm @xcite and genbert @xcite ). we conduct few-shot learning experiments where the model is trained on between 1% to 10% of the training data and the remaining data is split into a validation and test set evenly. we train all models with the adamw optimizer @xcite with a learning rate of 1e-5 and a batch size of 16. we use the majority vote labeling to choose the final label for each sentence in all subsequent experiments. 5 we report the test f1 scores averaged over 5 initialization seeds. ## conclusion we introduced a new task of predicting numerical correlation in text and build an annotated dataset to evaluate models on this task. using this dataset we show that pretrained language models have poor performance on this task and that current methods to add numerically aware pretraining to models are not effective. we identified that there exists a large gap between human performance and the best supervised model. in the future we hope to expand our annotation to include the slope of the correlation. we believe that predicting both the slope and correlation type of two numbers can be improve interpretability in numerical question answering and commonsense reasoning applications. in future work we also plan to expand the dataset to capture numerical correlation relationships in longer chunks of text such as paragraphs and documents.