Masked Measurement Prediction: Learning to Jointly Predict Quantities and Units from Textual Context (2022)

Abstract: Physical measurements constitute a large portion of numbers in academic papers, engineering reports, and web tables. Current benchmarks fall short of properly evaluating numeracy of pretrained language models on measurements, hindering research on developing new methods and applying them to numerical tasks. To that end, we introduce a novel task, Masked Measurement Prediction (MMP), where a model learns to reconstruct a number together with its associated unit given masked text. MMP is useful for both training new numerically informed models as well as evaluating numeracy of existing systems. To address this task, we introduce a new Generative Masked Measurement (GeMM) model that jointly learns to predict numbers along with their units. We perform fine-grained analyses comparing our model with various ablations and baselines. We use linear probing of traditional pretrained transformer models (RoBERTa) to show that they significantly underperform jointly trained number-unit models, highlighting the difficulty of this new task and the benefits of our proposed pretraining approach. We hope this framework accelerates the progress towards building more robust numerical reasoning systems in the future.

## introduction many natural language processing tasks require a deep understanding of numbers -for example, reading comprehension @xcite , textual entailment @xcite and hybrid table tasks such as fact-verification @xcite or question answering @xcite . masked number prediction (mnp) is a popular pretraining objective to imbue language models with numerical understanding and evaluate existing models for their numerical capacity. we display the fixed operations used during unit conversion in yellow. in black, we show the different components of the model's prediction. as an example of mnp, given the sentence "cats have [#num] paws." a model learns to predict the number 4. while appropriate for numerical commonsense, mnp is deficient when it is used to predict measurements. measurements, such as 2 meters or 13.2 square miles, are a special class of particularly common numbers in text that have a well-defined and typed system of units. given a simple question: "how long did alex honnold climb for?", a single number alone is an insufficient answer since it is meaningless without the unit. answers like 1000 meters or 4 hours could both suffice. current mnp systems do not jointly reason about numbers with units. it is reasonable to expect that pretrained models like bert could leverage information of units directly as text without any special treatment. however, in preliminary experiments we find that this yields poor numerical abilities (see appendix b). furthermore, including units as text directly raise more questions: should we evaluate using all units (meters, feet, inches)? should we equally weight across the units? current models have no opinion about which unit is appropriate because they are not required to make unit predictions during training. together, this indicates that current training objectives do not capture sufficient representations of measurements and that a direct application of mnp to evaluate numeracy of measurements is ill-suited. to address these shortcomings, we propose the more challenging task of masked measurement prediction (mmp) along with a new model. in this task, a model must reconstruct both the number together with the correct unit. in figure 1 we show how in a mmp setting our model generates a dimension ("length"), a number in metric log-space ("3.00"), the unit ("feet") and then uses the conversion factor ("3.28") to deterministically output the full measurement ("3280 feet"). this example illustrates a key distinction in that our model is flexible and can generate non-metric measurements (feet) but evaluates numerical prediction in canonical units (meters). 2 mmp is useful for two reasons: 1) as a way to train models to give them better numeracy 2) as a new kind of evaluation that allows for a much more fine-grained analysis of reasoning over numerical quantities. the task of measurement estimation decouples the different aspects of numeracy allowing for a more interpretable and thorough analysis of numerical reasoning. we introduce a new evaluation benchmark for mmp based on wiki-convert (wico) @xcite , a large scale dataset of english wikipedia sentences with ground truth measurement annotations. we compare the performance of our models on their ability to accurately predict the dimension, unit, and value of a measurement. we employ a large pretrained transformer model as our textual encoder and examine the performance of different discriminative, generative, and latent variable models along with several ablations. our contributions are as follows: furthermore there are numerous applications of better measurement prediction and unit reconstruction such as in table to text generation @xcite , answering numerical queries @xcite or for improving e-commerce product search @xcite . we hope that masked measurement prediction becomes a standard benchmarking tool from which we can gain insight how to best incorporate new numeracy modeling techniques as well as evaluate existing models. ## dataset we train and evaluate our models on wico @xcite , a dataset of english wikipedia sentences where the number and unit in each sentence are human-annotated. we canonicalize the units and map each to a single dimension. for example both feet per second and miles per hour map to velocity. we show the distribution of all measurements and lengths in figure 3 . the resulting dataset consists of 919,237 sentences with annotated (number, unit, dimension) triples. we provide more details on the data in appendix a. ## experiments we train all models using a batch size of 200 and use the adamw @xcite optimizer with a learning rate of 1e -4 and a linear warm-up schedule of 500 steps. we use the " " symbol to indicate that we freeze the transformer parameters for training. for all frozen models we use a log frequency weighted cross-entropy due to the highly imbalanced classes as well as a higher to evaluate the performance of our models, we report the macro averaged f1 score for dimension and unit prediction and log-mae to evaluate number prediction. we define log-mae in equation 2 where y is the predicted number and y * is the ground truth number. as a simple baseline for dimension and unit prediction, we employ majority class voting. for number prediction we use the median of all the numbers in the training set. ## limitations the pretrained roberta model we use in experiments was likely pretrained on data that included wico. thus, it is reasonable to be concerned about inflated test performance. that said, the task we consider is distinct from the selfsupervised task used to pretrain roberta (i.e. masked word classification vs. masked number regression). further, our experiments on directly probing roberta to predict masked numbers and units showed poor performance -indicating, perhaps, that even if roberta's pre-training set did include wico, roberta did not memorize aspects of our test set relevant to masked number prediction, partially mitigating these concerns. the human evaluation studies we conducted are a quite limited 'guesstimating' task. the human annotators were not allowed to use any external information from searching the internet or looking up answers in knowledge-bases. their total average completion time per question was 33 seconds. furthermore, many annotators may not have strong intuition about measurements with unfamiliar and uncommon unit types. for these reasons it is not surprising that our models outperform the human annotators in this limited experiment. however, these human evaluation studies do help calibrate the difficulty of the mmp task on wico. ## conclusion in this work we propose masked measurement prediction, a new task that requires models to jointly predict masked numbers and units in running text. we motivate this task as an important extension of existing masked number-only prediction tasks that addresses their limitations and allows for better evaluation of numeracy in nlp models. in our study, we show that probing of traditional pretrained transformers exposes a gap in their understanding of contextualized quantities. through careful quantitative and qualitative analysis of our new model, which directly reasons about underlying units and dimensions, we find that it is possible to learn good representations of measurements. for future work we aim to extend this dataset to cover more existing standardized units from organizations such as unece. 4 we hope our mmp task encourages research into further development of better numeracy methodologies.