Masked Measurement Prediction: Learning to Jointly Predict Quantities and Units from Textual Context
Abstract
Physical measurements constitute a large portion of numbers in academic papers, engineering reports, and web tables.
Current benchmarks fall short of properly evaluating numeracy of pretrained language models on measurements, hindering research on developing new methods and applying them to numerical tasks.
To that end, we introduce a novel task, Masked Measurement Prediction ( MMP ), where a model learns to reconstruct a number together with its associated unit given masked text.
MMP is useful for both training new numerically informed models as well as evaluating numeracy of existing systems.
To address this task, we introduce a new Ge nerative
M asked M easurement ( GeMM ) model that jointly learns to predict numbers along with their units.
We perform fine-grained analyses comparing our model with various ablations and baselines.
We use linear probing of traditional pretrained transformer models (RoBERTa) to show that they significantly underperform jointly trained number-unit models, highlighting the difficulty of this new task and the benefits of our proposed pre-training approach.
We hope this framework accelerates progress towards building more robust numerical reasoning systems in the future.
Introduction
Many natural language processing tasks require a deep understanding of numbers – for example, reading comprehension (Ran et al., 2019), textual entailment (Sammons et al., 2010; Roy, 2017) and hybrid table tasks such as fact-verification (Chen et al., 2020) or question answering (Chen et al., 2021).
Masked number prediction ( MNP ) is a popular pretraining objective to imbue language models with numerical understanding and evaluate existing models for their numerical capacity.
As an example of MNP , given the sentence “Cats have [#NUM] paws.”
a model learns to predict the number 4 .
While appropriate for numerical commonsense, MNP is deficient when it is used to predict measurements.
Measurements , such as 2 meters or 13.2 square miles , are a special class of particularly common numbers in text that have a well-defined and typed system of units .
Given a simple question: “How long did Alex Honnold climb for?” , a single number alone is an insufficient answer since it is meaningless without the unit.
Answers like 1000 meters or 4 hours could both suffice.
Current MNP systems do not jointly reason about numbers with units.
It is reasonable to expect that pretrained models like BERT could lever- age information of units directly as text without any special treatment.
However, in preliminary experiments we find that this yields poor numerical abilities (see Appendix B).
Furthermore, including units as text directly raise more questions: should we evaluate using all units ( meters, feet, inches )?
Should we equally weight across the units?
Current models have no opinion about which unit is appropriate because they are not required to make unit predictions during training.
Together, this indicates that current training objectives do not capture sufficient representations of measurements and that a direct application of MNP to evaluate numeracy of measurements is ill-suited.
To address these shortcomings, we propose the more challenging task of Masked Measurement
Prediction ( MMP ) along with a new model.
In this task, a model must reconstruct both the number together with the correct unit.
In Figure 1 we show how in a MMP setting our model generates a dimension (“Length”), a number in metric log-space (“3.00”), the unit ("feet") and then uses the conversion factor (“3.28”) to deterministically output the full measurement (“3280 feet”).
This example illustrates a key distinction in that our model is flexible and can generate non-metric measurements (feet) but evaluates numerical prediction in canonical units (meters).
MMP is useful for two reasons: 1) as a way to train models to give them better numeracy 2) as a new kind of evaluation that allows for a much more fine-grained analysis of reasoning over numerical quantities.
The task of measurement estimation decouples the different aspects of numeracy allowing for a more interpretable and thorough analysis of numerical reasoning.
We introduce a new evaluation benchmark for MMP based on WikiConvert ( WiCo ) (Thawani et al., 2021a), a large scale dataset of English Wikipedia sentences with ground truth measurement annotations.
We compare the performance of our models on their ability to accurately predict the dimension, unit, and value of a measurement.
We employ a large pretrained transformer model as our textual encoder and examine the performance of different discriminative, generative, and latent variable models along with several ablations.
Our contributions are as follows:
• We introduce a novel challenging task MMP
for pretraining and evaluating numeracy.
• We show that linear probing of existing pretrained models on MMP significantly underperforms fully finetuned models.
• We train a model that reasons jointly about numbers and units which predicts numbers 8.1 times more accurately than the probed pretrained models.
• We find our best performing generative model outperforms human annotators on two evaluations, achieving 7.4-7.8% better dimension accuracy and 33.5-39.9% better unit accuracy.
Furthermore, this model predicts a number closer to ground truth than our annotators 66.2-78.8% of the time.
Furthermore there are numerous applications of better measurement prediction and unit reconstruction such as in table to text generation (Moosavi et al., 2021), answering numerical queries (Sarawagi and Chakrabarti, 2014; Ho et al., 2019) or for improving e-commerce product search(Arici et al., 2021).
We hope that Masked
Measurement Prediction becomes a standard benchmarking tool from which we can gain insight how to best incorporate new numeracy modeling techniques as well as evaluate existing models.
Models 2. Background + Notation
The International System of Units (SI) defines seven fundamental dimensions (Length, Time,
Mass, etc.) and seven corresponding base SI units
(meters, seconds, kilograms, etc.).
The SI system is the most widely adopted measurement standard and is used internationally in domains such as commerce, finance, logistics, and science.
We designate D to be the set of composite dimensions obtained from (and including) the fundamental dimensions .
Values of D include velocity and power.
We let U be the set of all units: the various ways to describe dimensions.
For example, units of Length include meters and miles.
Each training example consists of a real number y , a dimension d ∈ D , a unit u ∈ U , and the remainder of the sentence S .
In MMP , our task is to predict y , d , and u given only S .
In the next sections we describe our generative model designed for MMP followed by the ablations we consider.
2. Model
Measurements have complex semantic meanings, shaped by many standards, particular instruments, and natural world phenomena.
Consider a text concerning rainfall.
From a dimensional analysis perspective, the units inches per year (in/y) and meters per second (m/s) share the same dimension velocity .
However, mentioning in/y usually implies that the text is discussing total rainfall in a region.
Likewise, the use of m/s suggests that the text is examining the speed of falling rain droplets.
To capture this complexity, we consider a generative model that learns the joint distribution of the number, dimension, and unit.
We now describe the generative process of our full model.
To start, conditioned on S , our model samples a discrete dimension variable D .
Then conditioned on the sampled dimension, our model samples a discrete unit variable U compatible with the dimension.
For example, conditioned on the dimension velocity our model will output a distribution over the units of velocity such as [miles per hour; meters per second, inches per year] as opposed to all of U .
We then separately predict a distribution on the canonicalized measurement, ¯ Y , which is the numerical quantity represented in a base canonical (metric) unit like meters.
During inference time, we use the highest scoring dimension and unit and choose the proper conversion factor to deterministically produce the final number y represented in the predicted unit.
We refer to this Ge nerative M asked M easurement model as
GeMM , where the joint p ( D , Y , U | S ) is given by the following equation:
[Equation]
We show the graphical model of GeMM in Figure 2.
We also consider, GeMM U (cid:41) Y , a slight variant where we have a direct dependence between the unit and number prediction with a joint equal to:
[Equation]
2. Discrete Latent Dimension Model
We also consider an unsupervised generative model which treats the dimension as a discrete latent variable.
We use the same number of dimension classes |D| and train to maximize the log-likelihood of the observed Y .
We refer to this model as Lat-Dim and is characterized by:
[Equation]
To evaluate this model we build a contingency matrix of the predicted classes and using a linear solver find the best mapping between our predicted and true dimensions.
We can then apply this mapping to the model predictions and calculate classification metrics for dimension prediction.
2. Model Ablations
We also consider several model ablations of GeMM .
Our first ablation is GeMM -Y -U which models p ( D | S ) .
The second, GeMM -Y , learns the distribution p ( U , D | S )
=
p ( D | S ) × p ( U | D , S ) .
The third, GeMM -U , models p
( Y , D | S )
=
p ( D | S ) × p
( Y | D , S ) .
Our final ablation is GeMM -U -D which learns P ( Y | S ) directly.
2. Model Architectures
For our textual encoder, we use the Huggingface
Transformers (Wolf et al., 2020; Liu et al., 2019) implementation of RoBERTa, a pretrained 12-layer transformer.
We refer to this text encoder as T such that given a sentence S , our model outputs a 768-dimensional vector h T .
We use a single linear layer, W S ∈ R 768 × M , to project h T to h and treat the dimension M as a hyper-parameter.
To form a distribution over the real number line R we use a Log-Laplace model, a competitive model used in the numeracy literature (Spokoyny and BergKirkpatrick, 2020; Thawani et al., 2021a; Zhang et al., 2020). This is equivalent to L 1 regression in log-space and yields the following loss function Split
Examples Max # Min # All 919, 5.5E+ 1ETrain 728, 5.5E+ 1EVal 91, 4.4E+ 1.2ETest 91, 1.6E+ 1.8E- where Y and Y ∗ are predicted and ground truth numbers, respectively:
[Equation]
As shown in Figure 1, we project h with a linear layer W D ∈ R M ×|D| to obtain a distribution over D .
We then use a separate linear layer,
W U ∈ R M ×|U| , to project h and obtain a distribution over U .
To predict ¯ Y , we project h with a linear layer W Y .
In the case of GeMM , we let
W Y ∈ R M ×|D| in order to parameterize a mean of a Log-Laplace distribution for each dimension in D .
For GeMM U (cid:41) Y , we set W Y ∈ R M ×|U| to output the mean of a Log-Laplace distribution for each unit in U and the remaining models, we set W Y ∈ R M × 1 resulting in a single mean of a Log-Laplace distribution.
For training, we use cross-entropy loss for the dimension and unit distributions, and the loss from the equation above for number prediction.
Dataset
We train and evaluate our models on WiCo (Thawani et al., 2021a), a dataset of English
Wikipedia sentences where the number and unit in each sentence are human-annotated.
We canonicalize the units and map each to a single dimension.
For example both feet per second and miles per hour map to velocity .
We show the distribution of all measurements and lengths in Figure 3.
The resulting dataset consists of 919,237 sentences with annotated (number, unit, dimension) triples.
We provide more details on the data in Appendix A.
Experiments
We train all models using a batch size of 200 and use the AdamW (Loshchilov and Hutter, 2019) optimizer with a learning rate of 1 e − 4 and a linear [Table] [Table] warm-up schedule of 500 steps.
We use the “ (cid:94) ” symbol to indicate that we freeze the transformer parameters for training.
For all frozen models we use a log frequency weighted cross-entropy due to the highly imbalanced classes as well as a higher learning rate of 1 e − 3 .
We employ early stopping with a patience of five epochs on validation score.
To evaluate the performance of our models, we report the macro averaged F1 score for dimension and unit prediction and log-mae to evaluate number prediction.
We define log-mae in Equation 2 where
Y is the predicted number and Y ∗ is the ground truth number.
As a simple baseline for dimension and unit prediction, we employ majority class voting.
For number prediction we use the median of all the numbers in the training set.
[Equation]
4. Few-Shot
To study the degree to which current pretrained models capture different aspects of numeracy, we consider the following few-shot experiment.
We sample a balanced dataset of dimensions where each class gets 10, 40, 70, or 100 labeled examples.
We train GeMM -Y -U and GeMM -U -D on the few-shot task where the pretrained text encoder
T parameters are frozen and compare their performance against full fine-tuning.
Due to the high variance of GeMM -Y -U , we report the average
Probing Type
Val
Test
Majority33.
GeMM (cid:94) p ( D | S ) 69.
GeMM -Y -U p ( D | S ) 88.
GeMM -Y p ( D | S ) 87.
GeMM -U p
( D | S ) 87.
[Table]
Lat-Dim p ( D | S ) 9.
GeMM p ( D | S ) 87.
GeMM U (cid:41) Y p ( D | S )
[Table]
[Table] [Table] of three random seeds.
In Table 2 and Table 3 we show results of GeMM -Y -U and GeMM -U -D respectively.
Although performance improves with more data, the frozen models significantly underperform their unfrozen counterparts across all dataset sizes. For example, in the 100-shot dataset, the frozen model shows 7.1 lower F1 and 0.34 higher log-mae .
These results suggest that current pretrained transformers do not capture numeracy to a large extent.
Model
Probing Type
Val
Test
Majority8.
GeMM (cid:94) p ( U | D , S ) 29.
GeMM -Y p ( U | D , S ) 52.
GeMM p ( U [Equation] D , S ) 51.
GeMM U (cid:41) Y p ( U | D , S ) 49.
4. Dimension Prediction
We train our models and their ablations on the full dataset and measure their performance on dimension prediction.
In Table 4, we show the results of dimension prediction conditioned on S .
We observe that the performance gap between the frozen and unfrozen GeMM grows to 19 . 5 F1 on the test split despite training on 3 orders of magnitude more training data than the few-shot setting. By using Bayes’ rule, we perform dimension prediction conditioned on both S and ¯ Y and show our results in Table 5. We observe that both models show improved dimension prediction ability when supplied with the number with GeMM U (cid:41) Y reaching 96.6 F1 score, an effective error rate reduction of 75%.
4. Unit Prediction
We show the unit prediction performance of our models in Table 6.
The strongest performing model for unit prediction was GeMM with a F1 score of 54 .
Again, the frozen GeMM (cid:94) produced a 25 .lower F1 score than its unfrozen counterpart.
We note that even though the F1 scores on unit [Table] prediction are much lower than dimension prediction, they are still significantly better than the majority baseline.
Although one can freely substitute a unit with one in the same dimensional class, we tend to be more systematic and choose units that allow for more straightforward human readability or reflect the actual instruments used for measurement.
As a result, we gravitate towards regularities that models can learn to recognize.
The converse of this is also interesting as it suggests that the expressed units imply more semantic meaning than what is captured in the standardized measurement
[Table]
4. Number Prediction
We show the number prediction performance of our models in Table 7.
Consistent with our previous experiments, all models outperform GeMM (cid:94) .
Furthermore, we observe that not modeling U and D (as is the case in GeMM -U -D ) increases log-mae , i.e., results in worse numerical prediction.
While competitive with GeMM and its variants on number prediction, Lat-Dim cannot predict dimensions with the same efficacy (Table 4).
We also experiment with the setting where
GeMM -U conditionally generates the number for a particular dimension.
In this setting, GeMM -U improves log-mae to 0 . 469 .
Extending this setting further, we condition GeMM
U (cid:41) Y on both a unit and a dimension to produce the best log-mae among our models: 0 . 401 .
We now revisit our original motivating example: “Alex Honnold climbed for [NUM] [UNIT]” .
Assume we want to know the distance of a climb.
To do this, we condition GeMM U (cid:41) Y on D = length and U = feet .
If, on the other hand, we want to know the duration of a climb, we change the conditioning to D = time and U
= hours .
Now, if we want to know the length of Alex Honnold’s climbing career, we condition GeMM
U (cid:41) Y on D = time and U = years .
These examples illustrate the flexibility of GeMM U (cid:41) Y and the importance of jointly modeling numbers, units, and dimensions.
4. Quantitative Analysis 4.5. Dimensions and Unit
In Figure 4a we visualize a confusion matrix of dimension predictions by GeMM U (cid:41) Y .
The low accuracy for electric charge and temperature is attributed to a mislabeling in the dataset.
For mass, we find many ambiguous situations where either mass or length are appropriate.
See the first row of Table 10 for such an example.
Thus far, we have treated dimensions as distinct classes with no relationships.
However, dimensions are compositions of the seven fundamental dimensions.
Therefore, dimensions that share fundamental dimensions are more similar than those that do not.
To quantify this similarity, we can treat dimensions as a vector where each element represents the exponent of a fundamental dimension.
Then to measure the similarity of two dimensions, we take their Manhattan distance.
To illustrate, assume there exist only two fundamental dimensions: Length and Time.
Let speed =
(1 , − 1) and length =
(1 , 0) where the first element represents
Length and the second represents Time.
The Manhattan distance between speed and length is equal to one.
In Figure 5, we visualize the Manhattan distance between the predictions of GeMM U (cid:41) Y and ground truth.
We observe that there is generally an inverse relationship between error count and the distance of the errors.
This observation suggests that our model has learned that some dimensions are more similar than others.
This suggestion is reinforced by Figure 4a where misclassifications tend to have small distances from the true dimension.
For example, velocity is most often misclassified as length.
For unit prediction, we find that most mistakes occur substituting units with ones that have [Figure] similar magnitudes like feet for meters or kilometers for miles.
4.5. Numeracy
In Table we show log-mae by dimension as predicted by GeMM U (cid:41) Y .
We note that errors are not uniform across dimensions, predicting areas is 2. times harder velocities .
We also observe that the magnitudes of errors seem to be positively correlated with the variances observed in Figure 3. [Table] [Table]
Human Evaluation
We perform two evaluations of GeMM U (cid:41) Y against human annotators.
In the first evaluation, we compare against the combined effort of three Technical
Annotators on a balanced set of 90 sentences randomly sampled from the test set.
The annotators worked together to predict the missing dimensions, units, and accurate measurement estimates.
Examples of sentences and annotations shown in Table 10.
[Table]
We show the results for both evaluations in Table 9.
In both evaluations, the model outperforms the human annotators on every task.
For dimension prediction, the model led by 7.4-7.8 percentage points.
Of the sentences where the dimension was correctly annotated, the model led by 33.5-39.9 percentage points on unit prediction.
For sentences where both the model and human correctly predicted the dimension, the model predicted a number closer to ground truth 66.2-78.8% of the time.
4. Qualitative Analysis 4.6. Semantic Head Embeddings
In Figure 6 we plot the t-SNE embeddings of the sentences’ h , the output of our text encoder.
We label each h with the masked measurement’s true dimension, unit and exponent of the number.
In 6a we observe that most embeddings labeled by their true dimension tend to form tight clusters.
In 6b we filter to only show embeddings that share the
Length dimension and label them by their units.
We find that clusters are organized by the relative magnitudes of their units: large ( Kilometers, miles ), medium ( feet, meters ), and small ( millimeters, inches, centimeters ).
Further we see that yards appear close to other imperial units of feet and miles .
Finally, in 6c when embeddings are binned by the exponent of their values we observe that the left to right direction appears to capture the increasing magnitude of a number.
Related Work 5. Numeracy
Multiple works have probed word embeddings like word2vec, GloVe, FastText (Naik et al., 2019) and contextual embeddings from models like BERT (Wallace et al., 2019; Zhang et al., 2020) or T (Pal and Baral, 2021) on a variety of numerical tasks like sorting, numeration, magnitude prediction, and common sense (Lin et al., 2020).
Several works have targeted numeracy pretraining using left to right language models (Spithourakis and
Riedel, 2018), CNN and RNN based models (Chen et al., 2019), pretrained transformers (Spokoyny and Berg-Kirkpatrick, 2020; Jin et al., 2021), for an overview (Thawani et al., 2021b).
Incorporating synthetic mathematical data augmentations (Geva et al., 2020) has improved question answering while numerical pretraining has been shown to lower masked language modelling perplexity (Thawani et al., 2021a).
Either directly or indirectly units have been involved in providing more interpretable explanation of quantities (Chaganty and Liang, 2016), solving Fermi problems (Kalyan et al., 2021) and resolving numeric FusedHeads (Elazar and Goldberg, 2019).
5.1. Numeracy Benchmarks
Several numeracy benchmarks have been proposed like quantitative reasoning in natural language entailment (Ravichander et al., 2019) and synthetic measurement estimation (Jin et al., 2021). The
[Figure] closest benchmark to our work is the Distribution over Quantities dataset (DoQ) introduced by Elazar et al.
A rule-based method was combined with simple heuristics to build DoQ resulting in its high-coverage albeit also higher noise.
Although,
WiCo is smaller, it has much higher fidelity since it utilizes a feature used by editors of Wikipedia to automatically convert quantities into different units.
Further, WiCo provides the whole sentence as context as opposed to triplets of words.
Zhang et al. (2020) use artificial templates to probe models on
DoQ and find little difference between numerically pretrained and frozen embeddings such as ELMo.
In contrast, our findings show there is a significant gap on WiCo between fully finetuned models and their frozen counterparts.
Limitations
The pretrained RoBERTa model we use in experiments was likely pretrained on data that included WiCo .
Thus, it is reasonable to be concerned about inflated test performance.
That said, the task we consider is distinct from the selfsupervised task used to pretrain RoBERTa (i.e. masked word classification vs. masked number regression). Further, our experiments on directly probing RoBERTa to predict masked numbers and units showed poor performance – indicating, perhaps, that even if RoBERTa’s pre-training set did include WiCo , RoBERTa did not memorize aspects of our test set relevant to masked number prediction, partially mitigating these concerns. The human evaluation studies we conducted are a quite limited ‘guesstimating’ task. The human annotators were not allowed to use any external information from searching the internet or looking up answers in knowledge-bases.
Their total average completion time per question was 33 seconds.
Furthermore, many annotators may not have strong intuition about measurements with unfamiliar and uncommon unit types.
For these reasons it is not surprising that our models outperform the human annotators in this limited experiment.
However, these human evaluation studies do help calibrate the difficulty of the MMP task on WiCo .  Conclusion In this work we propose Masked Measurement Prediction, a new task that requires models to jointly predict masked numbers and units in running text. We motivate this task as an important extension of existing masked number-only prediction tasks that addresses their limitations and allows for better evaluation of numeracy in NLP models.
In our study, we show that probing of traditional pretrained transformers exposes a gap in their understanding of contextualized quantities.
Through careful quantitative and qualitative analysis of our new model, which directly reasons about underlying units and dimensions, we find that it is possible to learn good representations of measurements.
For future work we aim to extend this dataset to cover more existing standardized units from organizations such as UNECE.
4 We hope our MMP task encourages research into further development of better numeracy methodologies.
A Dataset
We train and evaluate our models on Wiki-Convert ( WiCo ) (Thawani et al., 2021a), a dataset of
English Wikipedia sentences where the number and unit in each sentence are human-annotated.
The built-in template in Wikipedia can ensure the text contains numbers and units.
For example, {{convert|2|km|mi}} displays askilometres (1.2 mi) .
By searching within
Wikipedia articles for the use of this template, the authors of WiCo automatically extract humanannotated numbers.
To perform unit canonicalization, we use Pint 5 whenever the mapping is unambiguous.
In the ambiguous case, we manually inspect the sentence and perform the mapping.
For example, we map the unit sqmi in WiCo to square miles to let pint perform unit canonicalization.
Table 10 shows examples of the extended dataset.
The original dataset contains 924,473 sentence.
The median sentence length is 106 characters, with 29,597 sentences has a length shorter than 20 characters.
We provide statistics of the data in Table 1.
For preprocessing we exclude sentences which have more than 64 tokens to have efficient computing memory or where the number is negative for simplicity.
According to Thawani et al.
(2021a)
WiCo , “... has been extracted from
Wikipedia dumps, which are licensed under the
GNU Free Documentation License (GFDL) and the Creative Commons Attribution-Share-Alike 3.
License.”
Thawani et al.
(2021a) constructed WiCo with the intent that it be used to further numeracy
NLP research.
Our use of WiCo is aligned with its authors’ goals.
B MLM Preliminary Unit Probe
We perform a preliminary unit probe shown in Table 11.
The model predicts vastly different numbers when conditioned on different units.
We observe a mean of 3086.8 and a standard deviation offor all the converted metric output.
C Experiments
We train our model GeMM
U (cid:41) Y on a single Nvidia
GeForce RTX 2080 Ti for 4 hours and 14 minutes with a total parameter of 124,696,538.
C. Quantitative Analysis
In Figure 7, we show log-mae is relatively small for small magnitude units, which means predicting numbers for small magnitude units is easier than predicting numbers for their larger counterparts.
In Figure 4, we show confusion matrices of dimension and unit predictions by GeMM U (cid:41) Y .
[Table]
D Human Annotators D. Evaluation
The Technical Annotators have diverse scientific backgrounds ranging from chemistry, earth sciences, and computer science.
One annotator is a native Chinese speaker, and two are native English speakers.
D. Evaluation
In Figure 8 we show the instructions provided along with the interface we designed for our MMP task.
While the workers’ geographic location were not provided to us by Mechanical Turk, we aimed to compensate the workers above the US federal minimum wage of $7.25.
We paid workers $0.15 per annotation with an average completion time ofseconds.
This equates to an hourly rate of $12. after Mechanical Turk fees.
Other demographic information is only provided by Mechanical Turk for an extra fee.
E Ethical Considerations
Like any system that makes predictions, those made by GeMM are not necessarily accurate and may be used by malicious actors to generate fake information to mislead their audience.
Additionally,
GeMM is an extension of RoBERTa and therefore inherits the biases learned during the training of
RoBERTa.
Our work focuses exclusively on English and Arabic numerals.
As noted by Thawani et al.
(2021a)
, the units in WiCo are heavily biased towards European and American units as they are over