An Empirical Investigation of Contextualized Number Prediction (2020)

Abstract: We conduct a large scale empirical investigation of contextualized number prediction in running text. Specifically, we consider two tasks: (1)masked number prediction– predict-ing a missing numerical value within a sentence, and (2)numerical anomaly detection–detecting an errorful numeric value within a sentence. We experiment with novel combinations of contextual encoders and output distributions over the real number line. Specifically, we introduce a suite of output distribution parameterizations that incorporate latent variables to add expressivity and better fit the natural distribution of numeric values in running text, and combine them with both recur-rent and transformer-based encoder architectures. We evaluate these models on two numeric datasets in the financial and scientific domain. Our findings show that output distributions that incorporate discrete latent variables and allow for multiple modes outperform simple flow-based counterparts on all datasets, yielding more accurate numerical pre-diction and anomaly detection. We also show that our models effectively utilize textual con-text and benefit from general-purpose unsupervised pretraining.

## introduction pretraining large neural architectures (e.g. transformers @xcite ) on vast amounts of unlabeled data has lead to great improvements on a variety of nlp tasks. typically, such models are trained using a masked language modeling (mlm) objective and the resulting contextualized representations are finetuned for a particular downstream task like question answering or sentence classification @xcite . in this paper, we focus on a related modeling paradigm, but a different task. specifically, we investigate contextualized number prediction: predicting a real numeric value from its textual context using an mlm-style modeling objective. we conduct experiments on two specific variants: (1) masked number prediction (mnm), in which the goal is to predict the value of a masked number token in a sentence, and (2) numerical anomaly detection (nad), with the goal of deciding whether a specific numeric value in a sentence is errorful or anomalous. in contrast with more standard mlm training setups, here we specifically care about the accuracy of the trained masked conditional distributions rather than the contextualized representations they induce. while successful models for these tasks are themselves useful in applications like typo correction and forgery detection @xcite , better models of numeracy are essential for further improving downstream tasks like question answering, numerical information extraction @xcite or numerical fact checking @xcite , as well as for processing number-heavy domains like financial news, technical specifications, and scientific articles. further, systems that detect anomalous numbers in text have applications in practical domains -for example, medicine (thimbleby and cairns, 2010) -where identification of numerical entry errors is critical. our modeling approach to contextualized number prediction combines two lines of past work. first, following @xcite , we treat number prediction as a sentence-level mlm problem where only numerical quantities are masked. however, @xcite focused on predicting the discrete exponent of masked numbers as a classification problem. in contrast, @xcite demonstrate the utility of predicting full numerical quantities in text, represented as real numbers, but do so in a language modeling frame-work, conditioned only on left context. here, we propose a novel setup that combines full-context encoding (i.e. both left and right contexts) with real-valued output distributions for modeling numerical quantities in text. in figure 1 , we illustrate an example where we aim to predict "2 trillion" as a quantity on the real number line. we expand upon past work by conducting a large scale empirical investigation that seeks to answer three questions: (1) which encoding strategies yield more effective representations for numbers in surrounding context? (2) which encoding architectures provide the best representations of surrounding context? (3) what are the most effective real-valued output distributions to model masked number quantities in text? to answer these questions, we propose a suite of novel real-valued output distributions that add flexibility through the use of learned transformation functions and discrete latent variables. we conduct experiments for both mnm and nad tasks on two large datasets in different domains, combining output distributions with both recurrent and transformer-based encoder architectures, as well as different numeric token encoding schemes. further, while @xcite studied a specific type of nad (detecting exaggerated numbers in financial comments), we examine several nad variants with different types of synthetic anomalies that are found to arise in practice across different domains of data. finally, we further compare results with a strong discriminative baseline. ## models our goal is to predict numbers in their textual contexts. the way we approach this is similar to masked language modeling (mlm), but instead of masking and predicting all token types, we only mask and predict tokens that represent numeric values. for example in figure 1 we wish to predict that the value of the masked number [#mask] should be 2 × 10 12 ∈ r given the surrounding context. for notational simplicity, we describe our model as predicting a single missing numeric value in a single sentence. however, like other mlms (see section 4.3), during training we will mask and predict multiple numeric values simultaneously. let x be a sentence consisting of n tokens where the kth token is a missing numerical value, y. the goal of our model is to predict the value of y con-ditioned on x. we will use common notation for from similar setups and simply treat the kth token in x as a masked numeric value, [#mask]. our models p θ,γ (y|x) consist of three main components: an input representation of the sentence, a contextual encoder with parameters γ which summarizes the sentence, and an output distribution with parameters θ over the real number line. in this section we will describe our strategies for numerical input representation, the two types of contextual encoders we use, along with different formulations of numerical output distributions. ## experiments in this section we explain our experimental setup, starting with our evaluation metrics, implementation details, results, and ablation analyses. we use the following naming convention for models: we specify the encoder (bigru, bert) first, followed by one of our four output distributions (loglp, flowlp, dexp, gmm). ## conclusion in this work we carried out a large scale empirical investigation of masked number prediction and numerical anomaly detection in text. we showed that using the base-10 exponent as a discrete latent variable outperformed all other competitive models. specifically, we found that learning the exponent representation using pretrained transformers that can incorporate left and right contexts, combined with discrete latent variable output distributions, results is the most effective way to model masked number quantities in text. future work might explore combining more expressive flows with discrete latent variables.