Methods for Numeracy-Preserving Word Embeddings (2020)

Abstract: Word embedding models are typically able to capture the semantics of words via the distributional hypothesis, but fail to capture the numerical properties of numbers that appear in the text. This leads to problems with numerical reasoning involving tasks such as question answering. We propose a new methodology to assign and learn embeddings for numbers. Our approach creates Deterministic, Independent-of-Corpus Embeddings (the model is referred to as DICE) for numbers, such that their cosine similarity reflects the actual distance on the number line. DICE outperforms a wide range of pre-trained word embedding models across multiple examples of two tasks: (i) evaluating the ability to capture numeration and magnitude; and (ii) to perform list maximum, decoding, and addition. We further explore the utility of these embeddings in downstream tasks, by initializing numbers with our approach for the task of magnitude prediction. We also introduce a regularization approach to learn model-based embeddings of numbers in a contextual setting.

## introduction word embeddings capture semantic relationships between words by operationalizing the distributional hypothesis @xcite . they can be learned either non-contextually @xcite @xcite or contextually @xcite . non-contextual embeddings have worked well on various language understanding and semantic tasks @xcite . more recently, they have also been used as pre-trained word embeddings to aid more sophisticated contextual models for solving rigorous natural language processing (nlp) problems, including translation, paraphrasing, and sentence-similarity tasks @xcite . while word embeddings effectively capture semantic relationships between words, they are less effective at capturing numeric properties associated with numbers. though numbers represent a significant percentage of tokens in a corpus, they are often overlooked. in non-contextual word embedding models, they are treated like any other word, which leads to misinterpretation. for instance, they exhibit unintuitive similarities with other words and do not contain strong prior information about the magnitude of the number they encode. in sentence similarity and reasoning tasks, failure to handle numbers causes as much as 29% of contradictions @xcite . in other data-intensive tasks where numbers are abundant, like neural machine translation, they are masked to hide the translation models inefficiency in dealing with them @xcite . there are a variety of tests proposed to measure the efficiency of number embeddings. for instance, @xcite shows that glove @xcite , word2vec @xcite , and fasttext @xcite fail to capture numeration and magnitude properties of a number. numeration is the property of associating numbers with their corresponding word representations ("3" and "three") while magnitude represents a number's actual value (3 < 4). further, @xcite proposes several tests for analyzing numerical reasoning of number embeddings that include list maximum, decoding, and addition. in this paper, we experimentally demonstrate that if the cosine similarity between word embeddings of two numbers reflects their actual distance on the number line, the resultant word embeddings are useful in downstream tasks. we first demonstrate how deterministic, independent-of-corpus embeddings (dice) can be constructed such that they almost perfectly capture properties of numera-tion and magnitude. these non-contextual embeddings also perform well on related tests for numeracy @xcite . to demonstrate the efficacy of dice for downstream tasks, we explore its utility in two experiments. first, we design a dice embedding initialized bi-lstm network to classify the magnitude of masked numbers in the 600k dataset @xcite . second, given the popularity of modern contextual model-based embeddings, we devise a regularization procedure that emulates the hypothesis proposed by dice and can be employed in any task-based fine-tuning process. we demonstrate that adding such regularization helps the model internalize notions of numeracy while learning task-based contextual embeddings for the numbers present in the text. we find promising results in a numerical reasoning task that involves numerical question answering based on a sub-split of the popular squad dataset @xcite . our contribution can be summarized as follows: ## related work the major research lines in this area have been dedicated to (i) devising probing tests and curating resources to evaluate the numerical reasoning abilities of pre-trained embeddings, and (ii) proposing new models that learn these properties. @xcite surveyed a number of noncontextual word embedding models and highlighted the failure of those models in capturing two essential properties of numbers -numeration and magnitude. @xcite created a novel dataset named numeracy-600k, a collection of approximately 600,000 sentences from market comments with a diverse set of numbers representing age, height, weight, year, etc. the authors use neural network models, including a gru, bigru, crnn, cnn-capsule, gru-capsule, and bigrucapsule, to classify the magnitude of each number. @xcite compares and contrasts the numerical reasoning ability of a variety of noncontextual as well as contextual embedding models. the authors also proposed three tests -list maximum, decoding, and addition -to judge the numerical reasoning ability of embeddings of numerals. they infer that word embedding models that perform the best on these three tests have captured the numerical properties of numbers well. therefore, we consider these proposed tests in our evaluation. (spithourakis and riedel, 2018) used a variety of models to distinguish numbers from words, and demonstrated that this ability reduces model perplexity with neural machine translation. @xcite found that neural networks are capable of reasoning numbers with explicit supervision. numerically augmented qanet (naqanet) @xcite was built by adding an output layer on top of qanet @xcite to predict answers based on addition and subtraction over numbers in the drop dataset. our work, in contrast, offers a simple methodology that can be added to any model as a regularization technique. our work is more similar to @xcite , where embedding of a number is learned as a simple weighted average of its prototype embeddings. such embeddings are used in tasks like word similarity, sequence labeling and have been proven to be effective. ## methods to overcome nlp models inefficiency in dealing with numbers, we consider our method dice to form embeddings. to begin, we embed numerals and word forms of numbers as vectors e @xmath0, where i indexes numerals identified within a corpus. we first preprocess by parsing the corpora associated with each of our tasks (described below) for numbers in numeral and word forms to populate a number vocabulary. then, the dimensionality of the embeddings required for that task is fixed. we explicitly associate the embeddings of a numeral and word forms of numbers to have the same embedding. ## experiments to observe the numerical properties of dice, we consider two tasks: task 1 deals with the numeration (num) and magnitude (mag) properties as proposed by @xcite ; task 2 performs list maximum, decoding, and addition as proposed by @xcite . we then experiment on two additional tasks to demonstrate the applications of dice. ## conclusion in this work, we methodologically assign and learn embeddings for numbers to reflect their numerical properties. we validate our proposed approach with several experiments that test number embeddings. the tests that evaluate the numeral embeddings are fundamentally applicable to all real numbers. finally, we introduced an approach to jointly learn embeddings of numbers and words that preserve numerical properties and evaluated them on a contextual word embedding based model. in our future