Learning Numeral Embedding (2020)

Abstract: Word embedding is an essential building block for deep learning methods for natural language processing. Although word embedding has been extensively studied over the years, the problem of how to effectively embed numerals, a special subset of words, is still underexplored. Existing word embedding methods do not learn numeral embeddings well because there are an infinite number of numerals and their individual appearances in training corpora are highly scarce. In this paper, we propose two novel numeral embedding methods that can handle the out-of-vocabulary (OOV) problem for numerals. We first induce a finite set of prototype numerals using either a self-organizing map or a Gaussian mixture model. We then represent the embedding of a numeral as a weighted average of the prototype number embeddings. Numeral embeddings represented in this manner can be plugged into existing word embedding learning approaches such as skip-gram for training. We evaluated our methods and showed its effectiveness on four intrinsic and extrinsic tasks: word similarity, embedding numeracy, numeral prediction, and sequence labeling.

## introduction word embeddings have become an essential building block for deep learning approaches to natural language processing (nlp). the quality of pretrained word embeddings has been shown to significantly impact the performance of neural approaches to a variety of nlp tasks. over the past two decades, significant progress has been made in the development of word embedding techniques @xcite @xcite @xcite . however, existing word embedding methods do not handle numerals * corresponding author. previous work @xcite also shows that the numeral oov problem is more severe when learning word embeddings from corpora with abundant numerals such as clinical reports. even if a numeral is included in the vocabulary, its scarcity in the training corpus would negatively impact the learning accuracy of its embedding. the inadequate handling of numerals in existing word embedding methods can be problematic in scenarios where numerals convey critical information. for instance, if the numerals in the example are oov or their embeddings are not accurately learned, then it becomes impossible to judge the categories of the numerals or the reasonableness of the sentences. in this paper, we propose two novel methods that can produce reasonable embeddings for any numerals. the key idea is to represent the embedding of a numeral as a weighted average of a small set of prototype number embeddings induced from the training corpus using either a self-organizing map @xcite or a gaussian mixture model. the weights are computed based on the differences between the target numeral and the prototype numerals, reflecting the inductive bias that numerals with similar quantities are likely to convey similar semantic information and thus should have similar embeddings. numeral embeddings represented in this manner can then be plugged into a traditional word embedding method for training. we empirically evaluate our methods on four tasks: word similarity, embedding numeracy, numeral prediction, and sequence labeling. the results show that our methods can produce high-quality embeddings for both numerals and non-numerical words and improve the performance of downstream tasks. ## methods given a training corpus c, we first extract all the numerals using regular expressions and form a dataset x containing all the numbers represented by these numerals. a number (e.g., 2000) may appear for multiple times in x if its corresponding numerals (e.g., '2000', '2,000', etc.) appear for multiple times in c. we then induce a finite set p of typical numerals (i.e., prototypes) from x using a self-organizing map @xcite or a gaussian mixture model. we also define a function sim(n 1 , n 2 ) outputting the similarity between two arbitrary numbers n 1 and n 2 . now we represent the embedding of any target numeral n as a weighted average of the prototype number embeddings with the weights computed by the similarity function: we use e(•) to denote the embedding of a number, α is the normalization factor where @xmath0, p) = 1. this formulation satisfies the intuition that numerals with similar quantities are likely to convey similar semantic information and should have similar embeddings. our numeral embeddings can be integrated into traditional word embedding methods such as skip-gram for training. during training, we backpropagate the error gradient to update the prototype embeddings. in this way, the prototype embeddings (and hence all the numeral embeddings) are learned jointly with non-numerical word embeddings. ## experiments and results we evaluate our methods on four intrinsic and extrinsic tasks: word similarity, embedding numeracy, numeral prediction, and sequence labeling. we report the results of our methods based on som and gmm separately. we choose the hyper-parameters (e.g., the number of prototypes, gmm initialization and training methods) using validation sets and report the best hyper-parameters for each experiment in appendix.b. ## conclusion in this paper, we propose two novel numeral embedding methods that represent the embedding of a numeral as a weighted average of a set of prototype numeral embeddings. the methods can be integrated into traditional word embedding approaches such as skip-gram for training. we evaluate our methods on four intrinsic and extrinsic tasks, including word similarity, embedding numeracy, numeral prediction, and sequence labeling, and show that our methods can improve the performance of numeral-related tasks and has better generalizability. our code and sample data can be found at https://github.com/jeffchy/ learning-numeral-embeddings . an important future direction is to handle numeral polysemy. for example, the numeral "2019" may denote either a year or an ordinary number. one potential method is to assign a different embedding to each sense of a numeral. in this way, "2019" would have one embedding for representing a year and another for representing an ordinary quantity. the similarity function would treat different senses of a numeral differently. for example, the year sense of "2019" would be similar to the year sense of "19" but dissimilar to the sole sense of "2019.5", while the quantity sense of "2019" would be similar to that of "2019.5". our methods also have the potential to apply to contextual word embedding methods, so this would be another future direction.