L
EARNING N UMERAL E MBEDDING
Anonymous authors
Paper under double-blind review
A BSTRACT
Word embedding is an essential building block for deep learning methods for natural language processing.
Although word embedding has been extensively studied over the years, the problem of how to effectively embed numerals, a special subset of words, is still underexplored.
Existing word embedding methods do not learn numeral embeddings well because there are an inﬁnite number of numerals and their individual appearances in training corpora are highly scarce.
In this paper, we propose two novel numeral embedding methods that can handle the out-ofvocabulary (OOV) problem for numerals.
We ﬁrst induce a ﬁnite set of prototype numerals using either a self-organizing map or a Gaussian mixture model.
We then represent the embedding of a numeral as a weighted average of the prototype number embeddings.
Numeral embeddings represented in this manner can be plugged into existing word embedding learning approaches such as skip-gram for training.
We evaluated our methods and showed its effectiveness on four intrinsic and extrinsic tasks: word similarity, embedding numeracy, numeral prediction, and sequence labeling.
I NTRODUCTION
Word embeddings, the distributed vector representations of words, have become the essential building block for deep learning approaches to natural language processing (NLP).
The quality of pretrained word embeddings has been shown to signiﬁcantly impact the performance of neural approaches to a variety of NLP tasks.
Over the past two decades, signiﬁcant progress has been made in the development of word embedding techniques (Lund & Burgess, 1996; Bengio et al., 2003;
Bullinaria & Levy, 2007; Mikolov et al., 2013b; Pennington et al., 2014).
However, existing word embedding methods do not handle numerals adequately and cannot directly encode the numeracy and magnitude of a numeral (Naik et al., 2019).
Most methods have a limited vocabulary size and therefore can only represent a small subset of the inﬁnite number of numerals.
Furthermore, most numerals have very scarce appearances in training corpora and therefore are more likely to be outof-vocabulary (OOV) compared to non-numerical words.
For example, numerals account for 6 .
15% of all unique tokens in English Wikipedia, but in GloVe Pennington et al.
(2014) which is partially trained on Wikipedia, only 3 .
79% of its vocabulary is numerals.
Previous work (Spithourakis et al., 2016) also shows that the numeral OOV problem is even more severe when learning word embeddings from corpora with abundant numerals such as clinical reports.
Even if a numeral is included in the vocabulary, its scarcity in the training corpus would negatively impact the learning accuracy of its embedding.
The inadequate handling of numerals in existing word embedding methods can be problematic in scenarios where numerals convey critical information.
Take the following sentences for example,
“Jeff is 190 , so he should wear size XXL” (190 is a reasonable height for size XXL)
“Jeff is 160 , so he should wear size XXL” (160 is an unreasonable height for size XXL)
“Jeff is 10 , so he should wear size XS” (10 is an age instead of a height)
If the numerals in the example are OOV or their embeddings are not accurately learned, then it becomes impossible to judge the categories of the numerals or the reasonableness of the sentences.
In this paper, we propose two novel methods that can produce reasonable embeddings for any numerals.
The key idea is to represent the embedding of a numeral as a weighted average of a small set of prototype number embeddings.
The prototype numerals are induced from the training corpus using either a self-organizing map (Kohonen, 1990) or a Gaussian mixture model.
The weights are computed based on the differences between the target numeral and the prototype numerals, reﬂecting the inductive bias that numerals with similar quantities are likely to convey similar semantic information and thus should have similar embeddings.
Numeral embeddings represented in this manner can then be plugged into a traditional word embedding method for training.
We empirically evaluate our methods on four tasks: word similarity, embedding numeracy, numeral prediction, and sequence labeling.
The results show that our methods can produce high-quality embeddings for both numerals and non-numerical words and improve the performance of downstream tasks.
R ELATED W ORK
Word Embedding Word embeddings are vector representations of words which carry semantic meanings implicitly and are trained without supervision.
Most existing word embedding training methods can be divided into two classes.
The ﬁrst class of methods (Lund & Burgess, 1996;
Rohde et al., 2006; Bullinaria & Levy, 2007; Lebret & Lebret, 2013) extract word co-occurrence statistics from the training corpus, compute a word-word matrix based on measures such as PPMI, and then apply dimension reduction techniques such as principle component analysis to produce a low-dimensional vector representation for each word.
The second class of methods (Bengio et al., 2003; Collobert & Weston, 2008; Mikolov et al., 2013a;b) use a simple neural network to model the relation between a word and its context within a sliding window in the training corpus.
GloVe (Pennington et al., 2014) has been proposed as a method that combines the advantages of both classes.
All the above methods have a ﬁnite vocabulary size and use a ‘UNK’ symbol to represent OOV words.
Recent work (Naik et al., 2019) shows that these popular methods do not handle numerals adequately.
Wallace et al.
(2019) shows that existing word embedding methods can encode numeracy implicitly for high-frequency numerals, but the embedding’s numeracy for OOV numerals is not investigated.
Our goal is to design better numeral embedding methods that can be integrated into traditional word embedding methods and handle the OOV problem for numerals.
Numeracy in natural language Numeral understanding has been found important in textual entailment (Lev et al., 2004; De Marneffe et al., 2008; Roy et al., 2015) and information extraction (Intxaurrondo et al., 2015; Madaan et al., 2016), but existing systems often use manually deﬁned task-speciﬁc features and logic rules to identify numerals, which is hard to generalize to other tasks.
A lot of research has been done trying to solve math problems, using either manually designed features and rules (Roy et al., 2015; Mitra & Baral, 2016; Roy & Roth, 2016; Upadhyay et al., 2016) or sequence-to-sequence neural networks Wang et al.
(2017), but the quantity of numerals is not important in this task and hence existing methods often replace numerals by dummy symbols such as n 1 and n 2 .
Spithourakis & Riedel (2018) studied different strategies to better model numerals in language models.
Chen et al.
(2019) created Numeracy-600K dataset and studied the ability of neural network models to learn numeracy.
Our work differs from previous work in that we aim to produce general-purpose numeral embeddings that can be employed in any neural NLP approach.
M ETHODS
Given a training corpus C , we ﬁrst extract all the numerals using regular expressions and form a dataset X containing all the numbers represented by these numerals.
A number (e.g., 2000) may appear for multiple times in X if its corresponding numerals (e.g., ‘2000’, ‘2,000’, etc.) appear for multiple times in C .
We then induce a ﬁnite set P of typical numerals (i.e., prototypes) from
X using a self-organizing map (Kohonen, 1990) or a Gaussian mixture model.
We also deﬁne a function sim ( n 1 , n 2 ) outputting the similarity between two arbitrary numbers n 1 and n 2 .
Now we represent the embedding of any target numeral n as a weighted average of the prototype number embeddings with the weights computed by the similarity function:
[Equation]
We use e ( · ) to denote the embedding of a number α is the normalization factor.
This formulation satisﬁes the intuition that numerals with similar quantities are likely to convey similar semantic information and thus should have similar embeddings.
Our numeral embeddings can be integrated into traditional word embedding methods such as skipgram for training.
During training, we back-propagate the error gradient to update the prototype number embeddings.
In this way, the prototype number embeddings (and hence all the numeral embeddings) are learned jointly with non-numerical word embeddings.
3. S QUASHING NUMBERS TO LOG - SPACE
Inspired by psychological evidence that our brain compresses large quantities nonlinearly using a logarithmic scale on the mental number line (Nieder & Miller, 2003; Dehaene, 2011), we design the following squashing function to transform all the numbers in X into the log-space before prototype induction.
Alternatively, we can apply the function only in the similarity function.
Besides the psychological motivation, squashing is also necessary for our methods to avoid overﬂow during training when there are very large numbers such as 10 15 in the training corpus.
[Equation]
3. P ROTOTYPE I NDUCTION
We develop two methods for inducing a small prototype set P from the number dataset X .
Denote the number of prototypes by m .
Self-Organizing Map
A self-organizing map (SOM) (Kohonen, 1990) is an artiﬁcial neural network that can be viewed as a clustering method.
After training a SOM on the dataset X , we regard each cluster centroid as a prototype.
One advantage of using a SOM in comparison with traditional clustering methods is that it distributes prototypes more evenly on the number line and may assign prototypes to number ranges with few training samples, which we expect would lead to better generalizability.
Gaussian Mixture Model Inspired by psychological study of the mental number line (Dehaene et al., 2003) and previous work on language modeling (Spithourakis & Riedel, 2018), we train a
Gaussian mixture model (GMM) to induce number prototypes.
A GMM is deﬁned as follows.
[Equation]
where Z is a latent variable representing the mixture component for random variable U , and N is the probability density function of a normal distribution, and π k , µ k , σ k ∈ R represent the mixing coefﬁcient, mean and standard deviation of the k -th Gaussian component.
We train a GMM on the number dataset X using the expectation-maximization (EM) or hard-EM algorithm and regard the means of the learned Gaussian components as our prototypes P = { µ 1 , · · · , µ m } .
We use three
GMM initialization methods described in Appendix A.
3. S IMILARITY F UNCTION
For SOM-induced prototypes, we deﬁne the following similarity function:
[Equation]
where function g is equal to the squashing function f deﬁned in Eq.2 if we do not apply log transformation before prototype induction and is the identity function
I otherwise.
β is a hyper-parameter set to 1 . 0 by default.
For GMM-induced prototypes, we can naturally use the posterior probability of the component assignment to deﬁne the similarity function.
[Equation]
3. E MBEDDING T RAINING
We now describe how to integrate our numeral embeddings into traditional word embedding methods for training.
We choose skip-gram with negative sampling (Mikolov et al., 2013a;b) as the word embedding method here, but many other word embedding methods such as CBOW (Mikolov et al., 2013a), HAL (Lund & Burgess, 1996) and GloVe (Pennington et al., 2014) can be used as well.
Skip-gram is a word embedding method based on the idea of context word prediction.
The training corpus C is regarded as a sequence of words ( x 1 , . . .
, x T ) .
For token x t , we deﬁne the preceding and following c tokens as the context of x t .
Skip-gram aims to maximize p ( x t + j | x t ) ( − c ≤ j ≤ c ) , the probability of a context word given the center word x t .
To formulate p ( x t + j | x t ) , skip-gram associates each word x i with two vector representations: the input embedding v ix t for being a center word and the output embedding v ox t for being a context word.
The input and output embeddings of all the words in the vocabulary V constitute matrices E I ∈ R D ×| V | and E O ∈ R D ×| V | respectively, where D is the dimension of word embeddings.
The conditional probability p ( x t + j | x t ) is then deﬁned to based on the dot product s ( x t + j | x t ) = v ix t T v ox t + j .
Nagative sampling is used to approximate the normalization factor for the conditional probability.
[Equation]
where σ denotes the sigmoid function, and P n ( x ) is the negative word sampling distribution used to draw k negative samples.
We modify skip-gram by computing numeral embeddings differently from non-numerical word embeddings.
We associate each prototype number with an input embedding and an output embedding.
The input and output embeddings of all the prototypes constitute matrices M I ∈ R D ×| P | and
M O ∈ R D ×| P | respectively.
For any numeral, we can compute its input and output embeddings by taking a weighted average of the prototype input and output embeddings respectively based on Eq.
and use them in exactly the same way as the embeddings of non-numerical words to compute the learning objective (Eq.6).
When drawing negative samples, we ﬁrst set the ratio of numerals and non-numerical words to their actual ratio in the training corpus, to guarantee a sufﬁcient number of numeral negative samples.
Then we sample numerals and non-numerical words separately from their respective distributions in the training corpus raised to the power of 34 .
During training, we optimize the objective function Eq.6 by back-propagating the gradient of error to update both the embedding matrices both the non-numerical word embedding matrices
E I , E O and the prototype number embedding matrices M I , M O .
In this way, the embeddings of non-numerical words and numerals are learned jointly in the same space.
We show an example in Figure 1.
[Table] [Table]
E XPERIMENTS AND R ESULTS
We evaluate our methods on four intrinsic and extrinsic tasks: word similarity, embedding numeracy, numeral prediction, and sequence labeling.
We report results of our methods based on SOM and GMM separately.
We choose the hyper-parameters (e.g., the number of prototypes, GMM initialization and training methods) using validation sets and report the best hyper-parameters for each experiment in Appendix B.
4. B ASELINES
NumAsTok
This baseline treats numerals and non-numerical words in the same way, which is very similar to the original skip-gram.
The vocabulary includes both high-frequency words and highfrequency numerals.
OOV non-numerical words are replaced with symbol UNK word and OOV numerals are replaced with symbol UNK num .
D-LSTM Character-level RNNs are often used to encode OOV words (Graves, 2013).
Here we apply an LSTM (Hochreiter & Schmidhuber, 1997) to the digit sequence of a numeral and use the last hidden state of the LSTM as the embedding of the numeral.
We use the embedding to compute the skip-gram objective function and propagate the gradients back to update the LSTM.
The vocabulary of digits is: { 0-9, ‘.’, ‘+’, ‘ − ’, ‘e’ } .
Fixed
This baseline ﬁxed embeddings for numerals with no training.
We deﬁne the embedding a numeral with value n as [ f ( n ); 1 ] /Z where f is the squashing function deﬁned in Eq.2, 1 ∈ R D − 1 is an all-ones vector, and Z is a constant used to keep the vector norm close to those of non-numerical words and is set to 2 × D by default.
We compare the vocabularies of different methods in Table 1.
Our methods, D-LSTM, and Fixed have ﬁnite non-numerical vocabularies but inﬁnite numeral vocabularies.
In contrast, the NumAsTok baseline has a ﬁnite numeral vocabulary and treats all the OOV numerals as UNK num .
4. W ORD S IMILARITY FOR N ON - NUMERICAL W ORDS
To ensure that our methods can still generate high quality embeddings for non-numerical words, we evaluate our trained embeddings on classical intrinsic word similarity tasks, including WordSim353, (Finkelstein et al., 2001), MEN (Bruni et al., 2014) and Simplex-999 (Hill et al., 2014).
We train 300 -dimensional word embeddings on the 1B Wikipedia dump and set the context window size to 5 , the number of negative samples to 5, and the vocabulary size to 3 × 10 5 .
We use the evaluation tools 1 provided by Jastrzebski (Jastrzebski et al., 2017).
Note that while the training data contains numerals, the evaluation tasks do not involve numerals and are only designed to evaluate quality of non-numerical word embeddings.
The results are shown in Table 2.
It can be seen that our methods can achieve scores comparable to those of the baselines.
The performance of SG trained on 100B GoogleNews is much better than all the other methods probably because of its much larger training corpus.
The results show that adding our numeral embedding methods into skip-gram does not harm the quality of non-numerical word embeddings.
Additional results of our methods can be found in Appendix C.
[Table]
4. M AGNITUDE AND N UMERATION OF E MBEDDINGS
Naik et al.
(2019) propose a framework for evaluating the ability of numeral embeddings to capture magnitude and numeration.
Given a target numeral, its embedding is evaluated against a set of numerals using the OVA (One-vs-All)
[Table]
SC (Strict Contrastive) and BC (Broad Contrastive) tests:
• OVA :
The embedding vector distance between the target and its nearest neighbor on the number line should be smaller than that between the target and
any
other numeral in the set.
• SC :
The embedding vector distance between the target and its nearest neighbor on the number line should be smaller than that between the target and its second nearest neighbors on the number line.
• BC :
The embedding vector distance between the target and its nearest neighbor on the number line should be smaller than that between the target and its furthest neighbors on the number line.
We follow the settings described by Naik et al.
(2019): for the magnitude evaluation, we run the tests using a set of 2342 numerals that are most frequent in Wikipedia-1B, whose embeddings are well learned by all the methods; and for the numeration evaluation, we run the tests using 113 English words that represent numbers (e.g., ‘three’, ‘billion’) sampled from the same corpus and we measure the distance between the target numeral embedding and the word embeddings of these words.
We report the accuracy of various embedding models on these three tests, along with the average rank (denoted as AVGR ) of the target numeral’s nearest neighbor among all the candidates based on their vector distances to the target.
We use the embeddings trained on Wikipedia-1B.
Table 3 shows the results.
The Fixed baseline has the best performance in the magnitude evaluation, which is unsurprising because the numeral embedding vector explicitly contains the (squashed) magnitude.
NumAsTok performs very well in the numeration evaluation, which is because the number-representing words used in the evaluation are high-frequency words and their embeddings are adequately trained.
Except for these two special cases, our methods can be seen to outperform the baselines with a large margin.
Wallace et al.
(2019) recently show that classic embeddings of numerals may contain magnitude information that can be extracted by neural networks.
Following their methodology, we conduct two probing tests on our 2342 numerals using multi-layer perceptrons and bilinear functions and then use the resulting models to predict distances between numerals in the OVA , SC , and BC tasks.
The results again show the advantage of our methods over the baselines.
See Appendix D for details.
4. N UMERAL P REDICTION
To evaluate the quality of numeral embeddings, we design a new numeral prediction task: choosing the right numeral from a set of candidates given the context of the numeral in a sentence.
We randomly sample 2000 sentences containing numerals from a subset of Wikipedia that is not used in training, with 600 for validation and 1400 for testing.
For each sentence, we use the ﬁve words preceding and following the target numeral as its context.
An example is shown below, where the ten bold words are the context and 2 . 31 is the target numeral.
In Hollywood, the average household size was [2.31] and the average family size was 3.00.
[Table]
We use all the 1400 numerals in the test set as the candidates from which one has to select the right numeral for each test sentence.
Given the learned word and numeral embeddings, we deﬁne two score functions to rank candidate numerals given the context.
Following the skip-gram model, we ﬁrst deﬁne the score of center numeral n predicting context word c j as s ( c j | n ) =
v oc j T v in and the score of context word
c j predicting the center numeral n as s ( n | c j ) =
v onT v ic j .
Our ﬁrst candidateranking score function S A is the sum of log probabilities of center numeral n predicting each context word c j .
We use softmax here to calculate the probability.
[Equation]
where V t is the vocabulary of non-numerical words and Z ( n ) is the normalization factor.
The other candidate-ranking score function S B is the sum of log probabilities of each context word c j predicting center numeral n .
[Equation]
where V n is the set of numerals in the dataset.
There are a few other possible score functions, but we ﬁnd that they lead to results similar to S A and S B .
We use three metrics to evaluate numeral prediction (Spithourakis & Riedel, 2018).
MdAE is the median of the absolute errors between the predicted and true numerals, MdAPE is the median of the absolute percentage errors between the predicted and true numerals, and AVGR is the average rank of the true numeral among the candidates.
Detailed formulas of the three metrics are shown in
Appendix E.
We train embeddings on Wikipedia-1B and report the evaluation results in the left part of Table 4.
Our methods signiﬁcantly outperform the NumAsTok and Fixed baselines on all the three metrics.
D-LSTM also performs well but needs more parameters and computing time than our methods.
We also conduct a slightly different numeral prediction task on the recently released Numeracy600K dataset (the Article Title part) (Chen et al., 2019).
This dataset contains 600k sentences with numerals and in each sentence, one numeral is selected and tagged with its order of magnitude.
There are eight possible orders of magnitude and the goal is to predict the correct one for the target numeral from its context.
To solve this multi-class classiﬁcation problem, we sample 100 numerals for each order of magnitude and use the mean of their numeral embeddings to create a ‘meta’ embedding; we then use these ‘meta’ embeddings to replace the numeral embeddings in the score functions S A and S B and the highest-scoring order of magnitude is returned.
We split the dataset to 450k sentences for training, 50k for validation and 100k for testing.
We use micro-F1 and macro-F1 in addition to AVGR as the evaluation metrics.
The result is shown in the right part of Table 4.
The result shows that our methods achieve much better performance compared to the baselines.
4. S EQUENCE L ABELING ON C USTOMER S ERVICE D ATA
To verify the effectiveness of our methods in practice, we evaluate our methods with a sequence labeling task on a dataset of customer service chat log from an online apparel shopping website.
[Table]
The task is to assign a label to each word or numeral in the dataset indicating its information type.
We shows two examples below:
[Equation]
[Table]
W, H, O are labels representing weight, height and ordinary word respectively.
We show the statistics of the dataset in Appendix G.
In order to better evaluate the generalizability, we create two additional test sets.
The ﬁrst one is created by ‘augmenting’ the original test set with new sentences containing slightly perturbed numerals.
For example, we can create new sentences by replacing ‘177’ in the above example with ‘176’ and ‘178’.
The second one contains ‘hard’ sentences from the original test set that do not have explicit cues for label prediction.
For example, the ﬁrst sentence above contains ‘kg’ and ‘cm’ that can greatly facilitate the prediction of W and H, but the second sentence above does not contain such cues and hence is a ‘hard’ sentence.
More details about the two test sets can be found in
Appendix F. Finally, we also test the low-resource settings in which only 30% or 10% of the training set is used.
We learn embeddings from the training set using our methods and the baselines and use a validation set to do model selection.
We plug the learned embeddings into the Neural-CRF model (Yang &
Zhang, 2018) 2 to do sequence labeling without using part-of-speech and character-level features and embedding ﬁne-tuning.
The results are shown in Table 5.
Our methods consistently outperform all the baselines on the Accuracy, Recall, and F1 metrics in different conﬁgurations.
NumAsTok trained with 100% training samples has the highest precision on the original and hard test sets probably because it learns highquality embeddings for high-frequency numerals included in its vocabulary; but its recall is lower than that of our methods, most likely because of its numeral OOV problem.
Comparing the results on the original and augmented test sets, we see that NumAsTok shows a more signiﬁcant drop in performance than the other methods, which suggests that NumAsTok does not generalize well because of the numeral OOV problem.
In the low-resource settings, the advantage of our methods over the baselines becomes even larger, indicating better generalizability and less annotation required for our methods to achieve a promising performance.
C ONCLUSION
In this paper, we propose two novel numeral embedding methods that represent the embedding of a numeral as a weighted average of a set of prototype numeral embeddings.
The methods can be integrated into traditional word embedding approaches such as skip-gram for training.
We evaluate our methods on four intrinsic and extrinsic tasks, including word similarity, embedding numeracy, numeral prediction, and sequence labeling, and show that our methods can improve the performance of numeral-related tasks and has better generalizability.
Our code and sample data can be found at path/to/code/ .
An important future direction is to handle numeral polysemy.
For example, the numeral “2019” may denote either a year or an ordinary number.
One potential method is to assign a different embedding to each sense of a numeral.
In this way, “2019” would have one embedding for representing a year and another for representing an ordinary quantity.
The similarity function would treat different senses of a numeral differently.
For example, the year sense of “2019” would be similar to the year sense of “19” but dissimilar to the sole sense of “2019.5”, while the quantity sense of “2019” would be similar to that of “2019.5”.
A GMM I NITIALIZATION
Both EM and hard-EM are sensitive to initialization and we use the initialization methods described in (Bl ¨ omer & Bujna, 2013).
We ﬁrst initialize the mean µ k of the k -th Gaussian component using one of the following three strategies:
Random initialization : choose µ k from X randomly.
This
is
suitable when
X contains
a wide range of numbers, e.g., numbers collected from Wikipedia.
SOM-based initialization : initialize µ k to p k ∈ P produced by the SOM method.
K-means initialization : run randomly initialized k-means on X and then use k-means centroids to initialize µ k .
We then assign the data samples to their closest means.
The standard deviation of the data samples assigned to the k -th mean becomes σ k .
B H YPER - PARAMETERS
We list all of the important hyper-parameters we tune for each model.
General hyper-parameters : embedding dimension, context window size, SGD learning rate, batch size, vocabulary size, etc.
SOM hyper-parameters : number of prototypes, stage of applying the log-squashing function (stage before prototype induction; stage
2: only in the similarity
function).
GMM hyper-parameters : number of prototypes, whether we apply the log-squashing function to the numerals, EM initialization (from SOM, random initialization, or k-means initialization), type of EM (hard-EM or soft-EM)
We show the values of the SOM and GMM hyper-parameters in Table 6 and the values of the general hyper-parameters of all the methods in Table 7.
We ﬁnd that the general hyper-parameters inﬂuence the performance of our methods and the baselines in the same way, so in most cases, these hyperparameters are set to be identical for all the methods.
For large training corpora (Wiki1B, Numeracy600k), we use 2048 as the batch size for D-LSTM, because D-LSTM consumes much more GPU memory.
We set the batch size of the other methods to 4096.
For the sequence labeling tasks, because the data is relatively small and conﬁned to a very speciﬁc domain (chat log from online apparel shops), we set a small vocabulary size of 500 for all the methods except NumAsTok and set the vocabulary size of NumAsTok to 550 to ensure that different methods have similar numbers of parameters for word embedding training.
Consequently, our methods have (500 + | P | ) × D parameters for word embedding training and NumAsTok has 550 × D parameters, where P is the prototype set, whose size is typically smaller than 50 , and D is the embedding dimension.
Table 6 also shows that the optimal number of prototypes is around 200–500 for the Wiki1B corpus and 10–25 for the much smaller sequence labeling dataset.
As a rule of thumb, we suggest setting the number of prototypes to (log N ) 2 , where N is the number of distinct numerals in the training corpus
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
C M ORE R ESULTS ON W IKIPEDIA -1B
[Figure]
We show the histograms of numerals in the Wikipedia-1B dataset and the prototypes learned by
SOM and GMM in Fig.2.
It can be seen that the prototypes induced by our methods have a similar distribution compared to the original numerals.
We also show some examples of prototypes and their nearest non-numerical words in Table 8.
We use the embedding trained by the SOM model with 200 prototypes on Wikipedia-1B, and use log transformation in the similarity function.
In addition, we select several typical numerals and non-numerical words and project their embeddings to 2D using t-SNE (Maaten & Hinton, 2008) (Figure 3).
We use embeddings learned on
Wikipedia-1B corpus using the SOM and GMM methods.
The examples and the ﬁgures show that our model does capture some semantic relations between numeral quantities and normal words.
We show the training speed of each embedding method on the Wikipedia-1B dataset in Table 9.
The batch size is set to 2048 for all the methods.
Our methods are slower than NumAsTok but are faster than D-LSTM.
[Table] [Table]
D P ROBING T ESTS
We apply two probing tests using neural network on our methods and baselines in order to compare their ability to encode magnitude information in a non-linear way.
The ﬁrst test is Decoding (predicting the numeral value from its embedding using MLP).
The second is Subtraction (predicting the difference between two numerals from their embeddings using MLP or BiLinear functions).
We illustrate the tasks and the models we use in Figure 4.
We ﬁrst create the datasets for the two probing tests based on the dataset from the magnitude evaluation of Section 4.3 (containing 2342 numerals).
For Decoding, the dataset can be directly used.
For Subtraction, we randomly sample 10 5 pairs of numerals ( n 1 , n 2 ) from the dataset and assign n 1 − n 2 as the prediction target.
Following Wallace et al.
(2019), we randomly split 80% of each dataset for training and 20% for testing.
We use SGD to optimize the mean square error (MSE) loss.
We report the root-mean-square error (RMSE) results for the two tasks in Table 10.
[Table]
The results show that our two methods are signiﬁcantly better than the baselines on Decoding.
On
Subtraction, they are better than the baselines when using BiLinear and are comparable to NumAsTok but much better than the other baselines when using MLP.
We found that the performance is very sensitive to the neural network architecture and MLP with two hidden layers performs best.
We then use the MLP2 models trained on Subtraction to determine the distance between two numerals when conducting the magnitude evaluation of Section 4.3.
The results are shown in Table 11.
The results show that our methods have better performance than the baselines overall.
One interesting observation is that, although our SOM based method has worse RMSE than NumAsTok as shown in Table 10, it outperforms NumAsTok in the magnitude evaluation.
E N UMERAL P REDICTION E VALUATION M ETRICS
We denote the target numeral by n i , the numeral with the highest ranking score by ˆ n i , and the rank of the target numeral by r i .
The error e i and percentage error pe i can be calculated as:
[Equation]
[Figure]
Then we use the median of the absolute errors, the median of the absolute percentage errors, and the average rank as the evaluation metrics.
[Equation]
[Table]
F A UGMENTED AND H ARD T EST S ETS IN S EQUENCE L ABELING
The augmented test set is created by reasonably perturbing the numerals in a sentence.
For example, for a numeral ‘173’ that describes height, we generate new samples by changing ‘173’ to ‘174’ or ‘175’ while keeping the other non-numerical words in the sentence unchanged.
For a decimal such as ‘1.7 meters’, we change it to ‘1.6’ or ‘1.8’.
The perturbation will not change the decimal places of numerals and will only change the quantity slightly, which makes the generated sentences reasonable.
The hard test set is created by manually collect ‘hard’ samples in the original test set.
Hard samples do not have explicit patterns, meaning that a numeral’s tag cannot be easily inferred by its adjacent words.
For example, tags of numerals followed by units like ‘cm’, ‘m’, ‘kg’, ‘years’ and ‘feet’ can be ﬁgured out easily, so we exclude them from the hard test set.
Customers are very likely to use ambiguous expressions like: ‘I’m 16.5, can I buy 24?’
, where 16.5 is about foot length and 24 is the shoe size.
These ambiguous sentences are included in the hard test set.
G S TATISTICS OF S EQUENCE L ABELING D ATASET
We show the statistics of the customer-service dataset in the Table 12.
The vocabulary is small because the dataset is conﬁned to a speciﬁc domain: online customer service chat log about apparel purchase.
In this dataset, most of the sentences are about sizes of various kinds of clothes and are very short and ambiguous.
H S EQUENCE LABELING RESULT WITH STANDARD DEVIATION .
[Table] [Table]