R uminating W ord R epresentations with R andom N oise M asking
Anonymous authors
Paper under double-blind review
A bstract
We introduce a training method for better word representation and performance, which we call GraVeR ( Gra dual Ve ctor R umination).
The method is to gradually and iteratively add random noises and bias to word embeddings after training a model, and re-train the model from scratch but initialize with the noised word embeddings.
Through the re-training process, some noises can be compensated and other noises can be utilized to learn better representations.
As a result, we can get word representations further ﬁne-tuned and specialized in the task.
On six text classiﬁcation tasks, our method improves model performances with a large gap.
When GraVeR is combined with other regularization techniques, it shows further improvements.
Lastly, we investigate the usefulness of GraVeR 1 .
I ntroduction
Most machine learning methodologies can be formulated to get computational representations from real-life objects (e.g., images, languages, and sounds) and then get high-level representations using model architectures.
Therefore, there have been two main approaches to improve model performances: (1) starting with better representations (Melamud et al., 2016; Peters et al., 2018), and (2) building more sophisticated architectures that can extract important features and generate higherlevel representations (Vaswani et al., 2017; Conneau et al., 2017).
For better initial representations, many NLP researchers have used pretrained word vectors trained on substantially large corpus through unsupervised algorithms, such as word2vec (Mikolov et al., 2013a), GloVe (Pennington et al., 2014), and fastText (Bojanowski et al., 2016).
The pretrained word vectors represent the general meaning of words and increase the model performances on most NLP tasks (Turian et al., 2010).
In addition to the algorithms, word vector post-processing research (Faruqui et al., 2015; Vuli´c et al., 2017; Mrkˇsi´c et al., 2017; Jo & Choi, 2018) have attempted to enrich the pretrained representations using external resources.
They simply modiﬁed the values of the vector representations in some way, and showed improved performance.
It implies that we can get further improvement through better initial representations.
When training NLP models, we ﬁrst initialize word representations with pretrained word vectors and then update both the model parameters and the word representations.
However, in the training process, the model performance can be limited due to the initial word vectors.
For example, the pretrained word representations have general meanings of words, but, in some tasks, the words might not be used for the general meaning.
Although the gap between meanings can be learned through the training process, it could fail.
Since the pretrained representations are trained from a huge dataset, and their objective functions are based on language modeling, the word vectors are naturally biased to general and frequently used meaning.
Besides, the word vectors are updated through gradient descent algorithms, so the values are changed slightly.
The word vectors are thus easy to converge on local minima.
Therefore, our method starts with an idea–using the word representations ﬁne-tuned by a training process as pretrained word vectors in the next re-training process.
Then, word vectors can be trained to learn more appropriate representations to the task.
However, the model must be overﬁtted, and then the word representation would be stuck in local minima.
Thus, we add random noise and bias to the word representations before the re-training processes, in order to prevent the model from overﬁtting and take the word representations far from the local minima.
In this paper, we propose a simple training framework to ﬁnd better representations by adding random noise and bias on the word vectors during iterative training processes, which we call GraVeR
( Gra dual Ve ctor R umination).
We expect that the model makes good uses of the re-training processes with noises, both for learning better representation and for model regularization.
R elated W orks
The representations ﬁne-tuned by GraVeR can be considered as pretrained representations from the previous training process.
Also, GraVeR utilizes word-level noises, which are used for model regularization.
2.1 P retrained R epresentations
Pretrained
Embedding Vector is also called pretrained word representation.
According to the distributional representation hypothesis (Mikolov et al., 2013b), pretrained embedding vectors are composed of pairs of (token, n-dimensional ﬂoat vector).
Unsupervised algorithms (e.g., word2vec (Mikolov et al., 2013a), GloVe (Pennington et al., 2014), fastText (Bojanowski et al., 2016)) learn the word vectors on substantial corpora to represent general meanings of words.
The pretrained embedding vectors are widely used to initialize the word vectors in models.
Pretrained Embedding Model is suggested to get a deep representation of each word in the context.
Previous research McCann et al.
; Peters et al. ; Devlin et al. (2018) trained deep architecture models and then utilized the model weights to represent words by using the outputs of the models. Although recent advanced pretrained representations (Peters et al., 2018; Devlin et al., 2018) show good performances, we take the pretrained embedding vector approach because (1) retraining processes in the pretrained embedding models are very expensive and (2) we use word-level noises whereas the embedding models use token-level embeddings combined with position embeddings.
2.2 W ord - level N oises
Adding noises to input data is an old idea (Plaut et al., 1986).
However, a small number of studies were on word-level noises, since the noises on words can distort words’ meaning.
Word Dropping.
NLP tasks that utilize the text as a form of sentence and phrase have considered each word as features.
However, too many features can lead models to be overﬁtted to the training data due to the curse of dimensionality.
Therefore, the easiest way to reduce the number of features is to drop words in the sentence at random.
Word Embedding Perturbation.
Miyato et al.
(2016) tried to perturb word vectors and used them in the adversarial training framework for model regularization.
Cheng et al.
(2018) utilized the noises to build a robust machine translation model.
Also, Zhang & Yang (2018) considered the perturbation as a data augmentation method.
The previous works added the noises to all word embeddings.
It can regularize the model weights, but they ignored the change of word representations and its re-usability.
On the other hand, our method gradually adds noises to word embeddings by controlling the amount of noise.
Also, iterative training processes that re-use ﬁne-tuned word representation as pretrained word vectors for the next training process can beneﬁt from the noises and make better word representations.
2.3 R egularization T echniques
Some research explained that the normalization could be used for model regularization (van
Laarhoven, 2017; Luo et al., 2018;
Ho ﬀ er et al., 2018).
Dropout (Srivastava et al., 2014) is applied to neural network models, masking random neurons with 0.
Dropout randomly and temporarily removes the neural activations during training, so the masked weights are not updated.
As a result, the model is prevented from over-tuning on speciﬁc features, which involves regularization.
Batch Normalization (BN) (Io ﬀ e & Szegedy, 2015) normalizes the features according to minibatch statistics.
Batch normalization enables the features to avoid covariate shift–the weight gradients are highly dependent on the previous layers’ gradients.
Besides, batch normalization speeds up the training process by reshaping loss function.
Layer Normalization (LN) (Ba et al., 2016) also utilizes mini-batch statistics to normalize the features.
The di ﬀ erence with batch normalization is that layer normalization normalizes the inputs across the features.
The statistics are computed across each feature, which is the same for all the feature dimensions.
P roposed M ethod 3.1 O verall P rocess
The overall process of GraVeR is illustrated in Figure 1.
GraVeR is applied to a conventional training framework, but it needs a meta-level approach that trains the model again.
The iterative training process will also be denoted as a meta-epoch.
When a training process ﬁnishes, we extract the ﬁne-tuned word embeddings ( W (cid:48) ) and add bias ( W (cid:48) − W ) weighted by 1/[MaskingRate] .
The additive bias is optional but it takes the word embeddings far from the minima in which the ﬁne-tuned embeddings converged.
Next, we add maskers ﬁlled with random values to a portion of W (cid:48) , and then re-train the model from scratch with the noised word embeddings.
Observing the validation performance in the training process, we select the best-performing model and the ﬁne-tuned embeddings.
The additional details in GraVeR will be described.
In short, GraVeR is a training framework that adds random noises (and bias) to the ﬁne-tuned word embeddings and then repeats the training process with the word embeddings.
3.2 G ra V e R D etails
The details in GraVeR are based on the idea that moderate noises are required to change word vector distribution, but the noise scale should not be too large to distort the word vector distribution.
Random Noises and Bias.
We have two hyperparameters related to the noise maskers; Masking
Rate and Noise Scale .
The masking rate denotes a portion of word vectors masked in every (re- )training process.
We set the masking rate to 20% of the vocabulary size as default.
The maskers are ﬁlled with random values in a range, which is denoted as noise scale.
The default is sampled from a uniform distribution (-1,1) since word vectors in the range are widely used.
Well-deﬁned perturbation methods like Gaussian kernels (Vilnis & McCallum, 2014) can be an extension.
Next, we add a bias to the ﬁne-tuned word embeddings ( W (cid:48) )
in order to take W (cid:48) to the same embedding space with better initial values.
Intuitively, since we initialize with the word embeddings ﬁne-tuned in the previous training process, the embedding space in the next training process is the same as the space in the previous training process.
And, the di ﬀ erence of ﬁne-tuned embedding and initial embedding ( | W (cid:48) − W | ) in the space means the representations learned by a training process, which improved the performance in the previous training process.
So, we add the bias to W (cid:48) , weighted by 1 / [MaskingRate] where 0 < MaskingRate ≤ 1 in order to give a large bias when a small number of word vectors are noised; 1 / [MaskingRate] × ( W (cid:48) − W ).
When the validation performance decreases, the bias is [MaskingRate] × ( W − W (cid:48) ) for small noise.
When selecting a portion of words to noise, we use word frequency driven from training data.
Intuitively, if the representations of frequently used words are changed, which means the input data are changed a lot, the high-level representation such as sentence representation are largely a ﬀ ected.
Gradualness.
While following the aforementioned process, we change the number of random maskers according to validation performance.
If the re-trained model’s validation performance increases, we move the maskers to the next frequently used words.
Otherwise, we gradually increase the number of random maskers without moving them so that the maskers make noises on the words again in the next re-training process.
As a result, GraVeR can process both words in the previous training process and words in the current training process.
This gradualness lets the re-training processes beneﬁt from newly generated noises again, which makes GraVeR dynamic.
In the end, overall algorithms of GraVeR are summarized as follows: Algorithm [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
MaskedWords ← NextFrequentlyUsedWords
W ← W (cid:48)
+ 1 / MaskingRate
× ( W (cid:48) − W ) elseMaskedWords ← MaskedWords + NextFrequentlyUsedWords
W ← W − MaskingRate × ( W (cid:48) − W ) end if W [ MaskedWords ] ← W [ MaskedWords ]
+ U ( − 1 , 1) words are masked
GraVeR can be applied to any conventional training frameworks, so the method is independent of model architectures in that most NLP models use word-level embeddings.
The random noises might disturb the representation, but some of the noises which harm the performance are compensated during the re-training processes.
In contrast, other noises are used to update the word vectors over the initial values.
Besides, the additive bias makes the use of random noises stable.
By using this re-training with the noise process, we expect that the model is prevented from overﬁtting.
Meanwhile, the model is incrementally ﬁtted to the validation set through early-stopping in a training process and meta-level early-stopping in re-training processes.
Therefore, the model keeps being ﬁtted to the validation set with regularization, showing better performance on the test set since the model performance on the validation set is normally correlated to the performance on the test set.
E xperiment 4.1 D atasets
We prepare three topic classiﬁcation datasets: DBpedia ontology (Lehmann et al., 2015), YahooAnswer 2 (Chang et al., 2008), AGNews .
We also prepare two sentiment classiﬁcation datasets: Yelp reviews (Zhang et al., 2015), IMDB (Maas et al., 2011).
YahooAnswer dataset is used for two different tasks: classify upper-level categories and classify lower-level categories.
The data statistics are presented in Appendix.
We assign 15% of each train set to validation set, and each dataset has its own test set.
The validation set is used for early-stopping both at every epoch and every meta-epoch.
We use all words tokenized by space and all symbols using 300 dimensional embedding space.
4.2 C lassifier
We use TextCNN (Kim, 2014) classiﬁers.
The model consists of 2 convolutional layers withchannels and 16 channels, respectively.
We also adopt multiple sizes of kernels–2, 3, 4, and 5, followed by ReLU activation (Hahnloser et al., 2000) and max-pooling.
The kernels are concatenated after every max-pooling layer.
Finally, the features are classiﬁed into the classes using fully connected layer.
Although this model has few parameters (136K) compared with recent high-performance models like BERT (Devlin et al., 2018), we use this model to utilize multiple re-training processes.
Also, we employ simple techniques for model improvements such as Dropout and Layer Normalization to get similar performances to recent models and justify the use of a simple model as a basic classiﬁer.
The vanilla classiﬁer and the tuned classiﬁer are denoted as TextCNN base and TextCNN tune , respectively.
Additional comparison with very deep models is described in Appendix.
We optimize the model using Adam (Kingma & Ba, 2014) with 1e-3 learning rate and earlystopping.
If the validation accuracy does not increase over ﬁve epochs, we stop model training.
Initial word embeddings are random if we do not mention explicitly.
4.3 B aseline I mplementation
We cannot fully compare our method with Word Embedding Perturbation (Miyato et al., 2016) because we do not use adversarial training framework.
Instead, random noises are added to all word embeddings, as other word embedding perturbation methods did (Cheng et al., 2018; Zhang & Yang, 2018).
Inorder to compare the e ﬀ ect of regularization, we implement ﬁve regularization (including normalization) methods.
Word dropping is implemented in the pre-processing part, which removes random words in the text.
We set the random probability p as 0.1.
Dropout (Srivastava et al., 2014) is added to the ﬁnal fully connected layer with dropout probability 0.1, which performs the best in our experiments.
Batch Normalization (Io ﬀ e & Szegedy, 2015) is located between every convolutional layer and an activation function, as used in the original paper.
Layer Normalization (Ba et al., 2016) is implemented in the same position.
We report the performance averaged over ﬁve runs.
R esults 5.1 P erformance
[Table]
.78 ± . 75.98 ± 3. + GraVeR + 0.69(.02) + 9.01(.32) + 10.31(.87)
+ 2.21(.10) + 0.91(.31)
+ 9.71(.53)
Firstly, we present the e ﬀ ect of GraVeR on major pretrained word embeddings, as presented in Table 1.
We use three pretrained word embeddings: word2vec ( w2v ) (Mikolov et al., 2013a)
GoogleNews-vectors-negative300.bin , GloVe ( glv ) (Pennington et al., 2014) glove.42B.300d.txt , fastText ( ftt ) (Bojanowski et al., 2016)
wiki-news-300d-1M-subword.vec , and 1 random embedding.
GraVeR improves the model performance on most of the datasets, and the largest performance gain is in random embeddings.
Random embeddings with GraVeR even perform better than pretrained embeddings in a few datasets.
The result implies that we can learn better word representations through GraVeR in any embedding space since we train a model from scratch except for word embeddings.
However, with GloVe, GraVeR is not e ﬀ ective on the model performance because the
[Table]
+ GraVeR + 0.51(.02) + 5.29(.17) + 4.76(.07) + 2.38(.39) + 1.34(.25) + 2.62(.27)
+ DO (p=0.1) 98.00 ± .
67.01 ± .
44.79 ± .
88.93 ± .
62.18 ± .
86.39 ± .
+ DO&GraVeR + 0.48(.03) + 5.33(.21) + 5.11(.17) + 2.44(.13) + 1.14(.15) + 2.55(.09) + BN 97.86 ± .
67.68 ± .
44.72 ± .
88.07 ± .
62.24 ± .
85.53 ± .
+ BN&GraVeR + 0.66(.01) + 5.00(.42) + 5.17(.07) + 2.71(.49) + 1.29(.27) + 2.18(.11) + LN 98.18 ± .
67.10 ± .
45.15 ± .
88.64 ± .
62.89 ± .
85.18 ± .
+ LN&GraVeR + 0.43(.03)
+ 6.50(.16) + 6.51(.16)
+ 2.39(.10) + 0.80(.10) + 2.99(.51) + DO&BN 97.91 ± .
67.87 ± .
45.10 ± .
88.32 ± .
62.51 ± .
84.89 ± .
+ DO&BN&GraVeR + 0.58(.01) + 4.49(.12) + 4.53(.06) + 2.71(.12) + 0.75(.21) + 2.71(.46) + DO&LN(TextCNN tune ) 98.49 ±
[Table]
+ DO&LN&GraVeR
[Table]
distribution of word vectors in the pretrained embeddings is already good enough for the tasks.
In contrast, GraVeR makes substantial improvements when using relatively poor embeddings for the tasks.
Thecomparison with regularization techniques is presented in Table 2 (Top).
The result shows that the performance gain by GraVeR is larger than other regularization methods.
We also present the results when our method is combined with other regularization methods in Table 2 (Bottom).
The results show that GraVeR positively matches with the other regularization techniques, further improving the model performance.
Figure 2 also reports that our method deﬁnitely increases validation performance, which results in improvements in the test set.
Comparisons of a recent model (e.g., BERT) without extra data resources and word embedding perturbation methods will be discussed in Further Analysis ( § 6).
[Table]
5.2 W ord D istributions
To analyze GraVeR with respect to word representation, we extract the word representations updated on DBpedia dataset and present the list of top-20 nearest words of a cue word in Table 3.
In order to reduce the e ﬀ ect of randomness, we use GloVe for initial embedding.
We can see that the word vectors are further ﬁne-tuned and even ﬁnd other similar words not shown in the embedding ﬁnetuned once.
These results imply that our method can change the word vector distribution to be further ﬁne-tuned to the tasks.
We also present another cue word results and visualization of top-100 nearest word vector distribution using t-SNE (Maaten & Hinton, 2008) in Appendix.
F urther A nalysis
From this Section, we mainly use TextCNN tune to show the e ﬀ ect of GraVeR on the model whose performance is similar to state-of-the-art models.
6.1 R andom N oise and B ias
[Table]
Gaussian Noise
[Table]
No Additive Bias
[Table]
NoiseScale 0.
[Table]
Noises.
The range of random values ﬁlled in the maskers also a ﬀ ects the performance.
We ﬁrst re-train the model without any modiﬁcation to observe the e ﬀ ect of the re-training processes.
The model shows slightly better performance within a few meta-epochs, but it becomes overﬁtting, as we expected (see Table 4).
The performance when we use Gaussian noise instead of the uniform distribution is also presented.
It shows a comparable but slightly worse result than our default settings.
When the noise range is 0, the noise only consists of the additive bias, which also shows marginally worse performance.
The word perturbation method performs slightly better than just re-training, but there are large gaps between variations of GraVeR.
[Table]
Masking Rate
The amount of noise added by GraVeR is an important factor in that some noises should be small enough to be corrected during the re-training processes, while other noises should be large enough to change the word vector distribution.
We ﬁrst change the masking rate of how much random maskers move in every re-training process.
The larger the masking rate becomes, the more words are masked in a re-training process, so the noise increases.
Conversely, the amount of noise decreases as the masking rate becomes small.
The e ﬀ ect of the masking rate is presented in Table 5.
Masking Rate=0.1 also shows good performance, but it needs 2x re-training processes more than Masking Rate=0.2 .
We thus use 0.2 as a default.
is to increase the number of maskers when the validation performance decreases.
[Table]
Gradualness Policy Our proposed method is to increase the number of maskers when the validation performance decreases in order to make noises on the words masked in the previous re-training process again.
Otherwise
[Table]
we simply move to the next frequently used words.
We try changing the gradualness that (1) do not have gradualness
[Table]
(2) have gradualness only when the validation performance increase, which is reverse to our proposed method
[Table]
and (3) always have gradualness regardless of the validation performance.
The result is presented in Table 6.
Among the ablations
[Table]
our proposed approach performs the best
[Table]
6.3 O n other M odel A rchitecture
[Table]
We opt our method to the transformer-based classiﬁer.
The transformer classiﬁer has 300 embedding dimension with positional-embedding, 32 batch size, 512 sequence length.
It also has 10 multi-heads but only 1 encoder layer.
Stacking more than 2 encoder layers harms the performance; We guess that the number of training set is not enough to train the model parameters.
We do average-pooling to the encoded sentence vectors and use linear layer in order for classiﬁcation.
The other parameters follow the default setting of Pytorch nn.TransformerEncoderLayer.
Table 7 shows that GraVeR works well even in other model architectures.
[Table]
[Table]
retraining by T raining D ata
[Table] Table 8 shows the classiﬁcation performance according to the usage of pretrained word embeddings.
Gen means general representation trained on a large corpus, which is the pretrained embeddings what we mentioned in § 5.1 and bert-base-uncased from huggingface (Wolf et al., 2019).
Sp denotes specialized representation trained using only training data.
We set the hyperparameter for training as default settings used in their API.
Despite a small number of parameters, TextCNN tune with GraVeR shows comparable performance with BERT ( Sp ).
GraVeR’s improvement is even similar to glv ( Gen ), which means GraVeR can complement the information from external resources, in this case, Common Crawl (42B tokens, 1.9M vocab).
Although there is a performance gap with pretrained BERT, the advantages of GraVeR are (1) do not need any extra data resources, and (2) do not need to use very deep model architectures consisting of a bunch of parameters.
Therefore, GraVeR must be an attractive option when using pretrained word embeddings.
D iscussion
Re-training Cost.
Although GraVeR shows strong performance, its re-training processes take 1/[MaskingRate] more times than conventional training process.
However, we showed that a small model with GraVeR even performs on par with recent huge models.
When considering the parameter size, for example, TextCNN tune has 136K parameters and it needs ﬁve times re-training process in our default setting MaskingRate=0.2 while BERT has 110M parameters with one ﬁne-tuning process.
Then, the parameters need to be trained are 780K and 110M, respectively;
i.e.
141 times cheaper.
Furthermore, the training time for small model must be faster since it can be trained with larger mini-batch size.
As a representation learning.
GraVeR’s representations are learned from a given training set only.
In Table 8, general word embeddings trained from large data shows worse performance than domain (or data) specialized word embeddings.
That is, in order to solve the task, the word representation should be specialized in given data.
By using GraVeR, we can easily get specialized (further ﬁne-tuned) representation only with the random noise including bias and the iterative training processes.
Besides, we believe that using a sophisticated noising trick instead of simple random makes further improvement in GraVeR.
C onclusion
We propose GraVeR, which adds random noises and bias to word embeddings in order to change the word vector distribution and regularize a model.
Through the re-training process, we can make the use of noises to learn better representations.
In the experiments, as the model incrementally ﬁts the validation set, GraVeR largely improves model performances.
We expect that our general training approach can be used to a various models to improve model performances.