R EPRESENTATION D EGENERATION P ROBLEM IN T RAINING N ATURAL L ANGUAGE G ENERATION M ODELS
A BSTRACT
We study an interesting problem in training neural network-based models for natural language generation tasks, which we call the representation degeneration problem .
We observe that when we train a model in natural language generation tasks through likelihood maximization with weight tying trick, especially with big training dataset, most of the learnt word embeddings tend to degenerate and be distributed into a narrow cone, which largely limits the representation power of word embeddings.
We analyze the conditions and causes of this problem and propose a novel regularization method to address it.
Experiments on language modeling and machine translation show that our method can largely mitigate the representation degeneration problem and achieve better performance than baseline algorithms
I NTRODUCTION
Neural Network (NN)-based algorithms have made signiﬁcant progresses in natural language generation tasks, including language modeling (Kim et al., 2016;
Jozefowicz et al., 2016), machine translation (Wu et al., 2016; Britz et al., 2017; Vaswani et al., 2017; Gehring et al., 2017) and dialog systems (Shang et al., 2015).
Despite the huge variety of applications and model architectures, natural language generation mostly relies on predicting the next word given previous contexts and other conditional information.
A standard approach is to use a deep neural network to encode the inputs into a ﬁxed-size vector referred as hidden state 1 , which is then multiplied by the word embedding matrix (Vaswani et al., 2017; Merity et al., 2018; Yang et al., 2018; Press & Wolf, 2017).
The output logits are further consumed by the softmax function to give a categorical distribution of the next word.
Then the model is trained through likelihood maximization.
While studying the learnt models for language generation tasks, we observe some interesting and surprising phenomena.
As the word embedding matrix is tied with the softmax parameters (Vaswani et al., 2017; Merity et al., 2018; Inan et al., 2017; Press & Wolf, 2017), it has a dual role in the model, serving as the input in the ﬁrst layer and the weights in the last layer.
Given its ﬁrst role as input word embedding, it should contain rich semantic information that captures the given context which will be further used for different tasks.
Given its role as output softmax matrix, it should have enough capacity to classify different hidden states into correct labels.
We compare it with word embeddings trained from Word2Vec (Mikolov et al., 2013) and the parameters in the softmax layer [Figure] of a classical classiﬁcation task (we refer it as categorical embedding).
As shown in Figure 1, the word embeddings learnt from Word2Vec (Figure 1(b)) and the softmax parameters learnt from the classiﬁcation task (Figure 1(c)) are diversely distributed around the origin using SVD projection, but the word embeddings in our studied model (Figure 1(a)) degenerated into a narrow cone.
Furthermore, we ﬁnd any two pairs of word embeddings in our studied models are positively correlated.
Such phenomena are very different from those in other tasks and deteriorate model’s capacity.
As the role of the softmax layer, those parameters can not lead to a large margin prediction for good generalization.
As the role of word embeddings, the parameters do not have enough capacity to model the diverse semantics in natural languages (Yang et al., 2018; McCann et al., 2017).
We call the problem described above the representation degeneration problem .
In this paper, we try to understand why the problem happens and propose a practical solution to address it.
We provide some intuitive explanation and theoretical justiﬁcation for the problem.
Intuitively speaking, during the training process of a model with likelihood loss, for any given hidden state, the embedding of the corresponding ground-truth word will be pushed towards the direction of the hidden state in order to get a larger likelihood, while the embeddings of all other words will be pushed towards the negative direction of the hidden state to get a smaller likelihood.
As in natural language, word frequency is very low, the embedding of the word will be pushed towards the negative directions of most hidden states which drastically vary.
As a result, the embeddings of most words in the vocabulary will be pushed towards similar directions negatively correlated with most hidden states and thus are clustered together in a local region of the embedding space.
From the theoretical perspective, we ﬁrst analyze the extreme case of non-appeared words.
We prove that the representation degeneration problem is related to the structure of hidden states: the degeneration appears when the convex hull of the hidden states does not contain the origin and such condition is likely to happen when training with layer normalization (Ba et al., 2016; Vaswani et al., 2017; Merity et al., 2018).
We further extend our study to the optimization of low-frequency words in a more realistic setting.
We show that, under mild conditions, the low-frequency words are likely to be trained to be close to each other during optimization, and thus lie in a local region.
Inspired by the empirical analysis and theoretical insights, we design a novel way to mitigate the degeneration problem by regularizing the word embedding matrix.
As we observe that the word embeddings are restricted into a narrow cone, we try to directly increase the size of the aperture of the cone, which can be simply achieved by decreasing the similarity between individual word embeddings.
We provide empirical study on the proposed method on two tasks, language modeling and machine translation.
Experimental results show that the representation degeneration problem is mitigated, and our algorithm achieves superior performance over the baseline algorithms, e.g., with 2.
point perplexity improvement on the WikiText-2 dataset for language modeling and 1.08/0.93 point
BLEU improvement on WMT 2014 English-German/German-English tasks for machine translation.
R ELATED W ORK
Language modeling and machine translation are important language generation tasks.
Language modeling aims at predicting the next token given an (incomplete) sequence of words as context.
A popular approach based on neural networks is to map the given context to a real-valued vector as a hidden state, and then pass the hidden state through a softmax layer to generate a distribution over all the candidate words.
There are different choices of neural networks used in language modeling.
Recurrent neural network-based model and convolutional neural network-based model are widely used (Mikolov et al., 2010; Dauphin et al., 2017).
Neural Machine Translation (NMT) is a challenging task that has attracted lots of attention in recent years.
Based on the encoder-decoder framework, NMT starts to show promising results in many language pairs.
The evolving structures of NMT models in recent years have pushed the accuracy of NMT into a new level.
The attention mechanism (Bahdanau et al., 2015) added on top of the encoder-decoder framework is shown to be useful to automatically ﬁnd alignment structure, and single-layer RNN-based structure has evolved towards deep models (Wu et al., 2016), more efﬁcient
CNN models (Gehring et al., 2017), and well-designed self-attention models (Vaswani et al., 2017).
In this paper, we mainly study the expressiveness of word embeddings in language generation tasks.
A trend for language generation in recent years is to share the parameters between word embeddings and the softmax layer, which is named as the weight tying trick.
Many state-of-the-art results in language modeling and machine translation are achieved with this trick (Vaswani et al., 2017; Merity et al., 2018;
Yang et al., 2018; Press & Wolf, 2017).
The paper (Inan et al., 2017) shows that weight tying not only reduces the number of parameters but also has theoretical beneﬁts.
R EPRESENTATION D EGENERATION P ROBLEM
In this section, we empirically study the word embeddings learnt from sequence generation tasks, i.e., machine translation, and introduce the representation degeneration problem in neural sequence generation.
3. E XPERIMENTAL D ESIGN
Our analysis reported in this section is mainly based on the state-of-the-art machine translation model Transformer (Vaswani et al., 2017).
We use the ofﬁcial code (Vaswani et al., 2018) and set all the hyperparameters (the conﬁgurations of the base Transformer) as default.
We train the model on the WMT 2014 English-German Dataset and achieve 27.3 in terms of BLEU score.
Additionally, we also analyze the LSTM-based model (Wu et al., 2016) and ﬁnd the observations are similar.
In neural sequence generation tasks, the weights in word embeddings and softmax layer are tied.
Those parameters can be recognized not only as the inputs in the ﬁrst layer but also as the weights in the last layer.
Thus we compare this weight matrix with a word embedding matrix trained from conventional word representation learning task, and also compare it with the parameters in softmax layer of a conventional classiﬁcation task.
For simplicity and representative, we choose to use
Word2Vec (Mikolov et al., 2013) to obtain word embeddings and use MNIST as the classiﬁcation task trained with a two-layer convolutional neural network (Yue et al., 2018; Liu et al., 2016).
For the classiﬁcation task, We simply treat the row of the parameter matrix in the last softmax layer as the embedding for the category, like how the word embedding is used in the softmax layer for neural language generation model with weight tying trick.
To get a comprehensive understanding, we use low-rank approximation ( rank = 2 ) for the learned matrices by SVD and plot them in a 2D plane, as shown in Figure 1.
We also check the singular values distribution and ﬁnd that our low-rank approximation is reasonable: other singular values are much smaller than the chosen ones.
3. D ISCUSSION
In the classiﬁcation task, the category embeddings (blue triangles in Figure 1(c)) in the softmax layer are diversely distributed around the origin.
This shows the directions of category embeddings are different from each other and well separated, which consequently leads to large margin classiﬁcation results with good generalization.
For the word embeddings learnt from Word2Vec (Figure 1(b)), the phenomena are similar, the embeddings are also widely distributed around the origin in the projection space, which shows different words have different semantic meanings.
Observations on
GLOVE (Pennington et al., 2014) reported in Mu et al.
(2018) are also consistent with ours.
We observe very different phenomena for machine translation.
We can see from Figure 1(a) that the word embeddings are clustered together and only lying in a narrow cone.
Furthermore, we ﬁnd the cosine similarities between word embeddings are positive for almost all cases.
That is, the words huddle together and are not well separated in the embedding space.
Clearly, as the role of word embeddings, which are the inputs to the neural networks, the word representations should be widely distributed to represent different semantic meanings.
As the role of softmax in the output layer, to achieve good prediction of next word in a target sentence, a more diverse distribution of word embeddings in the space is expected to obtain a large margin result.
However, such representations of words limit the expressiveness of the learnt model.
We call such a problem the representation degeneration problem .
U NDERSTANDING THE P ROBLEM
We show in the previous section that in training natural language generation models, the learnt word embeddings are clustered into a narrow cone and the model faces the challenge of limited expressiveness.
In this section, we try to understand the reason of the problem and show that it is related to the optimization of low-frequency words with diverse contexts.
In natural language generation tasks, the vocabulary is usually of large size and the words are of low frequencies according to Zipf’s law.
For example, more than 90% of words’ frequencies are lower than 10e-4 in WMT 2014 English-German dataset.
Note that even for a popular word, its frequency is also relatively low.
For a concrete example, the frequency of the word “is” is only about 1% in the dataset, since “is” occurs at most once in most simple sentences.
Our analysis is mainly based on the optimization pattern of the low-frequency words which occupy the major proportion of vocabulary.
The generation of a sequence of words (or word indexes)
y =
( y 1 , · · · , y M ) is equivalent to generate the words one by one from left to right.
The probability of generating y can be factorized as P ( Y = y ) =
Π t P ( Y t =
y t | Y <t
= y <t ) , where y <t denotes for the ﬁrst t − 1 words in y .
Sometimes, the generative model also depends on other context, e.g., the context from the source sentence for machine translation.
To study the optimization of the word embeddings, we simply consider the generation of a word as a multi-class classiﬁcation problem and formally describe the optimization as follows.
Consider a multi-class classiﬁcation problem with M samples.
Let h i denote the hidden state before the softmax layer which can be also considered as the input features, i = 1 , · · · , M .
Without any loss of generality, we assume h
i is not a zero vector for all i .
Let N denote the vocabulary size.
The conditional probability of y i ∈ { 1 , · · · , N } is calculated by the softmax function: P ( Y i =
y i
| h i ) =
exp( (cid:104) h
i ,w
[Table]
) , where w l is the embedding for word/category l , l = 1 , 2 , · · · , N .
4. E XTREME C ASE : N ON - APPEARED WORD TOKENS
Note that in most NLP tasks, the frequencies of rare words are rather low, while the number of them is relatively large.
With stochastic training paradigm, the probabilities to sample a certain infrequent word in a mini-batch are very low, and thus during optimization, the rare words behave similarly to a non-appeared word.
We ﬁrst consider the extreme case of non-appeared word in this section and extend to a more general and realistic scenario in the next section.
We assume y i (cid:54)
=
N for all i .
That is, the N -th word with embedding w N does not appear in the corpus, which is the extreme case of a low-frequency rare word.
We focus on the optimization process of w N and assume all other parameters are ﬁxed and well-optimized.
By log-likelihood maximization, we have
[Equation]
As all other parameters are ﬁxed, this is equivalent to
[Equation]
where C i = (cid:80) N − 1 l =1 exp( (cid:104) h
i , w l (cid:105) ) and can be considered as some constants.
Deﬁnition 1.
We say that vector v is a uniformly negative direction of h
i , i = 1 , · · · , M , if (cid:104) v, h i (cid:105) < 0 for all i .
The following theorem provides a sufﬁcient condition for the embedding w N approaching unbounded during optimization.
We leave the proof of all the theorems in the appendix.
Theorem 1. A.
If the set of uniformly negative direction is not empty, it is convex.
B. If there exists a v that is a uniformly negative direction of h i , i = 1 , · · · , M , then the optimal solution of Eqn.satisﬁes (cid:107) w ∗ N (cid:107) = ∞ and can be achieved by setting w ∗ N = lim k
→ + ∞ k · v .
From the above theorem, we can see that if there exists a set of uniformly negative direction, the embedding w N can be optimized along any uniformly negative direction to inﬁnity.
As the set of uniformly negative direction is convex, w N is likely to lie in a convex cone and move to inﬁnity during optimization.
Next, we provide a sufﬁcient and necessary condition for the existence of the uniformly negative direction.
Theorem 2.
There exists a v that is a uniformly negative direction of a set of hidden states, if and only if the convex hull of the hidden states does not contain the origin.
Discussion on whether the condition happens in real practice From the theorem, we can see that the existence of the uniformly negative direction is highly related to the structure of the hidden states, and then affects the optimization of word embeddings.
Note that a common trick used in sequence generation tasks is layer normalization (Ba et al., 2016; Vaswani et al., 2017; Merity et al., 2018), which ﬁrst normalizes each hidden state vector into standard vector 2 , and then rescales/translates the standard vector with scaling/bias term.
In the appendix, we show that under very mild conditions, the space of hidden states doesn’t contain the origin almost for sure in practice.
4. E XTENSION TO RARELY APPEARED WORD TOKENS
In the previous section, we show that under reasonable assumptions, the embeddings of all nonappeared word tokens will move together along the uniformly negative directions to inﬁnity.
However, it is not realistic that there exist non-appeared word tokens and the weights of embedding can be trained to be unbounded with L2 regularization term.
In this section, we extend our analysis to a more realistic setting.
The key we want to show is that the optimization of a rarely appeared word token is similar to that of non-appeared word tokens.
Following the notations in the previous section, we denote the embedding of a rare word as w N and ﬁx all other parameters.
We study the optimization for this particular token with the negative log-likelihood loss function.
To clearly characterize the inﬂuence of w N to the loss function, we divide the loss function into two pieces.
Piece A w N contains the sentences that do not contain w N , i.e., all hidden states in such sentences are independently of w N and the ground truth label of each hidden state (denoted as w ∗ h ) is not w N .
Then w N can be considered as a “non-appeared token” in this set.
Denote the space of the hidden state in piece A w N as H A wN with probability distribution P A wN , where H A wN and
P A wN can be continuous.
The loss function on piece A w N can be deﬁned as
[Equation]
which is a generalized version of Eqn. 1.
Piece B w N contains the sentences which contain w N , i.e., w N appears in every sentence in piece B w N .
Then in piece B w N , in some cases the hidden states are computed based on w N , e.g., when the hidden state is used to predict the next token after w N .
In some other cases, the hidden states are used to predict w N .
Denote the space of the hidden state in piece B w N as H B wN with probability distribution P B wN .
The loss function on piece B w N can be deﬁned as
[Equation]
Based on these notations.
the overall loss function is deﬁned as
[Equation]
The loss on piece A w N is convex while the loss on piece B w N is complicated and usually nonconvex with respect to w N .
The general idea is that given the fact that L A wN ( w N ) is a convex function with nice theoretical properties, if P ( sentence s in A w N ) is large enough, e.g., larger than 1 − (cid:15) , and L B wN ( w N ) is a bounded-smooth function.
The optimal solution of L ( w N ) is close to the optimal solution of L A wN ( w N ) which can be unique if L A wN ( w N ) is strongly-convex.
A formal description is as below.
Theorem 3.
Given an α -strongly convex function f ( x ) and a function g ( x ) that satisﬁes its Hessian matrix H ( g ( x )) (cid:31) − βI , where I is the identity matrix, and | g ( x ) | < B . For a given (cid:15) > 0 , let x ∗ and x ∗ (cid:15) be the optimum of f ( x ) and (1 − (cid:15) )
f ( x ) + (cid:15)g ( x ) , respectively.
If (cid:15) < αα + β , then (cid:107) x ∗ − x ∗ (cid:15) (cid:107) 22 ≤ 4 (cid:15)B [Table]
We make some further discussions regarding the theoretical results.
In natural language generation, for two low-frequency words w and w (cid:48) , piece A w and A w (cid:48) has large overlaps.
Then the loss L A w ( w ) and L A w (cid:48) ( w (cid:48) ) are similar with close optimum.
According to the discussion above, the learnt word embedding of w and w (cid:48) is likely to be close to each other, which is also observed from the empirical studies.
A DDRESSING THE P ROBLEM
In this section, we propose an algorithm to address the representation degeneration problem.
As shown in the previous study, the learnt word embeddings are distributed in a narrow cone in the Euclid space which restricts the expressiveness of the representation.
Then a straightforward approach is to improve the aperture of the cone which is deﬁned as the maximum angle between any two boundaries of the cone.
For the ease of optimization, we minimize the cosine similarities between any two word embeddings to increase the expressiveness.
For simplicity, we denote the normalized direction of w as ˆ w , ˆ w = w (cid:107) w (cid:107) .
Then our goal is to minimize (cid:80) i (cid:80) j (cid:54)
=
i ˆ w iT ˆ w j as well as the original log-likelihood loss function.
By introducing hyperparameter γ to trade off the log-likelihood loss and regularization term, the overall objective is,
[Equation]
[Equation]
We call this new loss as MLE with Cosine Regularization (MLE-CosReg).
In the following, we make some analysis about the regularization term.
Denote R = (cid:80) Ni (cid:80) Nj (cid:54) =
i ˆ w iT ˆ w j , and denote the normalized word embedding matrix as ˆ W =
[ ˆ w 1 , ˆ w 2 , ..., ˆ w N ] T .
It is readily to check the regularizer has the matrix form as R
[Equation]
, where the Sum ( · ) operator calculates the sum of all elements in the matrix.
Since N is a constant, it sufﬁces to consider the ﬁrst term Sum ( ˆ W ˆ W T ) only.
Since ˆ W ˆ W T is a positive semi-deﬁnite matrix, all its eigenvalues are nonnegative.
Since ˆ w i is a normalized vector, every diagonal element of ˆ W ˆ W T is 1 .
Then the trace of ˆ W ˆ W T , which is also the sum of the eigenvalues, equals to N .
As the cosine similarities between the word embeddings are all positive.
According to Theorem 4, when ˆ W ˆ W T is a positive matrix, we have the largest absolute eigenvalue of ˆ W ˆ W T is upper bounded [Table] [Table] by Sum ( ˆ W ˆ W T ) .
Then minimizing R is equivalent to minimize the upper bound of the largest eigenvalue.
As the sum of all eigenvalues is a constant, the side effect is to increase other eigenvalues, consequently improving the expressiveness of the embedding matrix.
Theorem 4.
(Merikoski, 1984)
For any matrix A , which is a real and nonnegative n × n matrix.
The spectral radius, which is the largest absolute value of A ’s eigenvalues, is less than or equals to
Sum ( A ) .
E XPERIMENTS
We conduct experiments on two basic natural language generation tasks: language modeling and machine translation, and report the results in this section.
6. E XPERIMENTAL S ETTINGS 6.1. L ANGUAGE M ODELING
Language modeling is one of the fundamental tasks in natural language processing.
The goal is to predict the probability of the next word conditioned on previous words.
The evaluation metric is perplexity.
Smaller the perplexity, better the performance.
We used WikiText-2 (WT2) corpus, which is popularly used in many previous works (Merity et al., 2017; Inan et al., 2017; Grave et al., 2017).
WikiText-2 is sourced from curated Wikipedia articles and contains approximately a vocabulary of over 30,000 words.
All the text has been tokenized and processed with the Moses tokenizer (Koehn et al., 2006).
Capitalization, punctuation and numbers are retained in this dataset.
AWD-LSTM (Merity et al., 2018) is the state-of-the-art model for language modeling.
We directly followed the experimental settings as in Merity et al.
(2018) to set up the model architecture and hyperparameter conﬁgurations.
We used a three-layer LSTM with 1150 units in the hidden layer and set the size of embedding to be 400.
The ratio for dropout connection on recurrent weight is kept the same as Merity et al.
We trained the model with Averaged Stochastic Gradient
[Figure]
We normalize the so
Descent.
Our implementation was based on open-sourced code 3 by Merity et al.
For our proposed MLE-CosReg loss, we found the hyperparameter γ is not very sensitive and we set it toin the experiments.
We added neural cache model (Grave et al., 2017) to further reduce perplexity.
6.1. M ACHINE T RANSLATION
Machine Translation aims at mapping sentences from a source domain to a target domain.
We focus on English → German and German → English in our experiments.
We used the dataset from standard WMT 2014, which consists of 4.5 million English-German sentence pairs and has been widely used as the benchmark for neural machine translation (Vaswani et al., 2017; Gehring et al., 2017).
Sentences were encoded using byte-pair encoding (Sennrich et al., 2016), after which we got a shared source-target vocabulary with about 37000 subword units.
We measured the performance with tokenized case-sensitive BLEU (Papineni et al., 2002).
We used state-of-the-art machine translation model Transformer (Vaswani et al., 2017), which utilizes self-attention mechanism for machine translation.
We followed the setting in Vaswani et al.
(2017) and
used the ofﬁcial code (Vaswani et al., 2018) from Transformer.
For both English →
German and German → English tasks, we used the base version of Transformer (Vaswani et al., 2017), which has a 6-layer encoder and 6-layer decoder, the size of hidden nodes and embedding are set to 512.
For English → German task, we additionally run an experiment on the big version of Transformer, which has 3x parameters compared with the base variant.
All the models were trained with Adam optimizer, and all the hyperparameters were set as default as in Vaswani et al.
γ is set to 1 as in the experiments of language modeling.
6. E XPERIMENTAL R ESULTS
We present experimental results for language modeling in Table 1 and machine translation in Table 2.
For language modeling, we compare our method with vanilla AWD-LSTM (Merity et al., 2018) in three different settings, without ﬁnetune, with ﬁnetune and with further continuous cache pointer.
Our method outperforms it with 0.8/1.7/2.0 improvements in terms of test perplexity.
For machine translation, comparing with original base Transformer (Vaswani et al., 2017), our method improves performance with 1.08/0.93 for the English → German and German → English tasks, respectively, and achieves 0.54 improvement on the big Transformer.
Note that for all tasks, we only add one regularization term to the loss function, while no additional parameters or architecture/hyperparameters modiﬁcations are applied.
Therefore, the accuracy improvements purely come from our proposed method.
This demonstrates that by regularizing the similarity between word embeddings, our proposed MLE-CosReg loss leads to better performance.
6. D ISCUSSION
The study above demonstrates the effectiveness of our proposed method in terms of ﬁnal accuracy.
However, it is still unclear whether our method has improved the representation power of learnt word embeddings.
In this subsection, we provide a comprehensive study on the expressiveness of the model learnt by our algorithm.
For a fair comparison with the empirical study in Section 3, we analyze the word embeddings of our model on English → German translation task.
We project the word embeddings into 2-dimensional space using SVD for visualization.
Figure 2(a) shows that the learnt word embeddings are somewhat uniformly distributed around the origin and not strictly in a narrow cone like in Figure 1(a).
This shows that our proposed regularization term effectively expands word embedding space.
We also compare the singular values for word embedding matrix of our learnt model and the baseline model, as shown in the Figure 2(b).
According to the ﬁgure, trained with vanilla Transformer, only a few singular values dominate among all singular values, while trained with our proposed method, the singular values distribute more uniformly.
Again, this demonstrates the diversity of the word embeddings learnt by our method.
C ONCLUSION AND F UTURE W ORK
In this work, we described and analyzed the representation degeneration problem in training neural network models for natural language generation tasks both empirically and theoretically.
We proposed a novel regularization method to increase the representation power of word embeddings explicitly.
Experiments on language modeling and machine translation demonstrated the effectiveness of our method.
In the future, we will apply our method to more language generation tasks.
Our proposed regularization term is based on cosine similarity.
There may exist some better regularization terms.
Furthermore, it is interesting to consider other approaches instead of regularization to enrich the representation of word embeddings.  A CKNOWLEDGEMENTS
This work was partially supported by National Basic Research Program of China (973 Program) (grant no. 2015CB352502), NSFC (61573026) and BJNSF (L172037). We would like to thank Zhuohan Li and Chengyue Gong for helpful discussions, and the anonymous reviewers for their valuable comments on our paper. A PPENDICES A P ROOFS
In this section, we add proofs of all the theorems in the main sections.
Theorem 1. A.
If the set of uniformly negative direction is not empty, it is convex.
B. If there exists a v that is a uniformly negative direction of h i , i = 1 , · · · , M , then the optimal solution of Eqn.satisﬁes (cid:107) w ∗ N (cid:107) = ∞ and can be achieved by setting w ∗ N = lim k
→ + ∞ k · v .
Proof.
The ﬁrst part is straight forward and we just prove the second part.
Denote v as any uniformly negative direction, and k as any positive value.
We have lim k
→ + ∞ (cid:80) Mi =1 log(exp( (cid:104) h
i , k · v (cid:105) )
+ C i ) =
(cid:80) Mi =1 log( C i ) .
As (cid:80) Mi =1 log( C i ) is the lower bound of the objective function in Eqn. 2 and the objective function is convex, we have w ∗ =
lim k →∞ k · v is local optimum and also the global optimum.
Note that the lower bound can be achieved only if (cid:104) h i , w N (cid:105) approaches negative inﬁnity for all i .
Then it is easy to check for any optimal solution (cid:107) w ∗ N (cid:107) = ∞ .
Theorem 2.
There exists a v that is a uniformly negative direction of a set of hidden states, if and only if the convex hull of the hidden states does not contain the origin.
Proof.
We ﬁrst prove the necessary condition by contradiction.
Suppose there are M hidden states in the set.
If the convex hull of h i , i = 1 , · · · , M , contains the origin and there exists a uniformly negative direction (e.g. v ).
Then from the deﬁnition of convex hull, there exists α i , i = 1 , · · · , M , such that (cid:80)
Mi α
i h i = 0 , α i ≥ 0 and (cid:80) i α i = 1 .
Multiplying v on both sides, we have (cid:80) i
α i (cid:104) h
i , v (cid:105) = 0 , which contradicts with (cid:104) h i , v (cid:105) < 0 for all i .
For the sufﬁcient part, if the convex hull of h i , i = 1 , · · · , M , does not contain the origin, there exists at least one hyperplane H that passes through the origin and does not cross the convex hull.
Then it is easy to check that a normal direction of the half space derived by the H is a uniformly negative direction.
The theorem follows.
Theorem 3.
Given an α -strongly convex function f ( x ) and a function g ( x ) that satisﬁes its Hessian matrix H ( g ( x )) (cid:31) − βI , where I is the identity matrix, and | g ( x ) | < B . For a given (cid:15) > 0 , let x ∗ and x ∗ (cid:15) be the optimum of f ( x ) and (1 − (cid:15) )
f ( x ) + (cid:15)g ( x ) , respectively.
If (cid:15) < αα + β , then (cid:107) x ∗ − x ∗ (cid:15) (cid:107) 22 ≤ 4 (cid:15)B α − (cid:15) ( α + β ) .
Proof.
We ﬁrst prove the function (1 − (cid:15) )
f ( x ) + (cid:15)g ( x ) is α − (cid:15) ( α + β ) -strongly convex.
Let’s consider the Hessian matrix of it.
As f ( x ) is α -strongly convex and H ( g ( x )) (cid:31) − βI , using the deﬁnition of positive-deﬁnite matrix, the following inequality holds:
[Equation]
To make it clear, we omit x here.
Then for the Hessian matrix of (1 − (cid:15) )
f ( x ) + (cid:15)g ( x ) , we have: ∀ v, v T ( H ((1 − (cid:15) ) f + (cid:15)g ) − ( α − (cid:15) ( α + β )) I ) v (9) = v T ( H ((1 − (cid:15) )
f ) + H ( (cid:15)g ) − (1 − (cid:15) ) αI + (cid:15)βI ) v (10) = v T ((1 − (cid:15) )
H ( f ) + (cid:15) H ( g ) − (1 − (cid:15) ) αI + (cid:15)βI ) v (11) = v T ((1 − (cid:15) ) ( H ( f ) − αI ) + (cid:15) ( H ( g ) + (cid:15)βI )) v (12) = v T ((1 − (cid:15) )
{+( H ( f ) − αI )) v + v T ( (cid:15) ( H ( g ) + βI )) v (13) = (1 − (cid:15) ) v T ( H ( f ) − αI ) v + (cid:15)v T ( H ( g ) + βI ) v (14) > (1 − (cid:15) )
 + (cid:15)(15) = 0 .+}
Thus, H ((1 − (cid:15) )
f ( x )+ (cid:15)g ( x )) − ( α − (cid:15) ( α + β ))
I is positive-deﬁnite, which means (1 − (cid:15) )
f ( x )+ (cid:15)g ( x ) is α − (cid:15) ( α + β ) -strongly convex.
Then, with the properties in strong convexity, we have:
[Equation]
B C OMPUTATION OF THE C OSINE R EGULARIZATION
In this section, we provide analysis of the computational cost of the proposed regularization term.
Proposition 1.
The cosine regularization in Eqn. 6 can be computed in Θ( N ) time where N is the size of vocabulary.
Proof.
The regularization term can be simpliﬁed as below:
[Equation]
From the above equations, we can see that we only need to compute the sum of all the normalized word embedding vectors, and thus the computational time is linear with respect to the vocabulary size N .
In this section, we show the condition on the existence of the uniformly negative direction holds almost for sure in practice with models where layer normalization is applied before last layer.
Let h 1 , h 2 , · · · , h n be n vectors in R d Euclidean space.
Let −→ 1 / −→ 0 be the d-dimensional vector ﬁlled with ones/zeros respectively.
By applying layer normalization we have:
[Equation]
where µ i and σ 2 i are the mean and variance of entries in vector h
i , g and b are learnt scale/bias vectors and (cid:12) denotes the element-wise multiplication.
Here we simply assume that σ 2 i and each entry in g and b are not zero (since the exact zero value can be hardly observed using gradient optimization in real vector space).
If the convex hull of h (cid:48) 1 , h (cid:48) 2 , · · · , h (cid:48) n contains the origin, then there exist λ 1 , λ 2 , · · · , λ n , such that:
[Equation]
[Equation]
By combining 25 and 26 we have: n (cid:88)
[Equation]
Denote bg =
( b 1 g 1 , b 2 g 2 , · · · , b d g d ) , thus we have (cid:80) ni
{+=
 λ+}
i h i −−→ 1 µ i σ i = − bg .
Since −→ 1 T h i −−→ 1 µ i σ i =
for all i , there exist λ 1 , λ 2 , · · · , λ n that satisfy Eqn. 26 only if −→ 1 T bg = {+−−→ 1 T (cid:80) ni =
 λ i h i −−→ 1 µ i σ i = − (cid:80)+}
[Equation]
=
[Equation]
1 µ i σ i = 0 , which can hardly be guaranteed using current unconstrained optimization.
We also empirically verify this.
D (S UB )W ORD F REQUENCY DISTRIBUTION ON WMT E NGLISH -G ERMAN AND W IKI T EXT 2 D ATASETS
In this section, we show the (sub)word frequency distribution on the datasets we use in experiments in the Figure 3.
[Figure]
[Figure]