Revisiting Representation Degeneration Problem in Language Modeling (2020)

Abstract: Weight tying is now a common setting in many language generation tasks such as language modeling and machine translation. However, a recent study reveals that there is a potential flaw in weight tying. They find that the learned word embeddings are likely to degenerate and lie in a narrow cone when training a language model. They call it the representation degeneration problem and propose a cosine regularization to solve it. Nevertheless, we prove that the cosine regularization is insufficient to solve the problem, as the degeneration is still likely to happen under certain conditions. In this paper, we revisit the representation degeneration problem and theoretically analyze the limitations of the previously proposed solution. Afterward, we propose an alternative regularization method called Laplacian regularization to tackle the problem. Experiments on language modeling demonstrate the effectiveness of the proposed Laplacian regularization.

## introduction language modeling is a fundamental task in natural language processing, applications include machine translation @xcite , image captioning @xcite and speech recognition @xcite , to name a few. in the era of deep learning, a general model architecture usually contains a word embedding layer as input, multiple layers to encode word context as a fixed-size hidden state, and a softmax layer to transform the hidden-state into a categorical distribution of the next word @xcite @xcite @xcite . while in practice, the parameters of the embedding layer and the softmax layer are usually shared, which is called weight tying @xcite . however, we argue that the cosine regularization might not be the best choice for solving this problem, and the reasons are: i) the cosine regularization minimizes similarities between any two word embeddings without considering whether they are semantically close or not. but we wish two words with similar semantics stay close in the embedding space. ii) although the cosine regularization improves language generation performance, it does not fundamentally solve the representation degeneration problem. we prove that the degeneration still exists when there exists a certain regularization structure. finally, we analyze the general condition of degeneration and show that there still are many low-frequency words that meet the condition and thus degenerate. therefore, we argue that the degeneration is still likely to happen even with cosine regularization. motivated by these issues, we propose an alternative laplacian regularization to tackle the representation degeneration problem. as the distributional hypothesis @xcite states: two words that occur in similar contexts tend to have similar meanings. the general idea of laplacian regularization is to minimize the squared euclidean distance between two word embeddings when they have large context similarity. in contrast to cosine regularization, laplacian regularization prevents minimizing all similarities of word pairs indiscriminately. although the laplacian regularization does not theoretically solve the degeneration problem either, we empirically demonstrate that it achieves better performance in most cases of language modeling experiments, and word embeddings are less likely to degenerate. in summary, the main contributions of our work are listed as follows. ## representation degeneration problem in this section, we introduce the notations and review the representation degeneration problem. given a vocabulary of words (indices) @xmath0, ..., n }, and a text corpus represented as a sequence of words @xmath1, ..., y m ), where y @xmath2. the joint probability of sequence y is factorized into a product of conditional probabilities using the chain rule. next, we investigate the optimization process of word embeddings. we follow the analysis in @xcite and only focus on the extreme case of a non-appeared word w n in the following analysis, since the analysis can be extended to the case of rarely appeared words by applying theorem 3 in @xcite . assume y @xmath3, which means the n -th word with embedding w n does not appear in the corpus. under the loglikelihood maximization objective and fixing all other parameters, we write the objective function for optimizing variable w n as follows. and can be considered as a constant. let v be a uniformly negative direction of h i , i.e., v t h @xmath4. it is easy to see that the optimal solution of eq. ( 3 ) can be achieved by setting w * @xmath5. the authors prove that such a uniformly negative direction v exists if and only if the convex hull of the hidden states does not contain the origin. they discuss that the condition is very likely to hold, especially when layer normalization is applied. we further observe that the condition holds almost for sure in actual language modeling, even without layer normalization. from the above analysis, we have an intuition for the representation degeneration problem. we can see that the embedding w n can be optimized along any uniformly negative direction to infinity. as the set of uniformly negative direction is convex, w n is likely to lie in a convex cone and move to infinity during optimization. this conclusion also applies to the case of rarely appeared words to a large extent @xcite . as most words in natural language are low-frequency words according to zipf's law, the learned word embeddings tend to degenerate and lie in a narrow cone, which limits the model's semantic expressiveness. notably, @xcite also show that the learned word embeddings overly encode word frequency information rather than semantic information, which implicitly supports the existence of the degeneration problem. ## solutions to the problem in this section, we first introduce the solution proposed in @xcite . then we theoretically analyze the limitations of the previously proposed method. finally, we propose an alternative regularization to tackle the problem. ## experiments in this section, we conduct experiments on language modeling task to demonstrate the effectiveness of our method. ## discussion and future work from the above study, we analyze the limitations of the cosine regularization and empirically demonstrate the effectiveness of our proposed laplacian regularization method. however, there is also an issue in it. to this end, we make further discussion in this section. hopefully, it will provide some inspirations for later researches. there is one question that must be asked: does the laplacian regularization completely solve the representation degeneration problem? unfortunately, we cannot give a definite positive answer. from the above empirical studies, we have evidence that the laplacian regularization can ease the degeneration to an extent. however, there is also a failure case, the model cannot converge when the value of λ is set too large. because if λ is sufficiently large, the regularization term will dominate the objective value and all word embeddings will be optimized to huddle together. the premise of this failure case is that the similarity weights s ij are all positive. interestingly, we observe that almost similarity weights are positive, even though they are calculated by the cosine function, which further suggests that there may exist some intrinsic mechanism that causes the degeneration phenomenon. we will leave it to future study. despite this issue, the laplacian regularization is also a general framework to incorporate the external knowledge of word pair relations like semantic knowledge graph and synonymy/antonymy, which might bring benefits in certain applications. in addition, we find that the representation degeneration problem is highly related to the softmax bottleneck problem @xcite . as a matter of fact, we consider they are two sides of the same problem. the softmax bottleneck states that a language model's output log-probability matrix should be high-rank for natural language. but the rank is limited by the embedding dimension d and thus the expressiveness of a model is compromised. the softmax bottleneck problem roots in an insufficient embedding dimension d. however, what really matters is the rank of the embedding matrix rather than the dimension d. as for the representation degeneration problem, it reveals that for a n × d word embedding matrix, the rank could be smaller than d since word embeddings are correlated and lie in a narrow cone. thus, the true crux here is about the spectral density distribution of the embedding matrix. there is also evidence @xcite ) that an embedding matrix with more uniformly distributed singular values better improves downstream task performance. thus, we suggest two lines of researches to enhance the expressiveness of a language model. the first is to learn better expressive word embeddings @xcite @xcite . the second is to design better expressive output/activation functions @xcite @xcite @xcite . nonetheless, we want to clarify that only focusing on the embedding/output layers is far more insufficient for language modeling, since it is the middle layers that provide the major non-linearity which matters most for the expressiveness. exploring new architectures like the bert @xcite and the transformer-xl @xcite is also essential for the future study. the laplacian regularization has been widely used in various fields like semi-supervised learning @xcite , face recognition @xcite , graph embedding @xcite , and metric learning @xcite , to name a few. however, to the best of our knowledge, it has not been applied for regularizing the word embedding matrix yet. we are probably the first to propose the laplacian regularization on word embeddings. ## conclusion in this paper, we study the representation degeneration problem that is first pointed out by @xcite . we theoretically analyze the limitations of the previously proposed solution. afterward, we propose an alternative laplacian regularization method to tackle the problem. experiments on language modeling demonstrate the effectiveness of our method. in the future study, we will try to further investigate this problem from the perspective of spectral density of embedding matrix.