Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings (2022)

Abstract: Recent studies have determined that the learned token embeddings of large-scale neural language models are degenerated to be anisotropic with a narrow-cone shape. This phenomenon, called the representation degeneration problem, facilitates an increase in the overall similarity between token embeddings that negatively affect the performance of the models. Although the existing methods that address the degeneration problem based on observations of the phenomenon triggered by the problem improves the performance of the text generation, the training dynamics of token embeddings behind the degeneration problem are still not explored. In this study, we analyze the training dynamics of the token embeddings focusing on rare token embedding. We demonstrate that the specific part of the gradient for rare token embeddings is the key cause of the degeneration problem for all tokens during training stage. Based on the analysis, we propose a novel method called, adaptive gradient gating(AGG). AGG addresses the degeneration problem by gating the specific part of the gradient for rare token embeddings. Experimental results from language modeling, word similarity, and machine translation tasks quantitatively and qualitatively verify the effectiveness of AGG.

## introduction neural language models have been developed with various architectures during recent years @xcite @xcite . despite the improvement in model architectures, models usually share the same process for input and output. they process token embeddings as inputs to compute contextualized features and subsequently project the features into a categorical distribution of tokens at the output softmax layer whose weight is token embedding matrix @xcite @xcite . recent studies have determined that the learned embedding distribution is biased in a common direction, thereby resulting in a narrow cone-shaped anisotropy @xcite @xcite . this phenomenon, named the representation degeneration problem by @xcite , increases the overall similarity between embeddings, and leads to a problem in which the expressiveness of the token embeddings decreases. therefore, it is difficult for the model to learn the semantic relationship between the tokens and to generate high quality texts. existing studies addressing this problem suggest methods that apply post-processing or regularization techniques to all token embeddings based on the observed phenomena owing to the degeneration problem @xcite @xcite @xcite . although these works improve the quality of token embeddings and generated texts, it is still not clear how token embeddings become degenerate during training procedure. also, there exists the problem of over regularization for the token embeddings whose semantic relationships are trained well because the above methods are applied for all token embeddings. in this study, we conduct empirical studies about training dynamics of token embeddings, focusing on rare token embeddings. by observing the initial training dynamics of token embeddings grouped based on appearance frequency, we hypothesize that the degeneration of the rare token embeddings triggers the degeneration of the embeddings of the remaining tokens. we show that the entire degeneration problem is mitigated by only freezing rare tokens during training, and we demonstrate that the main cause of the entire degeneration problem is the specific part of the gradient for rare token embeddings. this gradient part pushes away rare token embeddings from the feature vector of the non-rare targets in the current training sample. based on the analysis, we propose a new method, adaptive gradient gating (agg). with a dynamic grouping of rare tokens at each training step, agg solves the entire degeneration problem by gating a specific part of the gradient that is solely about rare tokens. because agg is optimized to target the main cause of the degeneration problem, rare token embeddings, it can prevent the over regularization problem about frequent token embeddings which occurs in other methods addressing the degeneration problem. the proposed method is evaluated in three tasks: language modeling, word similarity, and machine translation. the agg outperforms the baseline and other existing methods in all tasks. in addition, it shows compatibility with other method that addresses the neural text degeneration problem. via qualitative studies, we identify a correlation between our method and the frequency bias problem of learned embeddings @xcite . ## experiments we evaluate our method on various tasks including language modeling, word similarity, and machine translation. in the language modeling task, we focus on verifying the diversity of the generated texts. we test the learning of the semantic relationships between tokens on the word similarity task. finally, we evaluate the quality of generated texts on the machine translation task. for all the experimental results below, we adopt the state-of-the-art model architecture as a baseline to properly demonstrate the effectiveness of our method. every detail on the experiment, such as model hyper-parameters and training configurations, regard the reproducibility are provided in appendix b. ## analysis of agg 6.1 ablation study in our method, agg, we introduce two gate vectors, g 1 , and g 2 , to handle the gradient for rare and very rare token embeddings. we conduct experiments on these gate vectors. ## conclusion in this study, we analyzed the training dynamics of the token embeddings concerning the representation degeneration problem of the learned embeddings, focusing on the rare tokens. based on the analysis, we propose an adaptive gradient gating method that solves the problem by solely handling the training for rare token embeddings. experiments and qualitative studies in various tasks of text generation demonstrate the effectiveness of our method. beyond the two-level approximation of rarity of rare tokens which is applied to our study, addressing multiple levels of rarity can be an interesting region to study for the future work. corporation through hmc/kia-snu ai consortium fund, and snu-naver hyperscale ai center.