U n C l E : Explicitly Leveraging Semantic Similarity to Reduce the Parameters of Word Embeddings (2021)

Abstract: Natural language processing (NLP) models often require a massive number of parameters for word embeddings, which limits their application on mobile devices. Researchers have employed many approaches, e.g. adaptive inputs, to reduce the parameters of word embeddings. However, existing methods rarely pay attention to semantic information. In this paper, we propose a novel method called Unique and Class Embeddings (UnClE), which explicitly leverages semantic similarity with weight sharing to reduce the dimensionality of word embeddings. Inspired by the fact that words with similar semantic can share a part of weights, we divide the embeddings of words into two parts: unique embedding and class embedding. The former is one-to-one mapping like traditional embedding, while the latter is many-to-one mapping and learn the representation of class information. Our method is suitable for both word-level and sub-word level models and can be used to reduce both input and output embeddings. Experimental results on the standard WMT 2014 English-German dataset show that our method is able to reduce the parameters of word embeddings by more than 11x, with about 93% performance retaining in BLEU metrics. For language modeling task, our model can reduce word embeddings by 6x or 11x on PTB/WT2 dataset at the cost of a certain degree of performance degradation.

## introduction recently, deep learning models like lstm networks @xcite , and transformer @xcite based models like bert @xcite have made remarkable progress in the field of natural language processing (nlp). however, the sizes of these models are usually too humongous, making them difficult to deploy on low-resource machines such as edgecomputing devices, sensors or mobile phones. for a typical neural language model, especially one with a large vocabulary size, the large memory consumption of the model is mostly due to the need of storing the input and output word embedding matrices. the dimension of recurrent layers (e.g., lstm), which corresponds to the hidden state, is typically small and independent of the vocabulary size. in contrast, the dimensions of the embedding and softmax layers grow with the size of vocabulary, which can easily reach the scale of hundreds of thousands. as a result, parameter matrices of the embedding and softmax layers are often responsible for the majority of memory consumption of a neural language model. for transformer-based models like universal transformer @xcite and albert @xcite , which share parameters across layers, word embeddings also consume a large amount of memory. researchers have sought to reduce word embeddings through many efforts, such as, adaptive method @xcite or designing mechanisms for weight sharing @xcite . however, previous methods rarely take semantic information into consideration. some words are very similar regarding the semantics. for example, "better" and "best" are similar, except that the former is comparative level and the latter is superior level. also, their suffixes "er" and "est", as subwords, have similar semantics to some extent. we hold the same assumption as @xcite and @xcite that learning independent embeddings of large dimensions causes more redundancy in the embedding vectors, as the inter-similarity among words is ignored. in this paper, we propose uncle, an intuitive method for reducing the dimensionality of word embeddings by explicitly leveraging semantic similarity with weight sharing. we divide traditional word embedding processes into two parts: unique embedding and class embedding. there is a oneto-one mapping between words and unique embeddings like traditional word embedding. for class embeddings, words in the same class will share a single class embedding. before being fed to the downstream neural network, each word will get its unique embedding and class embedding through its word index and class index. then, the two embeddings are concatenated to generate the final embedding of the word. our contributions are mainly three-fold: ## related works adaptive method there are many efforts towards reducing word embeddings. @xcite extend the adaptive softmax @xcite to input word representations and assign more capacity to frequent words and reduce the capacity for less frequent words with the benefit of reducing overfitting to rare words. @xcite point that projective embedding, including transformer-xl @xcite and albert @xcite ) is a special case of adaptive method when the number of clusters is one. similarly, @xcite group words into blocks based on their frequencies, and then refine the clustering iteratively by constructing the weighted low-rank approximation for each block. other works like @xcite @xcite also propose hierarchical clustering of words, but they mainly focus on training effectively rather than reducing word embeddings. weight sharing another line of work employs weight sharing to reduce the number of parameters. @xcite allocate all the words in the vocabulary into a table. the words in the same row share the row vector and the words in the same column share the column vector. slim embedding @xcite randomly shares the structured parameters at both the input and output embedding layers of the recurrent neural language models. press and wolf (2017) and @xcite share input and output embeddings to improve language model while significantly reducing the number of network parameters. @xcite use a deeper network with significantly fewer parameters to replace word embedding layer. @xcite introduce a novel knowledge distillation technique for training a student model with a significantly smaller vocab-ulary by mixing teacher vocabulary-tokenized with student vocabulary-tokenized words. different from the above researches, our approach leverages semantic similarity to reduce word embeddings. words with similar semantic will be grouped into the same class, and share a part of weights. inter-similarity among words similar to our method, several earlier studies @xcite @xcite are also based on the hypothesis that the inter-similarity among words is ignored in the conventional way of constructing word vectors. @xcite represent infrequent words' embeddings with frequent words' embeddings by sparse linear combinations. our method is different in that words in the same class share a part of weights, no matter their frequency. of course, in the same class, there are also infrequent words and frequent words. the training of both frequent and infrequent words will update shared weights. to some extent, frequent words will help infrequent words via shared weights. @xcite also construct the embeddings with a few basis vectors. for each word, the composition of basis vectors is determined by a hash code rather than precomputation. however, this method assigns the same length of codes to each word without considering the significance of downstream tasks. @xcite further compress word embeddings by adaptively assigning different lengths of codes to each word by considering downstream tasks. @xcite employ an autoencoder architecture to transform real-valued embeddings into binary embeddings while preserving semantic information. different from these works above that implicitly leverage semantic similarity, our method explicitly construct semantic classes. more importantly, @xcite ; @xcite focus on compressing the pretrained embeddings and our method is a novel representation of word embeddings. the reduction of parameters comes from the smaller size of dimensionality of unique embeddings rather than from compressing pre-trained embeddings. and our method also don't need to modify the objective function to learn the sparse linear combinations or the code-book. so our method is easier to implement and have a wider range of applications. ## experiments we demonstrate the performance of our method on two sequence-to-sequence modeling tasks: language modeling and machine translation. for language modeling, we compare our method with awd-lstm @xcite . for machine translation, we use transformer @xcite as base model. we also compare our method with projective method and adaptive method. following the same setting as in @xcite , we share input and output embeddings for all models in this paper. task dataset type vocab. language wikitext-2 word 33278 model ptb word 10000 mt wmt14 en-de subword 40724 ## conclusion in this paper, we propose a novel and intuitive method, uncle, to explicitly construct semantic classes to reduce the parameters of word embeddings. we divide the traditional word embedding into unique embedding and class embedding. class embedding is shared by all words belonging to the same class. so we can reduce traditional word embedding at a larger ratio. in the future, we plan to apply our method to other tasks, such as question answering and sentiment analysis. in addition, we will try to apply uncle on even larger corpora, such as the one billion word benchmark, which contains 768m word tokens and has a vocabulary of about 800k word types.