Tiny Word Embeddings Using Globally Informed Reconstruction (2020)

Abstract: We reduce the model size of pre-trained word embeddings by a factor of 200 while preserving its quality. Previous studies in this direction created a smaller word embedding model by reconstructing pre-trained word representations from those of subwords, which allows to store only a smaller number of subword embeddings in the memory. However, previous studies that train the reconstruction models using only target words cannot reduce the model size extremely while preserving its quality. Inspired by the observation of words with similar meanings having similar embeddings, our reconstruction training learns the global relationships among words, which can be employed in various models for word embedding reconstruction. Experimental results on word similarity benchmarks show that the proposed method improves the performance of the all subword-based reconstruction models.

## introduction word embeddings form the basis for many natural language processing (nlp) applications, e.g., text classification @xcite and machine translation @xcite . however, widely used pretrained word embeddings such as fasttext @xcite are considerably large, thereby making it difficult to develop nlp applications in limited memory environments such as mobile devices. for example, fasttext 1 (crawl-300d-2m-subword) requires approximately 2 gb of memory. in previous studies, the model size has been reduced by reconstructing word embeddings from characters @xcite and character n-grams @xcite . as the number of characters or character n-grams, is significantly smaller than that of words, reconstructing word embeddings with accuracy from these subwords can reduce the model size while preserving the performance of applications. 2 as shown in figure 1 , existing methods reconstruct word embeddings from subword embeddings and mimic the corresponding pre-trained word embeddings. these methods rely only on local information of subwords and pre-trained word embeddings. to improve the performance of word embedding reconstruction, we propose a global loss function that uses words other than the target word as clues. inspired by the observation of words with similar meanings having similar embeddings in pre-trained word embeddings, our reconstruction training learns the similarity among word embeddings. our method can be easily applied to any method for reconstructing a word embedding from subwords, regardless of the unit of the subword or the network structure. experimental results on word similarity tasks @xcite show that our global loss function improves the performance of word embedding reconstruction in all previous methods. when the proposed method was applied to the method based on a self-attention mechanism for reconstructing word embeddings from character n-grams @xcite , the model size was reduced to 74 mb (1/30) while preserving 97% of the quality of the original word embeddings. furthermore, even if the model size was reduced to 12 mb (1/200), 86% of the quality was preserved. cos cos 𝐿 "#$%" 𝐿 &"#'%" reconstructed embedding for target word pre-trained embedding for target word pre-trained embedding for other words figure 2: overview of our global similarity loss 2 word embeddings reconstruction based on global similarity loss we denote the pre-trained embeddings of word @xmath0, respectively. as in previous studies @xcite @xcite , we reconstruct a word embedding e w of word w as êw , from a set of subwords φ(w) by minimizing the following loss function. to improve word embedding reconstruction, we employ a loss function based on global information in addition to the loss function of equation ( 1 ), which depends only on the local information of the target word, w. as shown in figure 2 , we consider the relationship between the reconstructed and pre-trained embeddings for the target word and the relationship between the reconstructed embedding of the target word and the pre-trained embedding of other words. we define the global loss function using cosine similarity foot_1 among word embeddings as follows: in this study, we sample n words from w in each training batch. to balance the similarity distribution among the selected n words, we first select the top-n/2 words that have a high cosine similarity to the target word, and then randomly select the rest from the training batch. finally, we minimize the loss function that combines equations ( 1 ) and ( 2 ), as given below: we evaluate the effectiveness of the globally informed reconstruction of word embeddings using word similarity tasks foot_2 @xcite . our experiment employs the following five datasets: rubenstein-goodenough dataset (rg, 65 word-pairs) @xcite , miller-charles dataset (mc, 30 word-pairs) @xcite , wordsim-353 (ws, 353 wordpairs) @xcite , men test collection (men, 3, 000 word-pairs) @xcite , and stanford rare word similarity dataset (rw, 2, 034 word-pairs) @xcite . the performance of each method is evaluated using micro averaged spearman's rank correlation coefficient between the cosine similarities of the word embeddings and gold-standard similarities. ## conclusion we proposed a loss function that considers global relationships among words for the reconstruction of pre-trained word embeddings from subword embeddings. experimental results on word similarity benchmarks show that the proposed method improves the performance of all the reconstruction networks. by applying the proposed method, we can compress 2gb of fasttext to 12mb while preserving the quality of the original word embeddings. this method will help develop nlp applications in limited memory environments, e.g., mobile devices.