A Simple and Effective Usage of Word Clusters for CBOW Model (2020)

Abstract: We propose a simple and effective method for incorporating word clusters into the Continuous Bag-of-Words (CBOW) model. Specifically, we propose to replace infrequent input and output words in CBOW model with their clusters. The resulting cluster-incorporated CBOW model produces embeddings of frequent words and a small amount of cluster embeddings, which will be fine-tuned in downstream tasks. We empirically show our replacing method works well on several downstream tasks. Through our analysis, we show that our method might be also useful for other similar models which produce word embeddings.

## introduction word embeddings have been widely applied to various natural language processing (nlp) tasks. these embeddings can be pretrained on a large corpus and carry useful semantic information. one of the most well-known methods for obtaining word embeddings is based on continuous bag-of-words (cbow) @xcite and there have been many research efforts to extend it. in this paper, we focus on incorporating word clusters into cbow model. each word cluster consists of words that function similarly. by aggregating such words, we can alleviate data sparsity, even though each of those words is infrequent. in the past few years, word clusters have been applied to various tasks, such as named-entity recognition @xcite , machine translation @xcite and parsing @xcite . many word clustering algorithms can be applied to a raw corpus with different languages and help us obtain word clusters easily without additional language resources. in our method, we keep only very frequent words and replace the other words with their clusters for both input and output words in the cbow model. this is motivated by the fact that word clusters are more reliable than infrequent words. thus, only very frequent word embeddings and a small amount of cluster embeddings are produced as the output. when fine-tuning the trained embeddings on downstream tasks, the embeddings of infrequent words within one cluster are initialized by the embedding of their cluster to increase the coverage of pretrained word embeddings. since word embeddings are usually trained on the large-scale dataset. for making clusters on the large-scale dataset, we choose bidirectional, interpolated, refining, and alternating (bira) predictive exchange algorithm @xcite foot_0 as our clustering method. because bira was reported to be faster than many other methods. notably, it can produce 800 clusters on 1 billion english tokens in 1.4 hours. we evaluate our cluster-incorporated word embeddings foot_1 on downstream tasks, in which finetuning of word embeddings is involved. the evaluation for frequent words, for which our method also works well, on word similarity tasks can be found in appendix a. for the downstream tasks, we choose language modeling (lm) tasks, which are a fundamental task in nlp, as well as two machine translation (mt) tasks. to verify the effect of word clusters across different languages, 8 typologically diverse languages are further selected for the lm task. finally, an analysis is provided for our method. in summary, our replacing method can be used to improve the embeddings of frequent and infrequent words, to reduce the number of word embeddings and to make training more effective. ## related work a number of related research efforts have been done to help to learn better word embeddings aiming at different aspects. for example, @xcite proposed an extension that learns multiple embeddings per word type. @xcite proposed methods for estimating embeddings for different languages in a single shared embedding space. there is also a lot of work that incorporates internal information of words, such as character-level information @xcite and morpheme information @xcite . our research aims at another aspect and focuses on incorporating word clusters into the cbow model, which has not been studied before. there have also been some previous researches that utilized word clusters for reducing the number of word embeddings. @xcite used word clusters to reduce the network size for the part-of-speech tagging task. shu and nakayama (2018) attempted to compress word embeddings without losing performance by constructing the embeddings with a few basic vectors. our goal is different from the previous work in that we attempt to learn better word embeddings and do not aim at reducing the parameters when our embeddings are fine-tuned in downstream tasks. nonetheless, the reduction of the number of word embeddings from the cbow model before fine-tuning is still one of our goals as we can save space to store these embeddings and save time to download them. for example, google news vectors have around 3 million words, and we need only 2% of the number of the word embeddings if we choose 100k most frequent words and 10k word clusters in our method. ## experiments on lm and mt we applied our embeddings to downstream tasks: language modeling (lm) and low-resource machine translation (mt). when applying to the downstream tasks, we only used the training data of the specific task to obtain word clusters and embeddings without any extra data. we then used the learned embeddings to initialize the lookup table of word embeddings for the task. in this paper, we limit the applications of our model to relatively small datasets to demonstrate the usefulness of our method. we plan to conduct larger-scale experiments on more downstream tasks in future work. in the following tables, cbow and rein+reout indicate that they are initialization methods for specific downstream tasks. ## analysis in this section, we analyse rein+reout on the basis of lm experiments with en and de datasets. ## conclusion we proposed a simple and effective method to incorporate word clusters into the cbow model. our method is effective on several downstream tasks. for future work, we will test our methods on larger corpora and also add more downstream tasks. we will also study how to combine word clusters and subword information. siyu qiu, qing cui, jiang bian, bin gao, and tie-yan liu. 2014. co-learning of word representations and morpheme representations. in proceedings of col-ing 2014, the 25th international conference on computational linguistics: technical papers, pages 141-150. alan ritter, sam clark, oren etzioni, et al. 2011. named entity recognition in tweets: an experimental study. in proceedings of the conference on empirical methods in natural language processing, pages 1524-1534. association for computational linguistics. raphael shu and hideki nakayama. 2018. compressing word embeddings via deep compositional code learning. in international conference on learning representations. clara vania and adam lopez. 2017. from characters to words to in between: do we capture morphology? in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 2016-2027, vancouver, canada. association for computational linguistics. joern wuebker, stephan peitz, felix rietig, and hermann ney. 2013. improving statistical machine translation with word class models. in proceedings of the 2013 conference on empirical methods in natural language processing, pages 1377-1381. we first applied clustercat to the preprocessed corpus to obtain word clusters and then produced cluster-incorporated word embeddings with rein+reout. the results are shown in