Adaptive Compression of Word Embeddings (2020)

Abstract: Distributed representations of words have been an indispensable component for natural language processing (NLP) tasks. However, the large memory footprint of word embeddings makes it challenging to deploy NLP models to memory-constrained devices (e.g., self-driving cars, mobile devices). In this paper, we propose a novel method to adaptively compress word embeddings. We fundamentally follow a code-book approach that represents words as discrete codes such as (8, 5, 2, 4). However, unlike prior works that assign the same length of codes to all words, we adaptively assign different lengths of codes to each word by learning downstream tasks. The proposed method works in two steps. First, each word directly learns to select its code length in an end-to-end manner by applying the Gumbel-softmax tricks. After selecting the code length, each word learns discrete codes through a neural network with a binary constraint. To showcase the general applicability of the proposed method, we evaluate the performance on four different downstream tasks. Comprehensive evaluation results clearly show that our method is effective and makes the highly compressed word embeddings without hurting the task accuracy. Moreover, we show that our model assigns word to each code-book by considering the significance of tasks.

## introduction deep neural networks have greatly improved the performance in various tasks, such as image classification @xcite ), text classification @xcite , and machine translation @xcite . this break-through performance facilitates the demand to deploy such models to embedded systems (e.g., self-driving cars, mobile devices). however, the neural models typically require a large storage or memory footprint, which is a significant concern when deploying neural models to memory-constrained devices @xcite . to alleviate this limitation, several works have proposed methods that compress the neural models while minimizing loss of accuracy as much as possible @xcite @xcite . however, deploying models for natural language processing (nlp) tasks is challenging. unlike other domains, nlp models have an embedding layer which maps words and phrases to real-valued vectors. the problem is that these embeddings usually take more parameters than the remaining networks. in practice, for a neural translation model in opennmt @xcite , the word embedding parameters accout for 80% of the total parameters. therefore, it is significant to reduce the parameters of the embedding layer for deploying nlp models to memory-constrained devices. to compress word embeddings, several works proposed code-book based approaches @xcite , which represent each word as few discrete and shared codes. for example, the word dog and dogs could be represented as (3, 5, 2, 1) and (3, 5, 2, 7), respectively. this sharing scheme and discrete codes make the embeddings have smaller parameters and interpretability to some extent. however, these methods assign the same length of codes to each word without considering the significance of downstream tasks. it means that, for a sentiment analysis, excellent and the require the same amount of memory. this observation makes room for improvement in compressing word embeddings. in this paper, we attempt to further compress word embeddings by adaptively assigning different lengths of codes to each word in an end-to-end manner. we propose adacomp that adaptively learns to compress word embeddings by considering downstream tasks. the proposed compression works in two steps. first, each word in pre-trained word embeddings learns to select its code length in an end-to-end manner by applying gumbel-softmax tricks @xcite . after selecting its code length, each word learns discrete codes through a binary-constraint encoder and decoder network. to instill task-specific features to the selection process, we compress each word embedding by learning a downstream task. this allows us to learn the task-specific features naturally. compared to prior works, adacomp could give each word more options to represent their meaning since the proposed model utilizes a number of different code-books. to showcase the general applicability of ada-comp, we conduct four different nlp tasks, which are sentiment classification, chunking, natural language inference, and language modeling. comprehensive evaluation results not only show that our method could compress original word embeddings quite well without hurting task accuracy but also demonstrate that adacomp assigns each word to different code-books by considering the significance of a task. adacomp could be applied to most existing nlp systems with minor modifications since the proposed model is a network-agnostic, in-place architecture. we thus believe that existing nlp systems could benefit from our work. we organize the remainder of this paper as follows. in section 2, we discuss related work. in section 3, we describe the proposed method. we report our performance evaluation results and analyze our methodology in detail in section 4 and 5, respectively. finally, we conclude this paper in section 6. ## related work in this section, we review several studies that attempt to compress neural models, including an embedding layer. ## adaptive compression in this section, we describe the proposed method, which is denoted as adacomp, in detail. the primary strategy of adacomp is straightforward and is shown in figure 1 . we start with the pre-trained word embeddings (e.g., glove @xcite , word2vec @xcite ), and the compression method works in two steps. given an input embedding, adacomp learns to adaptively select its code length in an end-to-end manner by applying gumbel-softmax tricks @xcite (section 3.1). after selecting a code length, each word learns its discrete codes through an encoder and decoder, which has a binary latent space (section 3.2). ## conclusion in this paper, we have described adacomp that adaptively compresses word embeddings by using different lengths of code-books. to this end, we have used the gumbel-softmax tricks and the binary-constraint networks to learn the code-book selection and its discrete codes in an end-to-end manner. to showcase the general applicability of adacomp, we conduct four different nlp tasks, which are sentence classification, chunking, natural language inference, and language modeling. evaluation results have clearly shown that adacomp could obtain better results than other methods in terms of both accuracy and memory requirement. we also found that adacomp assigns each word to different code-books by considering the significance of tasks. although we have focused on compressing the embeddings by learning task-specific features, adacomp could be used at nlp tasks without fine-tuning. we believe that our method can benefit simultaneously from other compression techniques, such as pruning @xcite and low-precision representation @xcite . we leave this as an avenue for future work.