Block-wise Word Embedding Compression Revisited: Better Weighting and Structuring (2021)

Abstract: Word embedding is essential for neural network models for various natural language processing tasks. Since the word embedding usually has a considerable size, in order to deploy a neural network model having it on edge devices, it should be effectively compressed. There was a study for proposing a block-wise low-rank approximation method for word embedding, called GroupReduce. Even if their structure is effective, the properties behind the concept of the block-wise word embedding compression were not sufficiently explored. Motivated by this, we improve GroupReduce in terms of word weighting and structuring. For word weighting, we propose a simple yet effective method inspired by the term frequency-inverse document frequency method and a novel differentiable method. Based on them, we construct a discriminative word embedding compression algorithm. In the experiments, we demonstrate that the proposed algorithm more effectively finds word weights than competitors in most cases. In addition, we show that the proposed algorithm can act like a framework through successful cooperation with quantization.

## introduction deep neural networks have had lots of attention due to their great success in many applications. recently, deep learning is being actively applied to edge devices like a smartphone with important reasons including data privacy and low latency. however, deep neural networks usually have a tremendous number of parameters, so that one does not simply deploy them on such devices having limited resources. in order to resolve this issue, there is a line of research compressing neural networks. existing works for neural network compression mainly focus on convolutional layers and fully connected layers. in addition to those layers, there is a special and important layer called word embedding the major contribution of this work is to propose two effective word weighting methods for blockwise word embedding compression and to exploit a non-uniform partitioning method for lightweight embedding structure. based on them, we construct a discriminative block-wise word embedding compression algorithm (discblock) which significantly outperforms groupreduce. in addition, we show that it can be cooperated with another compression technique like quantization as a compression framework. outline. in this work, we first introduce a block-wise word embedding structure inspired by groupreduce of @xcite . next, we discuss better word weighting and clustering to build the structure. after that, we conduct extensive experiments to demonstrate the effectiveness of discblock with various downstream tasks such as language modeling, machine translation, text classification, and question and answering. since word embedding is generally represented by a matrix, decomposition-based compression techniques and efficient embedding structures were proposed. @xcite proposed the blockwise low-rank approximation method for word embedding. @xcite devised a way of interpreting an embedding matrix into a 3-dimensional tensor and proposed an embedding structure by decomposing it with tensor-train decomposition. @xcite proposed a smallsize word embedding structure inspired by quantum entanglement. (lioutas, 2020) proposed a re-cent study for word embedding factorization based on distillation. as (lioutas, 2020) conducted experiments for combining their approach with groupreduce, it can be also applied to our algorithm. there are lots of existing approaches for word embedding compression, but none of existing approaches deeply study word weights in sense of compressing word embedding. decomposition. since this work is based on lowrank approximation, we also study decompositionbased model compression approaches. @xcite proposed a low-rank tucker decomposition on kernel tensors. @xcite proposed a framework unifying low-rank approximation and pruning of kernel tensors, which assumes that kernels are likely to be low-rank and sparse. @xcite proposed a canonical polyadic decomposition-based compression method for approximating a convolutional layer. @xcite proposed a variation of the transformer of @xcite by decomposing multi-linear attention with block-term tensor decomposition @xcite . note that the transformer of @xcite ) contains word embedding, but it is not compressed in their work. ## proposed algorithms discblock uses the same block-wise word embedding structure of @xcite . in addition, given word weights, we assign a rank to each group in the same way of @xcite . the difference between discblock and groupreduce are made on word weighting and clustering, which are explained in this section.