EELBERT : Tiny Models through Dynamic Embeddings (2023)

Abstract: We introduce EELBERT, an approach for compression of transformer-based models (e.g., BERT), with minimal impact on the accuracy of downstream tasks. This is achieved by replacing the input embedding layer of the model with dynamic, i.e. on-the-fly, embedding computations. Since the input embedding layer occupies a large portion of the model size, especially for the smaller BERT variants, replacing this layer with an embedding computation function helps us reduce the model size significantly. Empirical evaluation on the GLUE benchmark shows that our BERT variants (EELBERT) suffer minimal regression compared to the traditional BERT models. Through this approach, we are able to develop our smallest model UNO-EELBERT, which achieves a GLUE score within 4% of fully trained BERT-tiny, while being 15x smaller (1.2 MB) in size.

## introduction it has been standard practice for the past several years for natural language understanding systems to be built upon powerful pre-trained language models, such as bert @xcite , t5 @xcite , mt5 @xcite , and roberta @xcite . these language models are comprised of a series of transformer-based layers, each transforming the representation at its input into a new representation at its output. such transformers act as the "backbone" for solving several natural language tasks, like text classification, sequence labeling, and text generation, and are primarily used to map (or encode) natural language text into a multidimensional vector space representing the semantics of that language. experiments in prior work @xcite ) have demonstrated that the size of the language model (i.e., the number of parameters) has a direct impact on task performance, and that increasing a language model's size improves its language understanding capabilities. most of the recent state-of-art results in nlp tasks have been obtained with very large models. at the same time as massive language models are gaining popularity, however, there has been a parallel push to create much smaller models, which could be deployed in resource-constrained environments such as smart phones or watches. some key questions that arise when considering such environments: how does one leverage the power of such large language models on these lowpower devices? is it possible to get the benefits of large language models without the massive disk, memory and compute requirements? much recent work in the areas of model pruning @xcite , quantization @xcite , distillation @xcite and more targeted approaches like the lottery ticket hypothesis @xcite aim to produce smaller yet effective models. our work takes a different approach by reclaiming resources required for representing the model's large vocabulary. the inspiration for our work comes from @xcite , who introduced dynamic embeddings, i.e. embeddings computed on-the-fly via hash functions. we extend the usage of dynamic embeddings to transformer-based language models. we observe that 21% of the trainable parameters in bert-base @xcite are in the embedding lookup layer. by replacing this input embedding layer with embeddings computed at run-time, we can reduce model size by the same percentage. in this paper, we introduce an "embeddingless" model -eelbert -that uses a dynamic embedding computation strategy to achieve a smaller size. we conduct a set of experiments to empirically assess the quality of these "embeddingless" models along with the relative size reduction. a size reduction of up to 88% is observed in our experiments, with minimal regression in model quality, and this approach is entirely complementary to other model compression techniques. since eel-bert calculates embeddings at run-time, we do incur additional latency, which we measure in our experiments. we find that eelbert's latency increases relative to bert's as model size decreases, but could be mitigated through careful architectural and engineering optimizations. considering the gains in model compression that eelbert provides, this is not an unreasonable trade-off. ## related work there is a large body of work describing strategies for optimizing memory and performance of the bert models @xcite . in this section, we highlight the studies most revelant to our work, which focus on reducing the size of the token embeddings used to map input tokens to a real valued vector representation. we also look at past research on hash embeddings or randomized embeddings used in language applications (e.g., tito svenstrup et al. ( 2017) ). much prior work has been done to reduce the size of pre-trained static embeddings like glove and word2vec. @xcite apply principal component analysis (pca) to reduce the dimensionality of word embedding. for compressing glove embeddings, @xcite proposed laspe, which leverages matrix factorization to represent the original embeddings as a combination of basis embeddings and linear transformations. @xcite proposed a method called word2bits that uses quantization to compress word2vec embeddings. similarly, @xcite proposed using variable size code-blocks to represent each word, where the codes are learned via a feedforward network with binary constraint. however, the most relevant works to this paper are by @xcite and @xcite . the key idea in the approach by @xcite is the use of projection networks as a deterministic function to generate an embedding vector from a string of text, where this generator function replaces the embedding layer. that idea has been extended to word-level embeddings by @xcite and @xcite , using an lsh-based technique for the projection function. these papers demonstrate the effectiveness of projection embeddings, combined with a stacked layer of cnn, bilstm and crf, on a small text classification task. in our work, we investigate the potential of these protable in bert jection and hash embedding methods to achieve compression in transformer models like bert. ## modeling eelbert eelbert is designed with the goal of reducing the size (and thus the memory requirement) of the input embedding layers of bert and other transformerbased models. in this section, we first describe our observations about bert which inform our architecture choices in eelbert, and then present the eelbert model in detail. ## experimental setup in this section, we assess the effectiveness of eel-bert. the key questions that interest us are: how much model compression can we achieve and what is the impact of such compression on model quality for language understanding? we conduct experiments on a set of benchmark nlp tasks to empirically answer these questions. in each of our experiments, we compare eel-bert to the corresponding standard bert model -i.e., a model with the same configuration but with the standard trainable input embedding layer instead of our dynamic embeddings. this standard model serves as the baseline for comparison, to observe the impact of our approach. ## results we present results of experiments assessing various aspects of the model with a view towards deployment and production use. ## conclusions in this work we explored the application of dynamic embeddings to the bert model architecture, as an alternative to the standard, trainable input embedding layer. our experiments show that replacing the input embedding layer with dynamically computed embeddings is an effective method of model compression, with minimal regression on downstream tasks. dynamic embeddings appear to be particularly effective for the smaller bert variants, where the input embedding layer comprises a larger percentage of trainable parameters. we also find that for smaller bert models, a deeper model with dynamic embeddings yields better results than a shallower model of comparable size with a trainable embedding layer. since the dynamic embeddings technique used in eelbert is complementary to existing model compression techniques, we can apply it in combination with other compression methods to produce extremely tiny models. notably, our smallest model, uno-eelbert, is just 1.2 mb in size, but achieves a glue score within 4% of that of a standard fully trained model almost 15 times its size.