S queeze BERT : What can computer vision teach NLP about efficient neural networks? (2020)

Abstract: Humans read and write hundreds of billions of messages every day. Further, due to the availability of large datasets, large computing systems, and better neural network models, natural language processing (NLP) technology has made significant strides in understanding, proofreading, and organizing these messages. Thus, there is a significant opportunity to deploy NLP in myriad applications to help web users, social networks, and businesses. Toward this end, we consider smartphones and other mobile devices as crucial platforms for deploying NLP models at scale. However, today’s highly-accurate NLP neural network models such as BERT and RoBERTa are extremely computationally expensive, with BERT-base taking 1.7 seconds to classify a text snippet on a Pixel 3 smartphone. To begin to address this problem, we draw inspiration from the computer vision community, where work such as MobileNet has demonstrated that grouped convolutions (e.g. depthwise convolutions) can enable speedups without sacrificing accuracy. We demonstrate how to replace several operations in self-attention layers with grouped convolutions, and we use this technique in a novel network architecture called SqueezeBERT, which runs 4.3x faster than BERT-base on the Pixel 3 while achieving competitive accuracy on the GLUE test set. A PyTorch-based implementation of SqueezeBERT is available as part of the Hugging Face Transformers library: https://huggingface.co/squeezebert

## introduction and motivation the human race writes over 300 billion messages per day @xcite @xcite . out of these, more than half of the world's emails are read on mobile devices, and nearly half of facebook users exclusively access facebook from a mobile device @xcite . natural language processing (nlp) technology has the potential to aid these users and communities in several ways. when a person writes a message, nlp models can help with spelling and grammar checking as well as sentence completion. when content is added to a social network, nlp can facilitate content moderation before it appears in other users' news feeds. when a person consumes messages, nlp models can help classify messages into folders, compose news feeds, prioritize messages, and identify duplicates. in recent years, the development and adoption of attention neural networks have led to dramatic improvements in almost every area of nlp. in 2017, vaswani et al. proposed the multi-head self-attention module, which demonstrated superior accuracy to recurrent neural networks on english-german machine language translation @xcite . 1 these modules have since been adopted by gpt @xcite and bert @xcite for sentence classification, and by gpt-2 @xcite and ctrl @xcite for sentence completion and generation. recent works such as elec-tra @xcite and roberta @xcite have shown that larger datasets and more sophisticated training regimes can further improve the accuracy of self-attention networks. considering the enormity of the textual data created by humans on mobile devices, a natural approach is to deploy the nlp models directly onto mobile devices, embedding them in the apps used to read, write, and share text. unfortunately, highlyaccurate nlp models are computationally expensive, making mobile deployment impractical. for example, we observe that running the bert-base network on a google pixel 3 smartphone approximately 1.7 seconds to classify a single text data sample. 2 much of the research on efficient selfattention networks for nlp has just emerged in the past year. however, starting with squeezenet @xcite , the mobile computer vision (cv) community has spent the last four years optimizing neural networks for mobile devices. intuitively, it seems like there must be opportunities to apply the lessons learned from the rich literature of mobile cv research to accelerate mobile nlp. in the following, we review what has already been applied and propose two additional techniques from cv that we will leverage to accelerate nlp models. ## implementing self-attention with convolutions in this section, first, we review the basic structure of self-attention networks. next, we identify that their biggest computational bottleneck is in their position-wise fully-connected (pfc) layers. we then show that these pfc layers are equivalent to a 1d convolution with a kernel size of 1. ## incorporating grouped convolutions into self-attention now that we have shown how to implement the expensive pfc layers in self-attention networks using convolutions, we can incorporate efficient grouped convolutions into a self-attention network. grouped convolutions are defined as follows. given an input feature vector of dimensions (p, c) with p positions and c channels outputting a vector with dimensions (p, c), a 1d convolution with kernel size @xmath0, c g ) can be defined as follows. let @xmath1. this is equivalent to splitting the the input vector into g separate vectors of size (p, c in g ) along the c dimension and running g separate convolutions with independent weights each computing vectors of size (p, cout g ). the grouped convolution, however, requires only 1 g as many floating-point operations (flops) and 1 g as many weights as an ordinary convolution, not counting the small (and unchanged) amount of operations needed for the channel-wise bias term that is often included in convolutional layers. 5 finally, to complement the mathematical explanation of grouped convolutions, we illustrate the difference between traditional convolutions and grouped convolutions in figure 1 . has c @xmath2, and ffn 3 has c @xmath3. in all pfc layers of the self-attention modules, and in the ffn 2 and ffn 3 layers, we use grouped convolutions with @xmath4. to allow for mixing across channels of different groups, we use @xmath5. note that in bert-base, ffn 2 and ffn 3 each have 4 times more arithmetic operations than ffn 1 . however, when we use @xmath6, now all ffn layers have the same number of arithmetic operations. we illustrate one block of the squeezebert encoder in figure 2 . finally, the embedding size (768), the number of blocks in the encoder (12), the number of heads per self-attention module (12), the word-piece tokenizer @xcite , and other aspects of squeezebert are adopted from bert-base. aside from the convolution-based implementation and the adoption of grouped convolutions, the squeezebert architecture is identical to bert-base. finetuning data. we finetune and evaluate squeezebert (and other baselines) on the general language understanding evaluation (glue) set of tasks. this benchmark consists of a diverse set of 9 nlu tasks; thanks to the structure and breadth of these tasks (see supplementary material for detailed task-level information), glue has become the standard evaluation benchmark for nlp research. a model's performance across the glue tasks likely provides a good approximation of that model's generalizability (especially for text classification tasks). ## results we now turn our attention to comparing squeeze-bert to other efficient neural networks. ## conclusions & future work in this paper, we have studied how grouped convolutions, a popular technique in the design of efficient computer vision neural networks, can be applied to natural language processing. first, we showed that the position-wise fully-connected layers of self-attention networks can be implemented with mathematically-equivalent 1d convolutions. further, we proposed squeezebert, an efficient nlp model which implements most of the layers of its self-attention encoder with 1d grouped convolutions. this model yields an @xmath7. we also successfully applied distillation to improve our approach's accuracy to a level that is competitive with a distillation-trained mo-bilebert and with the original version of bertbase. we now discuss some possibilities for future work in the design of computationally-efficient neural networks for nlp. as we observed in section 6, in recent months numerous approaches have been proposed for reducing the computational cost of self-attention neural architectures for natural language processing. these approaches include new model structures (e.g. mobilebert), rethinking the dimensions of attention calculations (e.g. linformer), grouped convolutions (squeezebert), and much more. further, once the neural architecture has been selected, approaches such as quantization and pruning can further reduce some of the costs associated with self-attention neural network inference. the combination of all of these potential techniques opens up a broad search-space of neural architecture designs for nlp. this motivates the application of automated neural architecture search (nas) approaches such as those described in @xcite to further improve the design of neural networks for nlp.