P ro F ormer: Towards On-Device LSH Projection Based Transformers (2021)

Abstract: At the heart of text based neural models lay word representations, which are powerful but occupy a lot of memory making it challenging to deploy to devices with memory constraints such as mobile phones, watches and IoT. To surmount these challenges, we introduce ProFormer – a projection based transformer architecture that is faster and lighter making it suitable to deploy to memory constraint devices and preserve user privacy. We use LSH projection layer to dynamically generate word representations on-the-fly without embedding lookup tables leading to significant memory footprint reduction from O(V.d) to O(T), where V is the vocabulary size, d is the embedding dimension size and T is the dimension of the LSH projection representation. We also propose a local projection attention (LPA) layer, which uses self-attention to transform the input sequence of N LSH word projections into a sequence of N/K representations reducing the computations quadratically by O(Kˆ2). We evaluate ProFormer on multiple text classification tasks and observed improvements over prior state-of-the-art on-device approaches for short text classification and comparable performance for long text classification tasks. ProFormer is also competitive with other popular but highly resource-intensive approaches like BERT and even outperforms small-sized BERT variants with significant resource savings – reduces the embedding memory footprint from 92.16 MB to 1.7 KB and requires 16x less computation overhead, which is very impressive making it the fastest and smallest on-device model.

## introduction transformers @xcite based architectures like bert @xcite , xl-net * work done during internship at google † work done while at google ai @xcite , gpt-2 @xcite , mt-dnn @xcite , roberta @xcite reached state-of-the-art performance on tasks like machine translation @xcite , language modelling @xcite ), text classification benchmarks like glue @xcite . however, these models require huge amount of memory and need high computational requirements making it hard to deploy to small memory constraint devices such as mobile phones, watches and iot. recently, there have been interests in making bert lighter and faster @xcite . in parallel, recent on-device works like sgnn @xcite , sgnn++ @xcite and @xcite produce lightweight models with extremely low memory footprint. they employ a modified form of lsh projection to dynamically generate a fixed binary projection representation, p(x) ∈ [0, 1] t for the input text x using word or character n-grams and skip-grams features, and a 2-layer mlp + softmax layer for classification. as shown in @xcite these models are suitable for short sentence lengths as they compute t bit lsh projection vector to represent the entire sentence. however, @xcite showed that such models cannot handle long text due to significant information loss in the projection operation. on another side, recurrent architectures represent long sentences well, but the sequential nature of the computations increases latency requirements and makes it difficult to launch on-device. recently, self-attention based architectures like bert @xcite have demonstrated remarkable success in capturing long term dependencies in the input text via purely attention mechanisms. bert's model architecture is a multi-layer bidirectional transformer encoder based on the original implementation in @xcite . the self-attention scores can be computed in parallel as they do not have recurrent mechanisms. but usually these architectures are very deep and the amount of computation is quadratic in the order of o(l • n 2 ), where l is the number of layers (transformer blocks) and n is the input sentence length. straightforward solutions like reducing the number of layers is insufficient to launch transformers on-device due to the large memory and quadratic computation requirements. in this paper, we introduce a projection-based neural architecture proformer that is designed to (a) be efficient and learn compact neural representations (b) handle out of vocabulary words and misspellings (c) drastically reduce embedding memory footprint from hundreds of megabytes to few kilobytes and (d) reduce the computation overhead quadratically by introducing a local attention layer which reduces the intermediate sequence length by a constant factor, k. we achieve this by bringing the best of both worlds by combining lsh projection based representations (for low memory footprint) and self-attention based architectures (to model dependencies in long sentences). to tackle computation overheard in the transformer based models, we reduce the number of self-attention layers and additionally introduce an intermediate local projection attention (lpa) to quadratically reduce the number of self-attention operations. the main contributions of our paper are: ## proformer: lsh projection based transformers in this section, we show the overall architecture of proformer in figure 1 . proformer consists of multiple parts: (1) word-level locality sensitive hashing (lsh) projection layer, (2) local projection attention (lpa) layer, (3) transformer layer @xcite and (4) a max-pooling + classifier layer. next, we describe each layer in detail. ## datasets & experimental setup in this section, we describe our datasets and experimental setup. we use text classification datasets from state-of-the-art on-device evaluations such as: mrda @xcite and atis @xcite , ag news @xcite and yahoo! answers @xcite . we train proformer on multiple classification tasks individually and report accuracy on corresponding test sets. we fix the projection size, @xmath0, n-gram @xmath1, skip-gram @xmath2, p. for the lpa layer, we experiment with two values for @xmath3, 4, where @xmath4. for the transformer layer, we fix the number of layers, @xmath5, @xmath6. 2 we compare our model with previous state of the art neural architectures, including on-device approaches. we also fine-tune the pretrained 12layer bert-base model @xcite on all classification tasks and compare to our model. bert-base consists 12-layers of transformer blocks @xcite and is pretrained in an unsupervised manner on a large corpus (bookscorpus @xcite and english wikipedia) using masked-language model objective. we fine-tune the pretrained bert-base @xcite to each of the classification tasks. for training, we use adam with learning rate of 1e-4, β 1 =0.9, β 2 =0.999, l2 weight decay of 0.01, learning rate warmup over the first 10, 000 steps, and linear decay of the learning rate. we use dropout probability of 0.1 on all layers and training batch size of 256. for further comparison, we also trained much smaller bert baselines with 2-layers of transformer blocks and smaller input embedding sizes. ## results tables 3 and 4 show the results on the atis & mrda short text classification and ag & y!a long text classification tasks. we compare our approach, proformer against prior state-of-the-art on-device works, fine-tuned bert-base, smaller 2-layer bert variants and other non-on-device neural approaches. overall, our model proformer improved upon non-on-device neural models while keeping very small memory footprint and high accuracy. this is very impressive since proformer can be directly deployed to memory constraint devices like phones, watches and iot while still maintaining high accuracy. proformer also improved upon prior on-device state-of-the-art neural approaches like sgnn @xcite and sgnn++ @xcite reaching over 35% improvement on long text classification. similarly it improved over on-device proseqo @xcite ) models for all datasets and reached comparable performance on mrda. in addition to the quality improvements, proformer also keeps smaller memory footprint than proseqo, sgnn and sgnn++. in addition to the non-on-device and on-device neural comparisons, we also compare against bert-base and other smaller variants. our experiments show that proformer outperforms the bert conf ig.json in bert-base model @xcite small bert baselines on all tasks. moreover, although the 12-layer fine-tuned bert-base @xcite model converged to the state-of-the-art in almost all of the tasks, proformer converges to ≈ 97.2% bert-base's performance on an average while occupying only 13% of bert-base's memory. proformer has 14.4 million parameters, while bert-base has 110 million. for fair comparison, we also test proformer with @xmath7, which only occupies 38.4% the memory footprint of 2-layer bert-base model and reduces the computation overhead by 16 times. the embedding look up table occupies nearly 23 million parameters out of 38 million parameters in the 2-layer bert model. we notice that @xmath8. overall, our experiments demonstrate that proformer reaches better performances that prior non-on-device and on-device neural approaches, and comparable performance to bert-base models while preserving smaller memory footprint. ## conclusion we proposed a novel on-device neural network pro-former, which combines lsh projection based text representations, with trans-former architecture and locally projected self-attention mechanism that captures long range sentence dependencies. overall, proformer yields low memory footprint and reduces computations quadratically. in series of experimental evaluations on short and long text classifications we show that proformer improved upon prior neural models and on-device work like sgnn @xcite , sgnn++ @xcite and proseqo @xcite . proformer reached comparable performance to our bert-base implementation, however it produced magnitudes more compact models than bert-base. this is very impressive showing both effectiveness and compactness of our neural model.