E dge F ormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation (2022)

Abstract: We introduce EdgeFormer – a parameter-efficient Transformer for on-device seq2seq generation under the strict computation and memory constraints. Compared with the previous parameter-efficient Transformers, EdgeFormer applies two novel principles for cost-effective parameterization, allowing it to perform better given the same parameter budget; moreover, EdgeFormer is further enhanced by layer adaptation innovation that is proposed for improving the network with shared layers.Extensive experiments show EdgeFormer can effectively outperform previous parameter-efficient Transformer baselines and achieve competitive results under both the computation and memory constraints. Given the promising results, we release EdgeLM – the pretrained version of EdgeFormer, which is the first publicly available pretrained on-device seq2seq model that can be easily fine-tuned for seq2seq tasks with strong results, facilitating on-device seq2seq generation in practice.

## introduction on-device modeling draws increasing attention for its unique advantages @xcite . on the other hand, strict resource constraints prevent many neural networks performing well in the on-device setting. in natural language processing (nlp), on-device sequence-to-sequence (seq2seq) generation remains challenging, especially for the transformer @xcite under strict resource constraints in both computation and memory. to customize the transformer for seq2seq tasks in the on-device setting, we propose edge-former -a novel parameter-efficient transformer of the encoder-decoder architecture. edge-former is structurally similar to the standard transformer with a deep encoder and shallow decoder, but with a modification that it uses an in-1 https://github.com/microsoft/unilm/tree/ master/edgelm self-a'n cross-a'n terleaved decoder with shared lightweight feedforward networks, as shown in figure 1 . the modified decoder architecture allows edgeformer to apply two novel principles that we propose for cost-effective parameterization: 1) encoder-favored parameterization that suggests we parameterize the encoder using as many parameters as possible; 2) load-balanced parameterization that suggests we balance the load of model parameters to avoid them being either underused or overused in a neural network with shared parameterization. in addition to cost-effective parameterization, edgeformer proposes and applies layer adaptation to further improve the model with tied layers, as figure 2 shows. inspired by parameterefficient task transfer, we investigate 3 efficient layer adaptation approaches for improving the performance with negligible cost. we show edge-former (with fewer than 10 million model parameters) largely outperforms the strong universal transformer baselines in the on-device setting with competitive results, and the int8-quantized edgeformer can perform high-quality on-device seq2seq generation within around 100ms latency (20-30 sequence length on average) using two midto-high-end cpu cores and less than 50mb ram. the contributions of this work are three-fold: ## constraints for on-device seq2seq computation on-device computer vision (cv) models tend to use 1g flops (0.5g macs) as a constraint, which is directly followed by previous work on on-device translation @xcite . in our work, however, we propose to relax the flops constraint for typical seq2seq tasks to 2g flops (1g macs) because the latency requirement for on-device seq2seq generation is not so rigid as cv tasks and it is uncommon for an on-device seq2seq model to handle too many concurrent requests in practice. the relaxed constraint allows better prediction quality that strongly correlates with user experience for seq2seq tasks, but still ensure the cpu on edge devices to process tens of sentences per second, which is more than sufficient for an ondevice seq2seq model. in addition to flops that is a theoretical hardware-independent measurement for computational cost, we also require the runtime latency for an input sentence (typically 20 ∼ 30 tokens on average) to be within around 100ms using two mid-to-high-end cpu cores. memory in contrast to deploying a model on a cloud server without caring about memory cost much, there is a very strict memory constraint for an on-device model in practice, because a user's edge device (e.g., pc) is not only for model hosting; instead, it usually runs many other (background) apps and programs at the same time besides the model. to ensure moderate memory cost, we limit the number of model parameters (excluding word embedding lookup table) up to 10 million, following previous work @xcite , and require the runtime memory footprint to be less than 50mb. ## edgelm -the pretrained edgeformer given the promising results, we introduce edgelm -the pretrained foot_4 edgeformer (adapter-la) with 8k sentenpiece vocabulary with factorized embedding (d @xmath0.e., masked span infilling) as t5 @xcite and make it publicly available for downstream on-device seq2seq task fine-tuning. we evaluate edgelm in the benchmarks of three popular seq2seq tasks: conll-14 for grammatical error correction (gec), xsum @xcite for abstractive summarization, and squad-nqg @xcite for question generation. according to ## related work on-device seq2seq generation in nlp is a research area that has been less explored than on-device cv and nlu @xcite . besides the general techniques like pruning, compression, quantization and knowledge distillation @xcite @xcite that are orthogonal to our effort, parameter-efficient transformerbased seq2seq modeling is the most related research branch to ours. in this branch, universal transformer @xcite uses crosslayer sharing method, which is the most popular solution to parameter efficiency. @xcite extends universal transformer by studying different ways for layer sharing, and @xcite proposes to free the first and last encoder layer and widen the intermediate layers for better performance. however, both the approaches consider parameter-efficiency only without caring about latency becoming worse. in addition to work improving parameter efficiency by weight sharing, there is research that studies lightweight model architecture for seq2seq learning where early work @xcite mainly focuses on cnns, while recent efforts have tended to switch to attentionbased models such as @xcite . also, low-rank factorization has been studied intensively to make the model tiny @xcite ; and hardware-aware network architecture search with elastic modeling @xcite has been proposed recently for facilitating deployment of seq2seq models on various devices. among previous studies, the work of @xcite is the most related to ours, which studies seq2seq generation in an on-device setting. however, it sets the computational constraint for ondevice seq2seq to be the same with the cv tasks, which is too strict and unnecessary, as discussed in section 3. as a result, their models focus on flops optimization much more than memory, leading to an undesirable tradeoff between the quality and model size for the practical on-device seq2seq setting which should care about memory much more than latency. in contrast, our work carefully evaluates bottleneck constraints, and proposes appropriate models with parameterization and layer adaptation innovations, largely improving the results for practical on-device seq2seq generation. ## conclusion and future work we formally study on-device seq2seq generation, including defining its practical resource constraint setting and proposing an appropriate modeling technology edgeformer. the cost-effective parameterization and layer adaptation innovations in edgeformer both prove effective to improve the results with negligible computation and memory cost, achieving state-of-the-art results in the on-device seq2seq generation setting. our released pretrained edgeformer -edgelm can be easily fine-tuned for downstream seq2seq tasks, largely facilitating on-device seq2seq generation in practice. for future work, we plan to further study loadbalanced parameterization for parameter-efficient models, which is an interesting and new but seemingly profound machine learning research problem: instead of naively assuming that all the parameters are equal in this preliminary study, we suspect that parameters in different modules (e.g., parameters in the self-attn and ffn; or parameters in different layers) should be under different amounts of load. we look forward to in-depth research on this problem, which might be helpful to deepen our understanding of neural networks.