DQ - BART : Efficient Sequence-to-Sequence Model via Joint Distillation and Quantization (2022)

Abstract: Large-scale pre-trained sequence-to-sequence models like BART and T5 achieve state-of-the-art performance on many generative NLP tasks. However, such models pose a great challenge in resource-constrained scenarios owing to their large memory requirements and high latency. To alleviate this issue, we propose to jointly distill and quantize the model, where knowledge is transferred from the full-precision teacher model to the quantized and distilled low-precision student model. Empirical analyses show that, despite the challenging nature of generative tasks, we were able to achieve a 16.5x model footprint compression ratio with little performance drop relative to the full-precision counterparts on multiple summarization and QA datasets. We further pushed the limit of compression ratio to 27.7x and presented the performance-efficiency trade-off for generative tasks using pre-trained models. To the best of our knowledge, this is the first work aiming to effectively distill and quantize sequence-to-sequence pre-trained models for language generation tasks.

## introduction pretrained sequence-to-sequence (seq2seq) models such as bart @xcite and t5 @xcite have shown great success in various natural language processing (nlp) tasks, such as text summarization @xcite @xcite , machine translation, question answering @xcite and information extraction @xcite . however, such large-scale pre-trained language models come with hundreds of millions of parameters: lewis et al. ## distilling and quantizing bart in this section, we consider two directions for reducing the size of our generative language model: quantization ( §2.1) and distillation ( §2.2). we apply distillation-aware training ( §2.3) to train a quantized and distilled low-precision model as a student model to emulate the full-precision teacher model. ## experiments and discussions in this section, we evaluate the efficacy of jointly distilling and quantizing bart (hereinafter, dq-bart) on text summarization and long-form question answering using three benchmarks: cnn/dai-lymail @xcite , xsum @xcite , and eli5 @xcite . we additionally study machine translation with mbart on wmt english-romanian (en-ro) @xcite . ## conclusion transformer-based pre-trained seq2seq language models like bart have greatly advanced the state of the art in a range of nlp tasks. yet, these extremely large-scale models pose a challenge in resource-constrained scenarios. to alleviate this issue, we proposed dq-bart, a jointly distilled and quantized bart model. empirical results show that, despite the difficult nature of language generation tasks, we achieve a 16.5x model footprint compression ratio with little performance drop on three generative benchmarks, and further present the performance-efficiency trade-off for seq2seq models up to a 27.7x compression ratio. additionally, we studied distillation and quantization for mbart on a machine translation task, and highlighted the challenge of joint low-bit quantization with distillation for deeper models on cross-lingual tasks. to the best of our knowledge, our method is the first to apply joint quantization and distillation on pretrained language models, and this is the first work aiming to effectively distill and quantize seq2seq pretrained models for language generation tasks. we hope this work could open doors for developing and applying efficient seq2seq language models. we leave additional compression methods like attention head pruning @xcite and sequence-level distillation @xcite , and the measurement of latency improvements in various settings for future work. our code is available at https://www.github.com/ amazon-research/dq-bart/ .