Current state-of-the-art approaches to summarization utilize large pre-trained Transformer models. Distilling these models to smaller student models has become critically important for practical use; however there are many different distillation methods in the NLP literature. Recent work on distilling BERT for classification and regression tasks shows strong performance using standard knowledge distillation. Alternatively, machine translation practitioners , have primarily distilled using pseudo labeling , where a small model is trained on the translations of a larger model. A third  approach is to 'shrink and fine-tune' (SFT), which avoids any explicit distillation by transferring parameters to a  student model and then fine-tuning. This work considers distillation of BART and Pegasus , two state of the art  summarization models, on two datasets across a variety of student models. We produce high quality, fast checkpoints across different computational budgets, and learn some patterns about which distillation techniques perform well in which situations. PyTorch code to rerun our methods, and use the distilled BART and Pegasus checkpoints is available in Hugging Face transformers .
Recent state-of-the-art approaches to summarization utilize large pre-trained Transformer models. Distilling these models to smaller student models has become critically important for practical use; however there are many different distillation methods proposed by the NLP literature. Recent work on distilling BERT for classification and regression tasks shows strong performance using direct knowledge distillation. Alternatively, machine translation practitioners distill using pseudo-labeling , where a small model is trained on the translations of a larger model. A third , simpler approach is to 'shrink and fine-tune' (SFT), which avoids any explicit distillation by copying parameters to a smaller student model and then fine-tuning. We compare these three approaches for distillation of Pegasus and BART, the current and former state of the art , pre-trained summarization models, and find that SFT outperforms knowledge distillation and pseudo-labeling on the CNN/DailyMail dataset, but under-performs pseudo-labeling on the more abstractive XSUM dataset. PyTorch Code and checkpoints of different sizes are available through Hugging Face transformers here URL
R
Current
Recent
R
in
proposed by
R
standard
direct
R
, have primarily distilled using pseudo labeling
distill using pseudo-labeling
A
, simpler
R
transferring
copying
A
smaller
R
This work considers distillation of BART and Pegasus , two
We compare these three approaches for distillation of Pegasus and BART, the current and former
A
, pre-trained
R
on two datasets across a variety of student models. We produce high quality, fast checkpoints across different computational budgets, and learn some patterns about which distillation techniques perform well in which situations. PyTorch code to rerun our methods, and use the distilled BART and Pegasus checkpoints is available in
and find that SFT outperforms knowledge distillation and pseudo-labeling on the CNN/DailyMail dataset, but under-performs pseudo-labeling on the more abstractive XSUM dataset. PyTorch Code and checkpoints of different sizes are available through
R
.
here URL