L ight F ormer: Light-weight Transformer Using SVD -based Weight Transfer and Parameter Sharing (2023)

Abstract: Transformer has become an important technique for natural language processing tasks with great success. However, it usually requires huge storage space and computational cost, making it difficult to be deployed on resource-constrained edge devices. To compress and accelerate Transformer, we propose LightFormer, which adopts a low-rank factorization initialized by SVD-based weight transfer and parameter sharing. The SVD-based weight transfer can effectively utilize the well-trained Transformer parameter knowledge to speed up the model convergence, and effectively alleviate the low-rank bottleneck problem combined with parameter sharing. We validate our method on machine translation, text summarization and text classification tasks. Experiments show that on IWSLT’14 De-En and WMT’14 En-De, LightFormer achieves similar performance to the baseline Transformer with 3.8 times and 1.8 times fewer parameters, and achieves 2.3 times speedup and 1.5 times speedup respectively, generally outperforming recent light-weight Transformers.

## introduction transformer @xcite has been widely used and achieved state-of-the-art results in natural language processing tasks such as machine translation, text summarization, and text classification. to improve the performance of networks, we usually stack deeper transformer blocks, or increase the dimension of the hidden layers @xcite . however, this will bring a huge amount of parameters and computations to the model, exceeding the capabilities of many edge devices. common methods of model compression mainly include pruning @xcite , quantization @xcite , knowledge distillation @xcite , low-rank factorization @xcite , and weight sharing @xcite . ## methodology in this section, we first introduce the motivation of our method. secondly, we introduce svd-based weight transfer with parameter sharing. finally, we introduce the framework of our lightformer. ## conclusion in this paper, we propose a lightweight transformer (lightformer) based on svd weight transfer and parameter sharing, which can guarantee the performance of the model but with fewer operations and parameters. compared with the randomly initialized low-rank matrix factorization, our method can effectively alleviate the low-rank bottleneck problem and speed up the convergence of the model. we validate our method on three machine translation tasks. experimental results show that our method is consistently better than recent light-weight transformers.