Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers (2021)

Abstract: Transformers have shown improved performance when compared to previous architectures for sequence processing such as RNNs. Despite their sizeable performance gains, as recently suggested, the model is computationally expensive to train and with a high parameter budget. In light of this, we explore parameter-sharing methods in Transformers with a specific focus on generative models. We perform an analysis of different parameter sharing/reduction methods and develop the Subformer. Our model combines sandwich-style parameter sharing, which overcomes naive cross-layer parameter sharing in generative models, and self-attentive embedding factorization (SAFE). Experiments on machine translation, abstractive summarization and language modeling show that the Subformer can outperform the Transformer even when using significantly fewer parameters.

## introduction recent improvements in nlp tasks can be attributed to the transformer @xcite model. the model has led to better deeply contextualized representations @xcite , better machine translation systems @xcite , and language models @xcite . despite their success, one main drawback of training these models is their computational cost, being a greatly limiting factor for many, with training times and memory usage ballooning as model sizes increase to attain better performance. with this in mind, there has been recent interest in making the transformer more parameter-efficient to reap its performance benefits while making the model more computationally efficient and able to scale better. many approaches have focused on automating model design with neural architecture search that aim at finding more efficient transformer variations using gradient descent @xcite @xcite . as such, these techniques are expensive, requiring a significant amount of gpu hours to find good designs. in contrast to these approaches, the model by @xcite ) has directly tackled model parameter reduction in the context of deeply contextualized word representations, still attaining similar (or better) performance. in this paper, we take a similar approach and look to explore whether these ideas can be applied to sequence-to-sequence models in a simple manner by designing the subformer. the subformer incorporates two novel techniques: (1) safe (self-attentive factorized embeddings), in which we use a small self-attention layer to reduce embedding parameter count, and (2) sandwich-style parameter sharing, in which we develop a simple and intuitive technique for parameter sharing to be effective in transformer models. we evaluate the subformer on machine translation, abstractive summarization, and language modeling, showing that our model can achieve similar or better performance compared with a base/big transformer with a ∼40% parameter reduction and minimal modification to the original architecture, reinforcing the existing over-parameterization claims @xcite @xcite . on wmt'14 en-de we achieve a bleu score of 29.3, compared to transformer-big's 28.6 with 13m fewer parameters, and we also outperform the transformer-xl model with a significant 3.6 lower perplexity and 37% fewer parameters. ## the subformer let us start by defining the notation to be used throughout the paper. we refer to the model dimension as d m , feed-forward projection dimension as d m , the vocabulary size as v , and the number of layers as l. note that, unlike standard transformer models, in which the embedding dimension is kept the same as d m , we disentangle to embedding dimension to reduce parameter count. for this reason, we denote the embedding dimension to be d e . ## experimental setup we apply our method to a variety of sequence modeling tasks: neural machine translation, summarization, and language modeling. we compare transformer-base and big @xcite with the subformer trained in the same setting. we also include simple sandwich-style parameter sharing (denoted as sandwich-{base,big}) and the usage of only safe as an ablation of what these techniques do in their naive forms when decoupled. additional implementation and training details with hyperparameter settings are in the appendix. ## results machine translation tables 3 and 4 foot_1 summarize our results on machine translation. firstly, we note that our transformer baselines outperform @xcite (base: 27.3 → 27.7, big: 28.4 → 28.6). we surmise that this is due to training for longer and with a larger batch size. the subformer outperforms our transformer baselines when trained in the same setting, with similar or fewer parameters. subformer-small reduces parameters by 40%, matching performance with our transformer baselines. subformer-base and mid, outperform our model significantly (0.4, 0.8 bleu) when using a fewer/similar number of parameters. furthermore, we note that subformer-mid performs similarly to the transformer-big model (210m params.) in for our large set of models ( language modeling results for language modeling can be seen in as seen in ## conclusion in this paper we have presented the subformer, a parameter-efficient transformer-based generative model primarily based on two simple parameter factorization/sharing techniques. our empirical re-