Do Transformer Modifications Transfer Across Implementations and Applications? (2021)

Abstract: The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread adoption. In this paper, we comprehensively evaluate many of these modifications in a shared experimental setting that covers most of the common uses of the Transformer in natural language processing. Surprisingly, we find that most modifications do not meaningfully improve performance. Furthermore, most of the Transformer variants we found beneficial were either developed in the same codebase that we used or are relatively minor changes. We conjecture that performance improvements may strongly depend on implementation details and correspondingly make some recommendations for improving the generality of experimental results.

## introduction much of the empirical success of deep learning can be attributed to advances in methods for building and training neural networks. these advances include improved optimizers @xcite @xcite , regularization schemes @xcite @xcite , and model architectures @xcite @xcite . an aspiration underlying much of this work is that an improvement to a particular machine learning pipeline will yield equal-or-better performance on any task that the pipeline is applicable to. for example, residual connections in convolutional networks @xcite are designed to ideally improve * correspondence to sharannarang@google.com † work completed while at google performance on any task where these models are applicable (image classification, semantic segmentation, etc.). in practice, when proposing a new improvement, it is impossible to test it on every applicable downstream task, so researchers must select a few representative tasks to evaluate it on. however, the proposals that are ultimately adopted by the research community and practitioners tend to be those that reliably improve performance across a wide variety of tasks "in the wild". the transformer architecture @xcite is an example of a seminal improvement in the field of deep learning. currently, the transformer is the de facto architecture of choice for processing sequential data and is starting to be applied to vision applications (e.g. @xcite ). since being introduced three years ago, many modifications to the transformer architecture have been proposed. however, the most widely-used applications of the transformer architecture (e.g. @xcite ; @xcite ; @xcite ; @xcite ) incorporate few of these modifications. instead, the standard practice is to use a slightly-modified version of the originally-proposed transformer. one possible explanation for this is that the originally-proposed transformer architecture was near-perfect, and there wasn't much that could be done to improve it. this is in contrast to, for example, convolutional neural networks, which have continually evolved over the past few decades (e.g. the replacement of pooling with striding @xcite , fullyconnected layers with convolutional layers @xcite , the addition of normalization @xcite and residual connections @xcite , etc.). another possible explanation is that the modifications proposed to the transformer do not "generalize" across applications, i.e. the modifications only help on the limited experimental setting considered when the modification was proposed, and/or rely on specific details that are not common across implementations of the transformer. the main goal of this paper is to try to determine why most modifications proposed to the transformer have not seen widespread adoption. to answer this question, we reimplemented and evaluated a wide variety of transformer variants on a suite of tasks that transformers are commonly applied to. our main finding is that many transformer modifications do not result in improved performance in our experimental setting. moreover, those variants that did yield better performance tended to be those that were quite small changes and/or were developed in the codebase where we carried out our evaluation. this suggests to us the possibility that transformer modifications exhibit a surprising lack of generalization across different implementations and tasks. ## modifications in this section, we enumerate all of the architectural modifications we consider. for a description of the transformer architecture, refer to the appendix d. due to space constraints, we are seldom able to thoroughly define each specific modification. moreover, we limit our study to the encoderdecoder architecture. please refer to the original sources for each modification for additional details. ## experiments in order to study the impact of each of the modifications described in section 2, we conduct a systematic study by comparing a baseline model to each modification while holding the task, hyperparameters, optimizer, and either the parameter count or flop budget (floating point operations per second) constant. we use the original transformer model as our baseline model with two modifications: first, we apply layer normalization before the selfattention and feedforward blocks instead of after. this small change has been unanimously adopted by all current transformer implementations because it leads to more effective training @xcite . secondly, we use relative attention with shared biases (as used in @xcite ) instead of sinusoidal positional embeddings, which makes it easier to train the model. our baseline model is a standard encoder-decoder with 12 layers in the encoder and decoder. the feedforward network in each layer consists of a dense layer with dimension of d @xmath0. all attention mechanisms have 12 heads and "key" and "value" matrices have a dimension of d @xmath1. all other sublayers have a dimension of d @xmath2. we refer to this model as the "vanilla transformer". we consider two experimental settings for evaluating the performance of each modification: transfer learning based on the t5 @xcite and supervised machine translation on the wmt'14 english-german translation. for transfer learning, we copy the methodology used by the t5 model, proposed in @xcite . for full details of this experimental setup, please refer to @xcite . we pre-train encoder-decoder models in a selfsupervised manner using the "span corruption" masked language modeling objective @xcite @xcite on the c4 dataset. we run experiments on version 2.3.1 of the c4 dataset available in tensorflow datasets foot_0 . we pre-train each architecture variant for 524, 288 steps with batches of 65, 536 tokens. as in t5, we use adafactor @xcite for optimization and an inverse square root learning rate schedule during pretraining. we use a maximum sequence length of 512 for both the inputs and targets during pre-training. to evaluate the performance of pre-trained models, we compute the perplexity on a held-out portion of the c4 dataset for each pre-trained model, with the expectation that improvements in perplexity will correlate with performance on fine-tuned tasks. to capture the inter-run variance on these models, we run each model 5 times for 65, 536 steps ( 1 8 th of the total pre-training steps). we report the mean and standard deviation of the loss (log perplexity) on held-out data of these five experiments and also report the final loss at the end of pre-training (524, 288 steps). we do not use any regularization during pre-training. in the transfer learning setting, after pretraining we fine-tune each model on three different tasks: the superglue @xcite natural language understanding metabenchmark, the xsum @xcite abstractive summarization dataset, and the closed-book variant @xcite of the webquestions @xcite questionanswering task. with these tasks, we hope to capture a broad variety of nlp problems including language understanding and classification, language generation, and knowledge internalization. for superglue and xsum, each model is fine-tuned for 262,144 steps. since the webquestions dataset is much smaller, we fine-tune the model for only 30,000 steps. we use a constant learning rate of 0.0005 with a linear warm-up of 20, 000 steps. similar to pre-training, each batch contains 65, 536 tokens. we save a checkpoint every 2, 500 steps (1, 000 steps for webquestions) and report results on the model checkpoint corresponding to the highest validation performance. we use a dropout of 0.1 during fine-tuning for all the tasks. all results are reported on the validation split of each dataset. for superglue, we report the average score across all tasks in the benchmark. we report rouge-2 @xcite for xsum and accuracy for webquestions. for supervised training on the wmt'14 english to german translation task @xcite , we use the same model and batch size as for the transfer learning setting. we train for a total of 150,000 steps. we use the same data splits as were used in @xcite and report the bleu score of the highest-scoring checkpoint on the validation set. we use a vocabulary of 37,000 tokens learned by byte pair encoding @xcite for supervised training as opposed to 32,000 tokens (created using sentencepiece @xcite ) for the transfer learning experiments. to compare the efficiency of the model, we also report the total number of parameters, the total number of floating point operations, and the measured steps per second in the pretraining experiments. reporting these parameters can help us understand the trade-off between quality and efficiency. for each architectural modification, we attempt to keep either the parameter count or total operations in the model approximately the same to perform a fair comparison with the baseline model. all hyperparameters are held constant for each architectural variant across pre-training and fine-tuning. however, we found that certain architectural (rezero and fixup) variants achieved significantly lower negative log perplexity than the baseline model with the adafactor optimizer. therefore, we use the adam optimizer (kingma and ba, 2014) for these variants. for pre-training, we use an inverse square root learning rate schedule with a linear warm-up of 4, 000 steps. for fine-tuning, we use a constant learning rate of 5e -5 with a linear warm-up of 20, 000 steps. we provide details of certain modifications in appendix b. all experiments are run using the t5 library foot_1 on "slices" of cloud tpu pods. all model variants are implemented in the mesh tensor-flow library @xcite . ## conjectures and recommendations as discussed above, we were surprised to find that so few of the architectural modifications produced improvements in the settings we considered. this largely contrasts the experiments included in the original papers that proposed each modification. we broadly grouped the modifications that actually did improve performance as either 1) being relatively simple (e.g. a change in activation function), 2) being developed in the same codebase where we ran experiments (e.g. the synthesizer variants @xcite ), or 3) incurring an increase in parameter count or flops (e.g. the switch transformer @xcite or universal transformer @xcite ). other modifications that don't fit into one of these categories generally didn't improve performance. there are various possible explanations as to why our results bore out the way they did: 1. the mesh tensorflow codebase and implementation are just so different than standard practice that most architectural modifications do not work. we believe this is unlikely due to the fact that the mesh tensorflow transformer implementation was created by one of the coauthors of the original transformer paper and has been used to attain state-of-the-art results (e.g. raffel et al. (2019); roberts et al. (2020); khashabi et al. (2020); kale (2020); nogueira et al. (2020); narang et al. (2020); xue et al. (2020); fedus et al. (2021), etc.). 2. the tasks we consider are non-standard or do not match the set of tasks used to vet the modifications in the first place. the transformer model is used for a variety of nlp problems including classification and generation tasks. we included transfer learning experiments on superglue, xsum, and webquestions and supervised training on wmt'14 ende, which covers the majority of use-cases.