Leveraging Pre-trained Checkpoints for Sequence Generation Tasks (2020)

Abstract: Unsupervised pre-training of large neural models has recently revolutionized Natural Language Processing. By warm-starting from the publicly released checkpoints, NLP practitioners have pushed the state-of-the-art on multiple benchmarks while saving significant amounts of compute time. So far the focus has been mainly on the Natural Language Understanding tasks. In this paper, we demonstrate the efficacy of pre-trained checkpoints for Sequence Generation. We developed a Transformer-based sequence-to-sequence model that is compatible with publicly available pre-trained BERT, GPT-2, and RoBERTa checkpoints and conducted an extensive empirical study on the utility of initializing our model, both encoder and decoder, with these checkpoints. Our models result in new state-of-the-art results on Machine Translation, Text Summarization, Sentence Splitting, and Sentence Fusion.

## introduction unsupervised and self-supervised pre-training methods, such as elmo @xcite , ulmfit @xcite , and more recently bert @xcite , gpt and gpt-2 @xcite , xlnet @xcite , and roberta @xcite have established a qualitatively new level of baseline performance for many widely used natural language understanding (nlu) benchmarks including some of the most popular, like glue @xcite and squad @xcite . the most appealing part about this massive shift towards using large architectures pre-trained on large collections of texts is that the pretrained checkpoints along with the inference code are made freely available. this saves hundreds of tpu/gpu hours, as warm-starting a model from a pre-trained checkpoint typically requires orders of magnitude fewer fine-tuning steps while delivering significant performance boosts. more importantly, the ability to bootstrap from a state-of-the-art performing model such as bert @xcite motivates the community to greatly speed up the progress towards developing better and easily reusable nlu systems. while we continue to observe an increasing number of papers building on top of bert and/or gpt models reporting encouraging improvements on glue, squad, and other similar benchmarks, very little attention has been paid to using these pre-trained models to warm-start sequence-tosequence (seq2seq) models. it has been argued that the pre-training objective used by bert is not well-suited for tasks that require decoding texts, for example, conditional text generation in machine translation and summarization @xcite . nevertheless, it remains unclear to what extent using such large models pre-trained on large collections of text can be beneficial to warm-start seq2seq generation models. in this paper, we report on a transformer-based seq2seq model that is compatible with publicly available pre-trained bert, gpt-2, and roberta checkpoints. we aim to provide an empirical answer to the following research question: what is the best way to leverage publicly available pretrained checkpoints for warm-starting sequence generation models? for example, one could imagine using a bert checkpoint to initialize the encoder for better input understanding and choosing gpt-2 model as the decoder for better text generation. one of the main contributions of this paper is that we rigorously experiment with a large number of different settings to combine bert, gpt, and roberta pre-trained checkpoints to initialize our transformer-based model. we report results on three canonical conditional text generation tasks of increasing complexity: sentence-level fusion @xcite and splitting @xcite , wmt14 en↔de machine translation using most common eval sets: newstest2014 and newstest2016, and abstractive summarization using three datasets: gigaword @xcite , @xcite , and bbc extreme @xcite . our models report significant improvements over randomly initialized models, demonstrating the benefit of leveraging unsupervised pre-trained models. more importantly, this simple strategy results in new state-of-the-art results on machine translation, text summarization, sentence splitting, and sentence fusion. our results also demonstrate that a pre-trained encoder is an essential component for sequence generation tasks and often these tasks benefit from sharing the weights between the encoder and the decoder. overall, we have run over 300 experiments spending thousands of tpu v3 hours to better accommodate the language modeling and understanding capabilities of these pre-trained models for text generation. we believe that nlp researchers and practitioners will derive actionable insights from our findings when tackling various seq2seq tasks. the code to query our models and predictions on various benchmarks will be available at https:// github.com/google-research/googleresearch/tree/master/bertseq2seq. if not stated otherwise, the implementation of the decoder layers are also identical to the bert implementation with two adjustments. first, the self-attention mechanism is masked to look only at the left context. secondly, we add an encoderdecoder attention mechanism. note, that if the model was randomly initialized, we found no difference between a bert compatible decoder and a gpt-2 compatible decoder. most of the models use the base checkpoint and therefore have 12 layers, a hidden size of 768, filter size of 3,072, and 12 attention heads. we chose the best-performing model and also collect numbers using larger pre-trained checkpoints. these models have 24 layers, a hidden size of 1,024, filter size of 4,096, and 16 attention heads. all models were fine-tuned on the target task using adam with a learning rate of 0.05. we used a linear learning rate warmup with 40k steps, normalization by the square root of the hidden size, and a square root decay. we did not perform any tuning of these hyperparameters (except for §5). the batch size and the number of training steps will be reported for each task individually. bert checkpoints. we tokenize our text using the wordpiece @xcite to match the bert pre-trained vocabulary. depending on the experiment, we use one of the following publicly available checkpoints: bert-base cased, bert-base uncased, bert-base multilingual cased @xcite . 1 the first two checkpoints have a vocabulary size of around ∼30k wordpieces, whereas the multilingual checkpoint has a much larger vocabulary size of ∼110k. bert also trains positional embeddings for up to 512 positions, which is the maximum input and output length in all experiments. gpt-2 checkpoints. we tokenize our text using the sentencepieces @xcite to match the gpt-2 pre-trained vocabulary. 2 note that, although the available checkpoint is frequently called 117m, which suggests the same number of parameters, we count 125m parameters in the checkpoint. this is the smallest architecture they trained, and the number of layers, hidden size, and filter size are comparable to bert-base. the model was trained mainly on english data but does contain some foreign language. the vocabulary size is ∼50k. while gpt-2 has positional embeddings for up to 1,024 positions, we only use the first 512 to make the results comparable with bert. roberta checkpoints. roberta @xcite is trained using pytorch, but we found that the learned parameters are fully compatible total embed. with the existing tensorflow bert architectures with some minor adjustments. 3 the vocabulary treatment in roberta is compatible with the sentencepiece tokenization in gpt-2. 4 as the conceptual differences between bert and roberta are minor, we might use bert as a hypernym to address both pretraining methods in this paper. ## investigated model variants in this section, we describe several combinations of model initialization. the number of total trainable parameters, the number of embedding parameters, and the number of parameters initialized from the checkpoint vs. randomly are shown in rnd2rnd a transformer encoder-decoder architecture with all weights initialized randomly. bert2rnd a bert-initialized encoder paired with a randomly initialized decoder. encoder and decoder share the embedding matrix initialized from a checkpoint. rnd2bert a randomly initialized encoder paired with a bert-initialized decoder. to perform autoregressive decoding, we mask the bidirectional self-attention mechanism of bert to look only at the left context. bert2bert a bert-initialized encoder paired with a bert-initialized decoder. all weights are initialized from a public bert checkpoint. the only variable that is initialized randomly is the encoder-decoder attention. bertshare like bert2bert, but the parameters between encoder and decoder are shared. this greatly reduces the memory footprint of the model (136m vs. 221m parameters). additionally, we experimented with a layer-wise attention mechanism @xcite , but obtained nearly identical numbers on most tasks. robertashare same as bertshare, but the shared encoder and decoder are initialized with the public roberta checkpoint. gpt a decoder-only architecture. we treat the input as a conditioning prefix of a language model. the decoder is warm-started with a public gpt-2 checkpoint. similarly to bertshare and robertashare, the memory footprint of this model is smaller compared to an encoder-decoder setup (125m parameters). rnd2gpt a randomly initialized encoder paired with a gpt-2-compatible decoder. we warmstart the decoder and the embedding matrix with a public gpt-2 checkpoint. bert2gpt a bert-compatible encoder paired with a gpt-2-compatible decoder. we warmstart both sides with the two separate, bert and gpt-2, public checkpoints. we use the bert vocabulary for the input and the gpt-2 vocabulary for the output. roberta2gpt same as bert2gpt, but we use a public roberta checkpoint to warm-start the encoder. roberta was trained using the gpt-2 vocabulary so we can use it for input and output. note that although the vocabulary is shared, this model still has two embeddings matrices, one for the input and one for the output. the pre-training objective in the bert models learns to predict a masked token using the bidirectional representation of the input text @xcite . our decoder, even when initialized with the bert or roberta checkpoints, always generates the output text in an autoregressive fashion as in tranformers @xcite and gpt-2 @xcite . discofuse 100% 10% 1% exact sari sari sari @xcite we performed the bulk of our experiments on the 12-layer checkpoints of bert, gpt-2, and roberta, assuming that the findings will also hold for the 24-layer checkpoints. we chose bertshare, robertashare and roberta to also report numbers using the 24-layer public pretrained checkpoints. we also experimented with the gpt setup with 24 layers and 345m parameters but as we did not achieve any better results we excluded this from the paper. ## discussion on ablation studies combining different checkpoints. combining bert and gpt-2 into a single model (bert2gpt) did not work and often underperformed than a randomly initialized baseline. this is presumably because the model has to learn two different vocabularies. this argument is backed by the fact that for mt, @xmath0. for this task the vocabulary setting is in favor of this particular task, meaning that two vocabularies have to be learned anyways and the output is english, on which gpt-2 was trained. because roberta and gpt-2 share the same vocabulary, combining both into a single model (roberta2gpt) showed strong results on several tasks but did not outperform a setup where roberta is used in the encoder and decoder. tuning gpt-2 based models. we were surprised that setups using the gpt-2 checkpoint performed relatively poorly given that it is trained as a language model on a large corpus; our intuition was that gpt-2 initialized decoders will be strong natural language generators. to ensure that this was not due to an unfortunate choice of hyperparameters, we tuned the learning rate, the warmup steps, and the @xmath1, adafactor} for the gpt-2 based setups (rnd2gpt, gpt, bert2gpt) on the discofuse dataset. naturally, this gave us slightly higher numbers but not at a magnitude that would suggest a previously suboptimal setting. specifically, we obtained a sari score of 88.8 compared with 88.4 for bert2gpt, 88.1 compared with 88.0 for gpt, and 87.7 compared with 86.5 for rnd2gpt. initializing only embeddings. we want to investigate the impact of the non-contextualized bert and gpt-2 embeddings. this means we are initializing the transformer model with only the embedding matrices. the advantage of this setup would be that we could freely choose the model architecture and size and adapt it to a specific task. we found almost no improvement over the fully randomly initialized model rnd2rnd. concretely, we compute a sari score of 87.1 using the bert embeddings and 87.0 using the gpt-2 embeddings, compared with 86.9 of the rnd2rnd baseline. we observe slightly higher improvements of up to 2 percentage points when training on only 10% of the training data. initializing only layers. contrary to the previous paragraph, we want to investigate the effect of initializing everything but the word embedding matrix. the embedding matrix accounts for only 10% to 31% of all learnable parameters, and sometimes the vocabulary given from a public checkpoint might not be optimal for a certain task. in these cases, it would be nice to redefine the vocabulary while still leveraging the checkpoint. first, we remove the embeddings matrices from the warm-started variables and observe a drop of 1.7 points using the bertshare setup and 11 points using the gpt setup ( initializing a subset of layers. motivated by the results of using 24 layers, we want to investigate whether only a subset of these 24 layers can be used. to account for the larger hidden layer size (1,024 vs. 768) and filter size (4,096 vs. 3,072), we limit ourselves to using only 10 layers and the embedding matrix of this model. this model still bertshare gpt discofuse 89.3 88.0 -embeddings from checkpoint 87.5 77.0 + task specific sentencepieces 87.5 84.2 + pre-training sentencepieces 88.0 69. 7 we then performed a hyperparameter search on the evaluation set using cma-es @xcite to find an optimal subset of layers to use. the best setup used the following layers: 9, 10, 13-18, 23, 24; and achieved a sari score of 89.1. although this is a remarkable improvement over using the first 10 layers, this setup is still outperformed by the base bert model. ## analysis of abstractive summaries finally, we present a qualitative analysis of these models for text generation. in particular, we focus on extreme summarization, which assesses models ability to do document-level inference and abstraction. we evaluated summaries from arandomly initialized model (rnd2rnd) and from best performing models initialized with gpt checkpoints (rnd2gpt), bert checkpoints (bertshare), and roberta checkpoints (robertashare). we also included gold summaries in our evaluation. results are presented in human assessment of summary quality. the study was conducted on the amazon mechanical turk platform using best-worst scaling, a less labor-intensive alternative to paired comparisons @xcite . our participants were presented with a document and summaries generated from two out our participants found the robertashare summaries to be the best in terms of their overall quality; the bertshare summaries ranked second after robertashare. we further carried out pairwise comparisons between all models to assess whether system differences are statistically significant. 9 we did not observe significant differences between rnd2rnd and rnd2gpt, rnd2rnd and bertshare, and, robertashare and gold. all other differences were statistically significant. ## conclusion we performed an extensive study on leveraging pre-trained checkpoints for sequence generation. our findings show that a pre-trained encoder is an essential part. most tasks also profit from sharing the weights between the encoder and the decoder, which additionally decreases the memory footprint. while combing bert and gpt-2 into a single model often underperformed a randomly initialized baseline, combining roberta and gpt-2 achieved strong results and shows the importance of sharing the vocabulary. training a language-specific bert model also improves performance over using the multilingual version.