Recipes for Sequential Pre-training of Multilingual Encoder and S eq2 S eq Models (2023)

Abstract: Pre-trained encoder-only and sequence-to-sequence (seq2seq) models each have advantages, however training both model types from scratch is computationally expensive. We explore recipes to improve pre-training efficiency by initializing one model from the other. (1) Extracting the encoder from a seq2seq model, we show it under-performs a Masked Language Modeling (MLM) encoder, particularly on sequence labeling tasks. Variations of masking during seq2seq training, reducing the decoder size, and continuing with a small amount of MLM training do not close the gap. (2) Conversely, using an encoder to warm-start seq2seq training, we show that by unfreezing the encoder partway through training, we can match task performance of a from-scratch seq2seq model. Overall, this two-stage approach is an efficient recipe to obtain both a multilingual encoder and a seq2seq model, matching the performance of training each model from scratch while reducing the total compute cost by 27%.

## introduction and related work transformer-based pre-trained language models (plms) have become the main building blocks when creating models for most natural language processing (nlp) tasks. plms come in three main architectures: decoder-only (e.g. gpt), sequence-to-sequence (seq2seq, e.g. bart, t5), and encoder-only (e.g. bert). multilingual models such as xlm-roberta (encoder-only) and mbart/mt5 (seq2seq) are also common. @xcite showed that seq2seq models can perform many nlp tasks on par with similarlysized encoder-only models trained via masked language modeling (mlm) by framing tasks such a sentence classification or sequence labeling as text generation. however, encoder models remain more efficient at inference for sequence labeling tasks * equal contribution. figure 1 : two-stage seq2seq pre-training. first (left), we train the encoder via masked language modeling (mlm). second (right), we attach a randomly initialized decoder to the pre-trained mlm encoder, and train on the same data with de-noising objective. the encoder may remain frozen for part or all of the second stage. motivated by the need for both an encoder model for efficient sequence labeling and a seq2seq model for generative tasks like semantic parsing and summarization, we explore recipes to pre-train both models. compared to training each model from scratch, we propose two sequential training recipes which reduce the total compute cost (section 2.1.6). the first recipe is to extract the encoder of a seq2seq model as proposed in @xcite . although it performs well on classification tasks, we show that the encoder from seq2seq underperforms a from-scratch encoder on sequence labeling tasks. variations of masking during seq2seq training and reducing the decoder size do not provide a consistent benefit to the encoder. we also explore continuing training the extracted encoder on mlm for a small number of updates. however, we show it cannot consistently close the gap in performance across different datasets. the second recipe is to warm-start seq2seq pretraining with an encoder pre-trained via mlm (fig-ure 1 ). @xcite proposed a similar idea for fine-tuning. alexatm 20b and alexatm 5b applied this recipe for pre-training, by warm-starting with alexa teacher model encoders @xcite @xcite . we add the novelty of comparing to a seq2seq model pre-trained from scratch with the same data and codebase. first, we observe that if the encoder is frozen the whole time, the model under-performs a from-scratch seq2seq model on semantic parsing and summarization tasks. while cross-attention fusion across different layers of the encoder reduces the performance gap, we find that we can match performance of a from-scratch model by using standard cross-attention and unfreezing the encoder partway through training. overall, the second recipe demonstrates a viable approach for efficient pre-training of both a multilingual encoder and a multilingual seq2seq model, matching the performance of training each model from scratch, while using 27% less total compute. see appendix a for additional related work. ## pre-training setup we describe our pre-training objectives, models, datasets, two recipes for initializing one model type from the other, and compare compute costs. ## fine-tuning results we present the results on fine-tuning our pretrained models. all runs are averaged over three random seeds and reported as @xmath0. see appendix c for hyperparameters. ## conclusion and future work in this work, we studied recipes to efficiently pretrain both a multilingual encoder and a seq2seq model by re-using the weights from one model for the other. we found that the most effective recipe is to start training of a seq2seq model from a pretrained encoder and unfreeze it partway through the training. future work can explore even more efficient pre-training strategies such as jointly training on mlm and sequence-level de-noising objectives, and probe further why the encoders trained as part of a seq2seq model do not do well on sequence labeling tasks.