BART : Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2020)

Abstract: We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and other recent pretraining schemes. We evaluate a number of noising approaches, finding the best performance by both randomly shuffling the order of sentences and using a novel in-filling scheme, where spans of text are replaced with a single mask token. BART is particularly effective when fine tuned for text generation but also works well for comprehension tasks. It matches the performance of RoBERTa on GLUE and SQuAD, and achieves new state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains of up to 3.5 ROUGE. BART also provides a 1.1 BLEU increase over a back-translation system for machine translation, with only target language pretraining. We also replicate other pretraining schemes within the BART framework, to understand their effect on end-task performance.

## introduction self-supervised methods have achieved remarkable success in a wide range of nlp tasks @xcite @xcite @xcite . the most successful approaches have been variants of masked language models, which are denoising autoencoders that are trained to reconstruct text where a random subset of the words has been masked out. recent work has shown gains by improving the distribution of masked tokens @xcite , the order in which masked tokens are predicted @xcite , and the available context for replacing masked tokens @xcite . however, these methods typically focus on particular types of end tasks (e.g. span prediction, generation, etc.), limiting their applicability. in this paper, we present bart, which pre-trains a model combining bidirectional and auto-regressive transformers. bart is a denoising autoencoder built with a sequence-to-sequence model that is applicable to a very wide range of end tasks. pretraining has two stages (1) text is corrupted with an arbitrary noising function, and (2) a sequence-to-sequence model is learned to reconstruct the original text. bart uses a standard tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing bert (due to the bidirectional encoder), gpt (with the left-to-right decoder), and many other more recent pretraining schemes (see figure 1 ). a key advantage of this setup is the noising flexibility; arbitrary transformations can be applied to the original text, including changing its length. we evaluate a number of noising approaches, finding the best performance by both randomly shuffling the order of the original sentences and using a novel in-filling scheme, where arbitrary length spans of text (including zero length) are replaced with a single mask token. this approach generalizes the original word masking and next sentence prediction objectives in bert by forcing the model to reason more about overall sentence length and make longer range transformations to the input. bart is particularly effective when fine tuned for text generation but also works well for comprehension tasks. it matches the performance of roberta @xcite with comparable training resources on glue @xcite and squad @xcite , and achieves new state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks. for example, it improves performance by 3.5 rouge over previous work on xsum @xcite . bart also opens up new ways of thinking about fine tuning. we present a new scheme for machine translation where a bart model is stacked above a few additional transformer layers. these layers are trained bidirectional encoder a _ c _ e b d (a) bert: random tokens are replaced with masks, and the document is encoded bidirectionally. missing tokens are predicted independently, so bert cannot easily be used for generation. autoregressive decoder a b c d @xmath0: tokens are predicted auto-regressively, meaning gpt can be used for generation. however words can only condition on leftward context, so it cannot learn bidirectional interactions. autoregressive decoder bidirectional encoder a b c d e a _ b _ @xmath1: inputs to the encoder need not be aligned with decoder outputs, allowing arbitary noise transformations. here, a document has been corrupted by replacing spans of text with a mask symbols. the corrupted document (left) is encoded with a bidirectional model, and then the likelihood of the original document (right) is calculated with an autoregressive decoder. for fine-tuning, an uncorrupted document is input to both the encoder and decoder, and we use representations from the final hidden state of the decoder. figure 1 : a schematic comparison of bart with bert @xcite and gpt @xcite . to essentially translate the foreign language to noised english, by propagation through bart, thereby using bart as a pre-trained target-side language model. this approach improves performance over a strong back-translation mt baseline by 1.1 bleu on the wmt romanian-english benchmark. to better understand these effects, we also report an ablation analysis that replicates other recently proposed training objectives. this study allows us to carefully control for a number of factors, including data and optimization parameters, which have been shown to be as important for overall performance as the selection of training objectives @xcite . we find that bart exhibits the most consistently strong performance across the full range of tasks we consider. ## model bart is a denoising autoencoder that maps a corrupted document to the original document it was derived from. it is implemented as a sequence-to-sequence model with a bidirectional encoder over corrupted text and a left-to-right autoregressive decoder. for pre-training, we optimize the negative log likelihood of the original document. ## fine-tuning bart the representations produced by bart can be used in several ways for downstream applications. ## comparing pre-training objectives bart supports a much wider range of noising schemes during pre-training than previous work. we compare a range of options using base-size models (6 encoder and 6 decoder layers, with a hidden size of 768), evaluated on a representative subset of the tasks we will consider for the full large scale experiments in §5. ## large-scale pre-training experiments recent work has shown that downstream performance can dramatically improve when pre-training is scaled to large batch sizes @xcite and corpora. to test how well bart performs in this regime, and to create a useful model for downstream tasks, we trained bart using the same scale as the roberta model. ## qualitative analysis bart shows large improvements on summarization metrics, of up to 3.5 points over the prior state-of-theart. to understand bart's performance beyond automated metrics, we analyse its generations qualitatively. ## related work early methods for pretraining were based on language models. gpt @xcite only models leftward context, which is problematic for some tasks. elmo @xcite concatenates left-only and right-only representations, but does not pre-train interactions between these features. @xcite demonstrated that very large language models can act as unsupervised multitask models. bert @xcite introduced masked language modelling, which allows pre-training to learn in-teractions between left and right context words. recent work has shown that very strong performance can be achieved by training for longer @xcite , by tying parameters across layers @xcite , and by masking spans instead of words @xcite . predictions are not made auto-regressively, reducing the effectiveness of bert for generation tasks. unilm @xcite fine-tunes bert with an ensemble of masks, some of which allow only leftward context. like bart, this allows unilm to be used for both generative and discriminative tasks. a difference is that unilm predictions are conditionally independent, whereas bart's are autoregressive. bart reduces the mismatch between pre-training and generation tasks, because the decoder is always trained on uncorrupted context. mass @xcite is perhaps the most similar model to bart. an input sequence where a contiguous span of tokens is masked is mapped to a sequence consisting of the missing tokens. bart differs in masking more but shorter spans from the input, and in always predicting the complete output. xl-net @xcite extends bert by predicting masked tokens auto-regressively in a permuted order. this objective allows predictions to condition on both left and right context. in contrast, the bart decoder works left-to-right during pre-training, matching the setting during generation. concurrently, @xcite pre-trained a denoising sequence-to-sequence model named t5, experimenting with a similar range of noising tasks. bart uses a slightly different objective, in which spans are masked from the input but the complete output is predicted, to improve the decoder's language modelling ability. bart achieves higher performance with similar model sizes, particularly on summarization. t5 demonstrates that by scaling to very large models sizes, denoising sequence-to-sequence pre-training can achieve new state-of-the-art results on many tasks. several papers have explored using pre-trained representations to improve machine translation. the largest improvements have come from pre-training on both source and target languages @xcite , but this requires pretraining on all languages of interest. other work has shown that encoders can be improved using pre-trained representations @xcite , but gains in decoders are more limited. we show how bart can be used to improve machine translation decoders. ## conclusions we introduced bart, a pre-training approach that learns to map corrupted documents to the original. bart performs comparably to roberta on discriminative tasks, and achieves new state-of-the-art results on several text generation tasks. future work should explore new methods for corrupting documents for pretraining, perhaps tailoring them to specific end tasks.