PALM : Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation (2020)

Abstract: Self-supervised pre-training, such as BERT, MASS and BART, has emerged as a powerful technique for natural language understanding and generation. Existing pre-training techniques employ autoencoding and/or autoregressive objectives to train Transformer-based models by recovering original word tokens from corrupted text with some masked tokens. The training goals of existing techniques are often inconsistent with the goals of many language generation tasks, such as generative question answering and conversational response generation, for producing new text given context. This work presents PALM with a novel scheme that jointly pre-trains an autoencoding and autoregressive language model on a large unlabeled corpus, specifically designed for generating new text conditioned on context. The new scheme alleviates the mismatch introduced by the existing denoising scheme between pre-training and fine-tuning where generation is more than reconstructing original text. An extensive set of experiments show that PALM achieves new state-of-the-art results on a variety of language generation benchmarks covering generative question answering (Rank 1 on the official MARCO leaderboard), abstractive summarization on CNN/DailyMail as well as Gigaword, question generation on SQuAD, and conversational response generation on Cornell Movie Dialogues.

## introduction self-supervised pre-training has achieved great success in a wide range of natural language understanding (nlu) tasks @xcite @xcite @xcite . different from language understanding, language generation aims at generating natural language sentences, including tasks like neural machine translation @xcite , abstractive summarization @xcite @xcite , generative question answering (qa) @xcite , question generation @xcite and conversational response generation @xcite . many of the language generation tasks require the models to read and to comprehend a given document, based on which output text is generated. in this paper, we present palm, a novel approach to pre-training an autoencoding&autoregressive language model for text generation based on reading comprehension of textual context. recently, several pre-training methods have been proposed for language generation. gpt @xcite and gpt-2 @xcite use a leftto-right transformer decoder to generate a text sequence token-by-token, which lacks an encoder to condition generation on context. in contrast, mass @xcite and bart @xcite both employ a transformer-based encoderdecoder framework, with a bidirectional encoder over corrupted (masked) text and a left-to-right decoder reconstructing the original text. while such denoising pre-training objectives work well for the downstream generation tasks where generated text comes from input but is manipulated, they are less related to the comprehension-based generation tasks asking for instead generating continuations, responses or answers by comprehending input context. palm is specifically designed to pre-train a backbone model on a large unlabeled corpus for fine-tuning on the downstream comprehensionbased generation tasks, one example of which is generative qa. in generative question answering, qa models are asked to generate an abstractive answer in natural language to a given question by reading and comprehending a contextual passage. abstractive answer generation is more than manipulating tokens in the passage. an abstractive answer reflects the understanding of the passage and the question, and can include content out of the passage to be self-contained and well-formed. to address comprehension-based generation like generative qa, palm uses the pre-training objectives that are closely related to the downstream tasks. specifically, it differs from existing generative pre-training methods in that palm goes beyond the solely autoencoding/autoregressive methods and combines the merits of autoencoding and autoregression in a single framework. moreover, it possesses a mechanism built in pre-training for generating coherent text from given context. with the new design, palm surpasses existing language generation methods with or without pre-training -it was trained on 16 nvidia v100 gpus for 3 days in our experiments, and expected to perform even better if trained for longer. palm gives surprisingly good empirical results on a variety of context-aware generation tasks, including pushing the state-of-the-art rouge-l on the marco natural language generation benchmark to 0.498 (rank 1 on the leaderboard 1 ) and on gigaword summarization to 36.75, as well as establishing the state-of-the-art rouge-1 (44.30) and rouge-l (41.41) on cnn/daily mail. we make the following major contributions in this paper: ## experiments in this section, we present the experimental setup and results of palm pre-training on a large unlabeled corpus and fine-tuning on a variety of language generation tasks, including generative qa, abstractive summarization, question generation, and conversational response generation. ## related work elmo @xcite ) is an early prominent pre-training method based on bidirectional lstms. it concatenates left-only and right-only representations, but does not pre-train interactions between these features. gpt @xcite , gpt-2 @xcite and gpt-3 @xcite are proposed to base language modeling on the transformer architecture, and use only the transformer decoder for pre-training. edunov et al. @xcite examine different strategies (e.g., elmo) to add contextualized embeddings to sequence-to-sequence models, and observe the most improvement by adding the learned embeddings to the encoder. bert @xcite introduces masked language modelling, which allows pre-training to learn interactions between left and right context words. recent work has shown that very strong performance can be achieved by training for longer @xcite , by tying parameters across layers @xcite , and by masking spans instead of words @xcite . however, bert does not make predictions autoregressively, so it is not effective for generation tasks. unilms @xcite fine-tune bert with an ensemble of masks, some of which use only leftward context, allowing unilms to be used for generation tasks. a difference between unilms and palm is that unilms are not fully autoregressive in the pre-training process. in contrast, palm reduces the mismatch between pre-training and context-conditioned generation tasks by forcing the decoder to predict the continuation of text input on an unlabeled corpus. mass @xcite and bart @xcite are the two pre-training methods most similar to palm. in mass, an input sequence with a masked span of tokens is mapped to a sequence consisting of the missing tokens, whereas bart is trained to reconstruct the original text from corrupted input with some masked tokens. the difference in input & output representations between palm and mass & bart is detailed in section 2.2. ## conclusions in this work, we propose palm, a novel approach to pre-training an autoencoding and autoregressive language model on a large unlabeled corpus, designed to be fine-tuned on downstream generation conditioned on context. it is built upon an extension of the transformer encoder-decoder, and jointly pre-trains the encoder and the decoder in an autoencoding denoising stage followed by an autoregressive generation stage. palm significantly advances the state-of-the-art results on a variety of context-conditioned generation applications, including generative qa (rank 1 on the marco leaderboard), abstractive summarization, question generation, and conversational response generation. it has been shown in prior work @xcite that training for more steps over a larger corpus can potentially improve the performance of pre-training. our future work will explore the potential of training palm for longer on much more unlabeled text data.