G an LM : Encoder-Decoder Pre-training with an Auxiliary Discriminator (2023)

Abstract: Pre-trained models have achieved remarkable success in natural language processing (NLP). However, existing pre-training methods underutilize the benefits of language understanding for generation. Inspired by the idea of Generative Adversarial Networks (GANs), we propose a GAN-style model for encoder-decoder pre-training by introducing an auxiliary discriminator, unifying the ability of language understanding and generation in a single model. Our model, named as GanLM, is trained with two pre-training objectives: replaced token detection and replaced token denoising. Specifically, given masked source sentences, the generator outputs the target distribution and the discriminator predicts whether the target sampled tokens from distribution are incorrect. The target sentence is replaced with misclassified tokens to construct noisy previous context, which is used to generate the gold sentence. In general, both tasks improve the ability of language understanding and generation by selectively using the denoising data. Extensive experiments in language generation benchmarks show that GanLM with the powerful language understanding capability outperforms various strong pre-trained language models (PLMs) and achieves state-of-the-art performance.

## introduction the pre-training-then-fine-tuning paradigm has been proven successful in many natural language processing tasks @xcite @xcite . while there are various pre-training approaches for the encoderonly architectures @xcite , the encoder-decoder pre-training is underexplored, which is essential for natural language generation. to pre-train the entire encoder-decoder model, bart @xcite proposes a denoising language model objective and t5 @xcite pre-trains the models with a span corruption objective. furthermore, mbart @xcite and mt5 (xue et al., 2021) extend them to be multilingual pre-trained language models. unlike most encoder-decoder pre-training methods that simply apply sequence-to-sequence tasks on a single encoder-decoder architecture, we explore the approaches to pre-train the model in a gan-style manner with an auxiliary discriminator. gan @xcite performs well on both text and image generation tasks by combining the generator and discriminator. it aims to improve the ability of the generator to produce highquality samples, which is important for the encoderdecoder pre-training when transferred to downstream generation tasks. similarly, maskgan @xcite shows the gan-like training can improve the quality of the autoregressive language model. therefore, it is intuitive to leverage gan to empower the encoder-decoder pre-training by unifying language understanding and generation. in this work, we propose a pre-training framework ganlm, using gan-style learning to im-prove the transferability of pre-trained language models for the natural language generation. specifically, the encoder reads the masked source sentence and the generator obtains target distribution. then, the discriminator distinguishes whether each token sampled from the target distribution matches the target gold sentence (replaced token detection). the misclassified tokens by discriminator are regarded as hard tokens for the generator to predict accurately. we replace original tokens in the target sentence with misclassified sampled ones to construct the noisy previous context for predicting the target sentence (replaced token denoising). in figure 1 , the generator predicts the masked words "guardian watered", where the incorrect token "guardian" and correct token "watered" are both misclassified into replaced and original by the discriminator. next, we resample a different token "watering" from the generated distribution. consequently, the target tokens "gardener watered" are replaced with the sampled tokens "guardian watering" to construct the noisy sample. the generator predicts the next word conditioned on previous noisy tokens (replaced token denoising). through combing two tasks, ganlm strengthen generation performance with the enhanced language understanding capability from the replaced token detection task. our method is effective for text generation and can be extended to natural language understanding tasks. we pre-train ganlm model on large-scale monolingual corpora and evaluate the performance of our pre-trained english model ganlm and multilingual model ganlm-m on various downstream tasks, including text summarization, machine translation, and data-to-text generation. experimental results demonstrate that our method substantially outperforms previous pre-trained encoder and sequence-to-sequence models on generation tasks. our method is further tested on glue @xcite and xnli @xcite to validate the transferability of our pre-trained model. analytic experiments emphasize the importance of the discriminator in both the pre-training and finetuning stage, leading to better performance. ## discriminator-enhanced fine-tuning to fully utilize the pre-trained parameters, we keep the auxiliary discriminator in downstream generation tasks (discriminator-enhanced fine-tuning) to enhance the generator, where both the pre-trained generator and discriminator are recycled. given the annotated corpus d s of k languages, the pretrained model {θ e , θ d , θ g } is optimized by: dataset following the previous work @xcite , our english pre-trained model ganlm is trained on 160gb english monolingual data from bookcorpus, cc-news, openwebtext, and cc-stories. in addition, we pre-train ganlm-m with 6tb multilingual data as the pioneering work @xcite , which is a combination of cc100, cc-net, and wikipedia, covering 100 languages. all texts are tokenized by sentencepiece @xcite and encoded by the dictionary from xlm-r @xcite . optimization for s2s-mlm, we randomly mask 15% of the words in each instance with an average span length of 3 @xcite . for the replaced token detection, we set the discriminator weight λ = 10.0. we adopt adam (kingma and ba, 2015) with a learning rate of 3e-4 and 10k warm-up steps for pre-training. the model is trained on 128 nvidia a100 gpus (40gb) from scratch and each batch contains 8k samples. the english pre-trained model ganlm and multilingual model ganlm-m are trained for 500k steps. specifically, all methods in ## comparing pre-training objectives to verify the potential of our pre-training task under a fair comparison, we re-implement previous pre-training tasks and pre-trains baselines on the same corpora with 500k steps, including bert/mbert @xcite , electra @xcite , bart (lewis et al., 2020)/ mbart (liu et al., 2020), and t5 (raffel et al., 2020)/mt5 (xue et al., 2021). we observe that the encoder-decoder pre-trained model (t5/mt5) outperforms the pre-trained encoder (electra, bert/mbert), which corroborates the encoder-decoder pre-training is more beneficial to the downstream generation task. experiments ⑥∼⑧ show the importance of the discriminator and replaced token denoising. experiment ⑧ demonstrates that only the replaced token detection task can still bring improvement through strengthening the encoder shared by both generator and discriminator. besides, the replaced token detection task is also helpful to downstream language understanding tasks with a powerful encoder. lastly, the results verify that fine-tuning with the help of the pre-trained auxiliary discriminator further improves performance. ## results of ganlm the english pre-trained model ganlm is evaluated on the abstractive text summarization task with the rouge @xcite scores. xsum as shown in ## results of ganlm-m to evaluate the multilingual pre-trained model ganlm-m, we report the bleu @xcite scores for machine translation and rouge @xcite scores for text summarization and datato-text generation. wikilingua ## analysis ablation study to analyze the effect of the proposed pre-training and fine-tuning strategies, we conduct an ablation study of each component of our method in ## conclusion in this work, we introduce ganlm, a state-ofthe-art pre-training encoder-decoder framework for both language generation and understanding tasks trained on large-scale corpora. our ganstyle models are pre-trained with replaced token detection and replaced token denoising by introducing an auxiliary discriminator. extensive experiments prove the effectiveness of ganlm on various language generation and translation benchmark datasets. we further verify the capability of the pre-trained model on multiple downstream understanding tasks.