Transfer Learning for Sequence Generation: from Single-source to Multi-source (2021)

Abstract: Multi-source sequence generation (MSG) is an important kind of sequence generation tasks that takes multiple sources, including automatic post-editing, multi-source translation, multi-document summarization, etc. As MSG tasks suffer from the data scarcity problem and recent pretrained models have been proven to be effective for low-resource downstream tasks, transferring pretrained sequence-to-sequence models to MSG tasks is essential. Although directly finetuning pretrained models on MSG tasks and concatenating multiple sources into a single long sequence is regarded as a simple method to transfer pretrained models to MSG tasks, we conjecture that the direct finetuning method leads to catastrophic forgetting and solely relying on pretrained self-attention layers to capture cross-source information is not sufficient. Therefore, we propose a two-stage finetuning method to alleviate the pretrain-finetune discrepancy and introduce a novel MSG model with a fine encoder to learn better representations in MSG tasks. Experiments show that our approach achieves new state-of-the-art results on the WMT17 APE task and multi-source translation task using the WMT14 test set. When adapted to document-level translation, our framework outperforms strong baselines significantly.

## introduction thanks to the continuous representations widely used across text, speech, and image, neural networks that accept multiple sources as input have gained increasing attention in the community @xcite . for example, multi-modal inputs that are complementary have proven to be helpful for many sequence generation tasks such as question answering @xcite single-source sg multi-source sg autoencoding bert-fused dualbert (e.g., bert) @xcite ) @xcite seq2seq in natural language processing, multiple textual inputs have also been shown to be valuable for sequence generation tasks such as multi-source translation @xcite , automatic postediting @xcite , multi-document summarization @xcite , system combination for nmt @xcite , and document-level machine translation @xcite . we refer to this kind of tasks as multi-source sequence generation (msg). unfortunately, msg tasks face a severe challenge: there are no sufficient data to train msg models. for example, multi-source translation requires parallel corpora involving multiple languages, which are usually restricted in quantity and coverage. recently, as pretraining language models that take advantage of massive unlabeled data have proven to improve natural language understanding (nlu) and generation tasks substantially @xcite @xcite , a number of researchers have proposed to leverage pretrained language models to enhance msg tasks @xcite @xcite . for example, @xcite show that pretrained autoencoding (ae) models like bert @xcite can improve automatic post-editing. as most recent pretrained sequence-to-sequence (seq2seq) models @xcite @xcite have demonstrated their effectiveness in improving single-source sequence generation (ssg) tasks, we believe that pretrained seq2seq models can potentially bring more benefits to msg than pretrained ae models. although it is easy to transfer seq2seq models to ssg tasks, transferring them to msg tasks is challenging because msg takes multiple sources as the input, leading to severe pretrain-finetune discrepancies in terms of both architectures and objectives. a straightforward solution is to concatenate the representations of multiple sources as suggested by @xcite . however, we believe this approach suffers from two major drawbacks. first, due to the discrepancy between pretraining and msg, directly transferring pretrained models to msg tasks might lead to catastrophic forgetting @xcite that results in reduced performance. second, the pretrained self-attention layers might not fully learn the representations of the concatenation of multiple sources because they do not make full use of the cross-source information. inspired by adding intermediate tasks for nlu @xcite , we conjecture that inserting a proper intermediate task between them can alleviate the discrepancy. in this paper, we propose a two-stage finetuning method named gradual finetuning. different from prior studies, our work aims to transfer pretrained seq2seq models to msg (see ## approach figure 1 shows an overview of our framework. first, the problem statement is described in section 2.1. second, we propose to use the gradual finetuning method (section 2.2) to reduce the pretrainfinetune discrepancy. third, we introduce our msg model, which consists of the coarse encoder (section 2.3), the fine encoder (section 2.4), and the decoder (section 2.5). ## conclusion we propose a novel task-agnostic framework, trice, to conduct transfer learning from single-source sequence generation including selfsupervised pretraining and supervised generation to multi-source sequence generation. with the help of the proposed gradual finetuning method and the novel msg model equipped with coarse and fine encoders, our framework outperforms all baselines on three different msg tasks in four different data scales, which shows the effectiveness and generalizability of our framework.