Transfer-Learning based on Extract, Paraphrase and Compress Models for Neural Abstractive Multi-Document Summarization (2024)

Abstract: Recently, transfer-learning by unsupervised pre-training and fine-tuning has shown great success on a number of tasks. The paucity of data for multi-document summarization (MDS) in the news domain, especially makes this approach practical. However, while existing literature mostly formulate unsupervised learning objectives tailored for/around the summarization problem we find that MDS can benefit directly from models pre-trained on other downstream supervised tasks such as sentence extraction, paraphrase generation and sentence compression. We carry out experiments to demonstrate the impact of zero-shot transfer-learning from these downstream tasks on MDS. Since it is challenging to train end-to-end encoder-decoder models on MDS due to i) the sheer length of the input documents, and ii) the paucity of training data. We hope this paper encourages more work on these downstream tasks as a means to mitigating the challenges in neural abstractive MDS.

## introduction text summarization aims at presenting salient points of a text, concisely and fluently. in mds the sources of text albeit multiple, convey a central idea or topic. for example, news article from different sources on a defined topic @xcite , questionnaires completed by various individuals @xcite or varied reviews from different users on a certain product @xcite . this paper addresses summarization of multiple news articles. despite the applications of mds, not much neural-based approaches @xcite @xcite @xcite exist in literature due to two main challenges -the lack of enormous parallel training data and the lengthy size of the input documents. the latter makes it especially challenging to encode and decode in an end-to-end fashion owing to memory constraints of the machine @xcite . to solve both problems, we propose transfer-learning @xcite ) from pre-trained supervised models. first, we extract salient sentences by directly applying a pre-trained extractive summarization model. next, we implement key abstraction techniques such as paraphrase generation @xcite and sentence compression @xcite on the extracted sentences using supervised pretrained models to generate abstractive summaries. in contrast to existing works that require further adaptation @xcite @xcite of the pre-trained models, our transferlearning method is direct and requires no mds training data. our main contributions are highlighted as follows: (1) we present a method for transfer-learning from transformer-based models pre-trained on downstream tasks, (2) we demonstrate the utility of downstream tasks, such as sentence extraction, paraphrase generation and sentence compression on mds, and (3) our method is simple and requires no mds training data. ## methodology our method investigates how models tailored specifically for downstream tasks pre-trained on their dedicated labelled datasets can be directly beneficial for mds. we investigate the utility of three (3) downstream tasks for this experimentsentence extraction, paraphrase generation and sentence compression. additionally, we compare our method against the performance of two (2) recent pre-trained language models -gpt2 and t5. ## related work our work is related to paradigms such as extractand-compress @xcite @xcite ; extract-and-paraphrase @xcite @xcite . however it differs significantly from these models in the following ways: i) it requires zero training on data for the task it is being applied -mds ii) it requires no architectural changes or augmentations to the pre-trained models iii) it consists of three (3) pipe-lined downstream tasks instead of two (2) in comparison to existing work. the simplicity of the pipeline enables various instantiations. despite, its simplicity it is competitive with current state-of-the-art mds models -pegasus (zhang et al., 2019) and mg-sumabs (jin et al., 2020) which are specifically tailored for abstractive text summarization with lots of parameters. zhang et al. (2019) proposed a large transformer-based encoder-decoder model pre-trained on a massive text corpora with new selfsupervised objective and fine-tuned on a variety of summarization datasets. jin et al. (2020) proposed a multi-granularity interaction network that encodes semantic representations for documents, sen-tences, and words for mds. lebanoff et al. (2018) and zhang et al. (2018a) adapt the neural model trained on abstractive sds for mds. our methods utilize transfer-learning from tasks/models not specifically engineered for abstractive summarization yet yields impressive results. existing mds methods are mostly extractive. these extractive methods are majorly modelled as graph operations with peculiarities on edge weight assignment. yasunaga et al. (2017) recently proposed a graph convolutional neural (gcn) network with sentence embeddings obtained from rnns as input node features. abstractive mds on the other hand, has met with limited research due to data limitations. @xcite proposed a neural model which is capable of encoding multiple input documents hierarchically. @xcite handled mds in two stages -extract and abstract. abstraction was performed by a decoder-only sequence transduction model. our approach is much similar to @xcite and @xcite that adapt the neural model trained on sds for mds by fine-tuning on the mds dataset. we use the sds model as-is in the extractive stage, making no changes to the encoder or decoder. additionally, different from their methods, we incorporate other downstream tasks like paraphrasing and sentence compression. ## conclusion we demonstrated the utility of sentence extraction, paraphrase generation and sentence compression for mds. we show that these tasks need not be pre-trained on abstractive summarization corpora or with abstractive summarization learning objectives. we hope this paper serves as a test bed for experiments in mds driven by transfer-learning and encourages similar approach to related tasks and for problems with limited training data.