LTRC @ M u P 2022: Multi-Perspective Scientific Document Summarization Using Pre-trained Generation Models (2022)

Abstract: The MuP-2022 shared task focuses on multiperspective scientific document summarization. Given a scientific document, with multiple reference summaries, our goal was to develop a model that can produce a generic summary covering as many aspects of the document as covered by all of its reference summaries. This paper describes our best official model, a finetuned BART-large, along with a discussion on the challenges of this task and some of our unofficial models including SOTA generation models. Our submitted model out performedthe given, MuP 2022 shared task, baselines on ROUGE-2, ROUGE-L and average ROUGE F1-scores. Code of our submission can be ac- cessed here.

## introduction with the rapidly growing research community, the volume of scientific papers being published every year is also going up. which makes it nearly impossible for researchers to stay on top of the latest research. scientific document summarization plays a crucial role in mitigating this problem. however, generating generic summaries for scientific documents is a non-trivial task due to their specific structure, varied content and inclusion of citation sentences. scientific articles often represent salient information through tables, figures, and pseudocodes (altmami and menai, 2020) and mathematical equations. and, generic text does not usually contain such elements. the two widely used approaches for scientific document summarization are content-based @xcite and citation-based @xcite @xcite . the former relies on traditional extractive and abstractive methods whereas, the latter locates the target paper by matching a portion of text with the citation sentences. almost all traditional summarization models, whether extractive or abstractive, follow supervised learning approach. that means, given a document the model learns to generate its summary based on its given gold (target) summary. however, in real world, summary writing is very subjective. for a given document, there could be multiple different yet valid summaries where each summary writer has written a summary of the same document from their perspective of the document. this subjectivity raises concerns about the evaluation ability of the model that is presented with only one gold summary. the mup-2022 shared task is a novel attempt to address this concern. the goal of multiperspective summarization task is to develop models that are capable of leveraging multiple gold summaries to generate one generic summary. mup-2022 shared task data contains a collection of scientific documents with multiple summaries. these summaries were collected by first taking (one or) multiple scientific peer reviews for each document and then extracting the introductory paragraph that summarizes the key contributions of the paper from the reviewer's perspective. for this task, we explored several pretrained sequence-to-sequence models such as bart @xcite , t5 @xcite , and prophetnet @xcite . we also experimented with: a two-stage fine-tuning approach using the scitldr dataset @xcite and the divide and conquer approach, by @xcite , that first divides the document into multiple sections to obtain section-wise summaries, and then aggregates all partial summaries to form the complete summary. for the mup 2022 shared task dataset, our finetuned bart large model remained the best among all our experiments by achieving 40.68 rouge-1 f1-score and 26.04 average rouge f1-score. ## related work research on summarizing scientific documents has been widely explored in recent years. it is perti- various generation models, including bart, t5, prophetnet, and pegasus, have shown great performance in summarization tasks. in particular, models like big bird @xcite and longformer @xcite were released to handle long documents. ## corpus description the multi-perspective scientific document summarization task aims to generate a summary that covers various aspects of the document. evaluating such a system with just one gold (or reference) summary negatively impacts the goal, as summaries are usually very subjective. considering the fact that multiple summaries would help cover more different perspectives of the scientific document, which a single summary might have missed. mup2022 @xcite shared task data foot_0 contains multiple reference summaries for majority of the training set documents, and all of the development set documents also had a minimum of 3 reference summaries. the corpus consists of around 10k papers and 26.5k summaries. the ## methodology self-supervised pretrained models like bart @xcite , t5 @xcite , xlnet @xcite , prophetnet @xcite , pegasus @xcite have been effective for many generative tasks. we experiment with these pre-trained models and fine-tune them on mup dataset for this task. bart is a transformer-based @xcite standard sequence-to-sequence model modified to work as an auto-encoder @xcite . a self-supervised autoencoder is trained on the corrupted text (addition of noise) and uses a language model to reconstruct the original text with the true replacement of corrupted tokens. bart uses five "noising" methods: token masking, token deletion, text infilling, sentence permutation, and document rotation. t5 or text to text transfer transformer (raffel et al., 2020) is a transformer-based approach that converts all the text-based language problems into the text-to-text format. this strategy allows the use of the same model architecture across a diverse set of tasks. t5 is pretrained on a multi-task mixture of supervised and unsupervised tasks using the common crawled corpus. we fine tune t5 base model on mup corpus. prophetnet @xcite is a sequence-tosequence pretraining model. the unique objective of this model is to predict the future n-grams as the self-supervised training strategy. unlike the traditional sequence-to-sequence models, prophetnet is optimized by n-step ahead prediction instead of one-step-ahead prediction. we experimented with prophetnet models with and without fine-tuned on the cnn/dailymail dataset. utilizing scitldr the tldr @xcite ) approach aims at creating extremely short summaries (tldrs) for scientific documents. for this task, the authors introduced a scitldr dataset of 5400 tldrs over 3200 papers. dancer (gidiotis and tsoumakas, 2020) most of the extractive and abstractive methods for scientific document summarization typically consider the input as abstract and/or full text of the article to generate the abstract-like summary. in contrast, dancer divides the source text into multiple sections, generates an individual summary for ## experiments all of our experiments were performed on the same splits of train, validation and test sets as provided by the organizers. the following subsections detail various categories of experiments. we hypothesise that various sections of the source document may contribute in multi-perspective reviews of the document reviews. the subsection 5.3 and 5.4 detail the experiments conducted, specifically, to capture various sections of the document. ## results & discussion for the mup task, we experimented with various pre-trained generation models, a couple of scien while the mup task considers summaries from multiple reviewers as different "perspectives", most of these summaries cover only the major contributions of the paper. these summaries, though diverse in their construction, do not look at the research paper from different points-of-view. we see a validation of this claim from the results in