An Extractive-Abstractive Approach for Multi-document Summarization of Scientific Articles for Literature Review (2022)

Abstract: Research in the biomedical domain is con- stantly challenged by its large amount of ever- evolving textual information. Biomedical re- searchers are usually required to conduct a lit- erature review before any medical interven- tion to assess the effectiveness of the con- cerned research. However, the process is time- consuming, and therefore, automation to some extent would help reduce the accompanying information overload. Multi-document sum- marization of scientific articles for literature reviews is one approximation of such automa- tion. Here in this paper, we describe our pipelined approach for the aforementioned task. We design a BERT-based extractive method followed by a BigBird PEGASUS-based ab- stractive pipeline for generating literature re- view summaries from the abstracts of biomedi- cal trial reports as part of the Multi-document Summarization for Literature Review (MSLR) shared task1 in the Scholarly Document Pro- cessing (SDP) workshop 20222. Our proposed model achieves the best performance on the MSLR-Cochrane leaderboard3 on majority of the evaluation metrics. Human scrutiny of our automatically generated summaries indicates that our approach is promising to yield readable multi-article summaries for conducting such lit- erature reviews.

## introduction the effectiveness of medical treatments following medical diagnosis can have both acknowledgments and contradictions with respect to various studies conducted. prior to any medical treatment, evidence synthesis is essential to understand and stay up-to-date with medical advances from different clinical studies. a literature survey provides high-quality evidence for healthcare. however, such a task is very time-consuming if done manually. to mitigate these issues high-quality largescale multi-document summarization datasets, e.g., the cochrane dataset @xcite and multi-document summarization of medical studies (ms2) dataset @xcite were developed. both the datasets consists of a wide variety of task-oriented summaries from clinical trials. to further encourage community research in multi-document summarization of biomedical reviews, the allen institute for artificial intelligence (or "ai2" for short) proposed a shared task named multi-document summarization for literature review (mslr) 2022 4 . the mslr shared task aims at summarizing and analyzing medical evidence from different clinical studies. the task consists of two datasets -cochrane and ms2, which provide a brief narrative summary from the abstracts of different clinical studies communicating the main findings. in this paper, we describe our system submission for the task. we participated in the cochrane subtask. in our system submission, we design a pipelined approach leveraging state-of-the-art neural extractive and abstractive summarization models. our system first extracts the vital information from the abstracts of all papers under a particular review id and then generates an abstractive summary, with the help of pre-trained bigbird pega-sus model @xcite , as the literature review test for that review id. ## related work the concerned task in mslr is a novel one and hence not much prior works were conducted except the papers that proposed the datasets. however, in this section we discuss some relevant recent works on multi-document summarization. @xcite propose an unsupervised method of using topic based clustering of fragments extracted from ## dataset description the mslr2022 shared task consists of two subtasks based on the cochrane dataset @xcite and the ms2 dataset @xcite . in the cochrane dataset there are approximately 4.5k systematic reviews of all trials relevant to a given clinical question, compiled by members of the cochrane collaboration. the dataset consists of the summarized systematic reviews along with the titles and abstracts of, on an average, 10 clinical trials each. the average length of the abstracts of the included trials is 245 words and the target summary is of the average length of 75 words. ms2 is a dataset containing 20k medical systematic reviews from approximately 470k studies collected from pubmed, created as an annotated subset of the semantic scholar research corpus. the ms2 dataset is much larger than the cochrane dataset, but the latter contains cleaner data. for this shared task, the inputs and the target summaries are oriented in the same format which is then split into train, dev and test. ## methodology multi-document summarization aims to have a summary with maximum coverage and cohesiveness with less redundant data from the given set of papers pertaining to a topic. sequence-tosequence models do not perform well with large input sizes @xcite . hence, we choose to leverage an extractive-abstractive summarization technique in our approach, to summarize biomedical reviews of correlated papers. in extractive summarization, we select a pre-decided number of statements from a given text as a relatively shorter representation of the entire text. we choose the lecture summarizer model in order to extract the most important sentences. this extraction is done by using a clustering algorithm on a set of embeddings, which are basically the contextual representations of sentences obtained from a bert encoder. hence, this also assists in maintaining some sort of coherence withing the input text. we primarily use the provided abstracts as inputs to the extractive summarizer. for the titles that do not have any abstract, we use the titles as the inputs instead. we shorten these inputs to have at most five sentences from every different paper within a given review_id. we use bert extractive summarizer @xcite , a model that performs extractive text summarization on lecture transcripts. we pass the abstracts separately to this model. the model first generates the contextual embeddings of the the input sentences. further, the k-means clustering algorithm is used to find the k-sentences closest to the cluster's centroids. we proceed with the top 5 sentences from the cluster. the workflow of the model is provided in figure 1 . for every review_id, we join the short extracts from different papers under that particular id, and use the resulting sequence as the input sequence for training the abstractive summarization module. these shortened extracts, put together with the target summaries from original cochrane dataset, give us a new data. we choose the bigbird pe-gasus model from @xcite , and finetune it on this newly obtained dataset. this model uses global attention and random attention on the input sequences apart from sparse-attention, which theoretically approximates to full attention. this sparse-attention mechanism can handle sequences of length up to 8x compared to what was possible prior to this and simultaneously reduces the quadratic dependency to linear, hence making the model suitable to learn using longer input sequences. we finetune the model from the checkpoint 'google/bigbird-pegasus-large-pubmed' using the newly created data for 6 epochs with a batch size of 4 and an initial learning rate of 2e-5 accompanied by fp16 precision training. the final output of the abstractive summarization module is the 'related works' text corresponding to the research topic aligned with a particular review_id. figure 1 shows the workflow of our hybrid extractiveabstractive system. ## result and analysis the task realizes rouge (-1,2,l) @xcite , bertscore f1 @xcite , along with delta ei average divergence and macro f1 to be best suitable metrics for evaluation. hence, to monitor the training, we use rouge as the basis of evaluation. rouge scores do not sufficiently measure the factual correctness of statements. ## limitations there are no ground truth summaries for lecture summarizer and therefore no metric for evaluating the outputs that we receive from the model. due to the use of a clustering algorithm, the extractive part of out system is not readily trainable. we notice that the same model could not perform well in the subtask using the ms2 dataset. this can pertain to the long input sequences which is much greater than the cochrane input sizes. sequenceto-sequence models tend to not perform well with larger input sizes. even if we shorten the input sequences, we would be losing out of essential information from the original data. ## conclusion with the increasing rate of research and publications, literature reviews help keep track of the various advancements in the respective domains. automation, although essential, also opens up new challenges including summarization over contradictory information present in different studies over a particular topic and summarization quality. although our results show that our hybrid approach can be used for generating fluent high-quality literature review summaries, there is still significant scope for improvement. additionally, ethical concern involving the factuality of the summaries also comes into play because deploying such a system without proper monitoring is speculative when it comes to such a high-impact domain as healthcare. this task helps us understand the challenges in multi-document summarization in the high-impact biomedical domain. the future scope of research can include trying real-world applications of such systems having proper evaluation and monitoring strategies to test the correctness of the summaries.