Evaluating Pre-Trained Language Models on Multi-Document Summarization for Literature Reviews (2022)

Abstract: Systematic literature reviews in the biomedical space are often expensive to conduct. Automation through machine learning and large language models could improve the accuracy and research outcomes from such reviews. In this study, we evaluate a pre-trained LongT5 model on the MSLR22: Multi-Document Summarization for Literature Reviews Shared Task datasets. We weren’t able to make any improvements on the dataset benchmark, but we do establish some evidence that current summarization metrics are insufficient in measuring summarization accuracy. A multi-document summarization web tool was also built to demonstrate the viability of summarization models for future investigators: https://ben-yu.github.io/summarizer

## introduction with recent advances in natural language processing and deep learning, large language models are now capable of generating summaries of large volumes of documents that are arguably human readable and logically consistent. with the growing amount of research being published, it has become increasingly difficult to process all the available research and literature in any particular field of study. this has become exceedingly important within the biomedical field as the community has learned with the global covid-19 pandemic. speed of research directly impacts patient outcomes and how fast medical practitioners can respond to a constantly changing health landscape. the mslr22: multi-document summarization for literature reviews shared task proposes a challenging research problem that pushes current state of the art multidocument summarization models to generalize over two different datasets: ms^2 dataset @xcite and cochrane dataset @xcite we will evaluate in this research study if pre-trained summarization models can successfully solve the proposed task. ## related work recent studies in document summarization have mostly focused on transformer-based models, but applied to the biomedical context either through transfer learning or fine-turning on a specific biomedical dataset @xcite . biobert-sum is a recent example of using such pre-training methodologies, which used a pre-trained model as an encoder and fine-tuned on a specific task @xcite . @xcite innovated in this space by applying hierarchical clustering to group contextual embeddings of sentences to select the most informative sentences from a given group to generate summaries. @xcite also recently proposed a mechanism to leverage domain knowledge and embed it into their scibertbased clinical abstractive summarization model. scaling such transformer models to longer input sizes has been difficult since the attention layers get exponentially larger and become computationally infeasible to train. recent advances in model architecture like pegasus @xcite and longformer @xcite have introduced different ways around this by introducing sparse attention mechanisms like local attention which replaces the full-attention mechanism with a sparse sliding window. researchers at google were able to innovate on these findings further by combining pre-training strategies from pegasus along with a new sparse attention mechanism called transient global which mimics etc's local/global attention mechanism and achieve state of the art performance on multiple summarization benchmarks. @xcite 3 data analysis ## experiments the original goal of this study was to experiment with two different approaches to the mslr22 shared task: to aid in the model development process and also as a validation that these summarization models have a practical use, we created an online tool that allows anyone to invoke the models for any 6 paper abstracts. the tool can be found at: https: //ben-yu.github.io/summarizer ## discussion unsurprisingly the pre-trained models were unable to exceed the dataset benchmarks on the shared task. one key failing came from our inability to configure target generation length using hugging-face's accelerated inference text2text generation api. on the ms^2 dataset our outputs only had an average sentence length of 1.1 and character count of 87.97, which significantly deviated from our target length of 2.74 sentences and 435.6 characters. this likely due to the out-of-the-box model not properly generalising over the entire pubmed dataset as the model was also only trained for about 3k steps and further training steps would have improved it's performance. the rouge-l scores were particularly indicative, scoring sometimes up to 50% worse than the benchmarks. increasing our model output length would have likely dramatically improved our rouge scores. our model didn't score that poorly in terms of a delta ei on the ms^2 dataset with only a 0.06 difference from the long- we also found that experimenting with language models and training these large language models can be extremely cost prohibitive and potentially inaccessible to hobbyists and novice machine learning practitioners. these models are getting increasingly large and can't be built unless one has access to sufficient gpu-computing or cloud resources. training these models can take upwards of 48 hours and there is no guarantee that your model is improving or converging at a reasonable rate. ## conclusion we weren't able to improve upon existing benchmarks for either the ms^2 or cochrane datasets. we did show there is a need for stronger summarization metrics that can capture different linguistic dimensions such as factual correctness and readability. the summaries from our pre-trained model were significantly shorter than the target summaries and often factually incorrect upon manual inspection, but this couldn't directly be inferred from our model scores outside of comparing it to task benchmarks.