GRASUM at B io L ay S umm Task 1: Background Knowledge Grounding for Readable, Relevant, and Factual Biomedical Lay Summaries (2023)

Abstract: Communication of scientific findings to the public is important for keeping non-experts informed of developments such as life-saving medical treatments. However, generating readable lay summaries from scientific documents is challenging, and currently, these summaries suffer from critical factual errors. One popular intervention for improving factuality is using additional external knowledge to provide factual grounding. However, it is unclear how these grounding sources should be retrieved, selected, or integrated, and how supplementary grounding documents might affect the readability or relevance of the generated summaries. We develop a simple method for selecting grounding sources and integrating them with source documents. We then use the BioLaySum summarization dataset to evaluate the effects of different grounding sources on summary quality. We found that grounding source documents improves the relevance and readability of lay summaries but does not improve factuality of lay summaries. This continues to be true in zero-shot summarization settings where we hypothesized that grounding might be even more important for factual lay summaries.

## introduction automatic lay summarization of biomedical research is a promising approach to help inform non-experts of vital scientific and clinical discoveries. however, known issues with factuality in automatic summarization systems @xcite are still a barrier that prevents their safe deployment. to improve factuality, some have suggested using grounding sources with retrieval augmentation @xcite @xcite . this has been show to help maintain factuality in biomedical nlp tasks @xcite without harming readability or relevance. additionally, @xcite suggests that retrieval augmentation is especially helpful for lay summarization because those summaries need to provide necessary background knowledge (see figure 1 ) such as definitions which are not often found in the source text. in this paper we wanted to understand (i) how we might develop a retrieval augmentation solution for lay summarization when using whole scientific papers for models with limited input context length and (ii) what is the effect of different grounding sources that contain different types of background knowledge on readability, relevancy, and factuality. contributions: we develop (i) a simple method for selecting and using grounding sources for lay summarization. we assess this method using the biolaysumm @xcite biomedical paper lay summarization dataset 2 and find that (ii) grounding has the largest effect on readability where in the zero-shot summarization setting definitional background knowledge from unified medical language system (umls) and simplified encyclopedic background knowledge from wikipedia simple provide better readability scores. contrary to popular opinion, we found that (iii) grounding does not improve factuality. ## method summarizing scientific papers requires more input tokens than a large language model can typically support due to memory constraints. the average token count for articles in biolaysumm are 8,963 tokens for plos and 13,942 tokens for elife with articles up to 45,563 tokens. since we want to explore the effect of grounding articles with additional retrieved sources there is an even greater need for large token input support. because of these factors, the base model we use in our experiments is the longformer encoder-decoder (led) @xcite which supports an input token length of 16,384 tokens (see appendix b for additional training and inference details). our method was designed to test the effect of different grounding sources on downstream summarization quality. in addition to definitional background knowledge from umls and encyclopedic background knowledge from wikipedia which were used in @xcite , we introduced two other retrieval sources, wikipedia simple for access to encyclopedic background knowledge in simpler terms and scientific abstracts for access to further contextual background knowledge that might emulate the additional supplementary knowledge an expert has when crafting a lay summary. in all, we tested the following four grounding sources (1) umls named entity definitions (2) scientific abstracts (from crossref) (3) wikipedia (english) (4) wikipedia simple (english). see appendix c for a full description of these grounding sources and how they were used. our retrieval augmentation consisted of two steps: (i) retrieving and (ii) re-ranking documents, first we took each sentence in the leading 1,024 tokens of the article (roughly corresponding to the abstract) and searched them using bm25 on indexes constructed for each grounding source (except umls which uses another method discussed). indexes were constructed using pyserini @xcite . the top 1 most relevant passage is selected and then added to a pool of candidate passages. in the case of scientific abstracts, we remove the abstract of the document we are enhancing if it was in the pool. in the case of umls, we follow @xcite by using the scispacy entity linker (neumann et al., 2019) over the first 1,024 tokens and provide definitions for the umls named entities as the pool of candidate passages. the above procedure results in too many results to fit within context length. in order to resolve this, we rank the pool of candidates passages against the first 1,024 tokens of the using a crossencoder (reimers and gurevych, 2019) (see appendix b for details). finally we construct our inputs by selecting the first 8,192 tokens of the original document and the top relevant grounding passages up to 8,192 tokens. @xmath0. we also supplement all grounding sources with a bibliographic reference string containing the title, authors, and year of the paper being summarised. this was motivated by seeing many ground truth summaries which cited the source document by first authors name (for example: "parks et al. analyzed data on us deaths between 1980 and 2016" which is the first sample in the elife training subset of biolaysumm). ## discussion grounding the original document in retrieval results helped most with readability, marginally with relevance and not at all with factuality. we believe this gives credence to the idea that grounding for lay summarization is primarily helpful for providing the model with background information that helps explain concepts, define terms, and otherwise situate the reader with the necessary information to be able to understand a scientific finding. we saw that the umls and wikipedia simple sources provided best effect on improving readability in the zero-shot summarization setting which is intuitive in that they provide clear definitions and encyclopedic background knowledge in simple terms. scientific abstracts had the best effect of all sources on relevancy which is also intuitive because related abstracts such as ones with similar findings might help construct a more robust summary. these results indicate that we should continue to investigate the role of background information on improving readability and relevance in lay summarization. in particular, our retrieval method is quite simple and more sophisticated methods of retrieval such as dense passage retrieval @xcite could be used to enhance the relevancy of grounding documents. additionally, future work should investigate methods that learn more strongly to take adavantage of grounding sources. the lack of improvement in factuality is an important aspect that future work should investigate. one explanation is that grounding sources could introduce factual or relevancy errors if the retrieved documents are irrelevant or incorrect and they end up being used in the generated summary. however, factuality metrics only measure the summary against original document, this means that statements that cannot be grounded in the original document may be penalized. this is an issue in lay summarization where there can be statements that are factual and provide necessary background knowledge but cannot be found in the original document. future work should investigate methods of measuring factuality that are able to incorporate the necessary background knowledge when measuring the factuality of a lay summary. ## related work there are a number of works looking at automatic lay summarization in the biomedical domain @xcite ; @xcite ; @xcite ; @xcite . one central issue is understanding the effect of text simplification on various aspects of summary quality such as relevancy. @xcite explores the effect of text simplification on factuality by introducing a taxonomy of different error types that allows them to observe that while factual errors of missing information are a common error across generated and gold summaries, errors of substitution such as mixing up entities is a common occurrence in text simplification summarization models. supplementing source documents with external knowledge has been one of the main interventions discussed for mitigating factual errors and hallucinations in natural language generation systems @xcite . with the idea that having access to grounding sources allows models to draw on those sources when generating text rather than being forced to rely on parametric knowledge which may be flawed @xcite . in summarization, researchers have used factual knowledge from external sources to improve factuality by encoding external knowledge in models during training @xcite or to correct already generated summaries @xcite . @xcite evaluates retrieval augmentation as a method for enhancing abstracts with background knowledge. they use definitions from umls and wikipedia as different retrieval corpora and find their method improves both the readability and relevancy of summaries while maintaining a similar level of factuality as models not using grounding. however, they did not perform retrievalaugmented generation for the lay summary generation task which is the novel contribution in this work.