Pathology Dynamics at B io L ay S umm: the trade-off between Readability, Relevance, and Factuality in Lay Summarization (2023)

Abstract: Lay summarization aims to simplify complex scientific information for non-expert audiences. This paper investigates the trade-off between readability and relevance in the lay summarization of long biomedical documents. We introduce a two-stage framework that attains the best readability metrics in the first subtask of BioLaySumm 2023, with 8.924 FleschKincaid Grade Level and 9.188 DaleChall Readability Score. However, this comes at the cost of reduced relevance and factuality, emphasizing the inherent challenges of balancing readability and content preservation in lay summarization. The first stage generates summaries using a large language model, such as BART with LSG attention. The second stage uses a zero-shot sentence simplification method to improve the readability of the summaries. In the second subtask, a hybrid dataset is employed to train a model capable of generating both lay summaries and abstracts. This approach achieves the best readability score and shares the top overall rank with other leading methods. Our study underscores the importance of developing effective methods for creating accessible lay summaries while maintaining information integrity. Future work will integrate simplification and summary generation within a joint optimization framework that generates high-quality lay summaries that effectively communicate scientific content to a broader audience.

## introduction the burgeoning volume of biomedical literature in recent years has posed significant challenges for researchers, healthcare professionals, and the general public in staying abreast of the wealth of information generated. the task of manually summarizing long-form documents has become increasingly impractical, requiring a disproportionate amount of effort and domain-specific knowledge @xcite @xcite @xcite . automatic text summarization, which seeks to distill source texts while retaining their core ideas, continues to be a demanding task, especially with long, content-rich documents laden with domainspecific complexities @xcite @xcite @xcite . as such, there is an urgent need to develop effective summarization techniques tailored for extensive biomedical documents to cater to diverse audiences @xcite @xcite . meanwhile, computational complexity persists as a major obstacle, with specialized hardware potentially paving the way for more energy-efficient implementations @xcite @xcite . as we continue to tackle these challenges, the field is poised for advancements that will fundamentally reshape how we interact with and benefit from the biomedical literature. lay summarization simplifies and distills complex scientific information into an accessible format for non-experts @xcite . it is vital for bridging the gap between specialized knowledge and the broader community. controlled summarization can further enhance the accessibility of biomedical research findings by ensuring that generated summaries are both informative and comprehensible through readability. readabilitycontrolled summarization can maximize the use of scientific knowledge and allow various stakeholders to make informed decisions in healthcare and research @xcite . generating lay summaries for long documents poses unique challenges due to the inherent complexity of the subject matter and the specialized language used in the original documents. balancing the simplification of language with the preservation of accurate and relevant information is crucial. reducing jargon and technical terminology may lead to the loss of essential details or the introduction of errors. the bionlp 2023 workshop at acl recently introduced a new shared task biolaysumm, focusing on lay summarization of biomedical research articles @xcite . it comprises two subtasks with distinct objectives. the first subtask aims to generate lay summaries from plos and elife articles, including the abstract @xcite , striving to maximize relevance and factuality metrics while minimizing readability score metrics. the second subtask focuses on readability-controlled summarization, which seeks to maximize relevance and factuality scores and generate both abstracts and lay summaries with readability levels comparable to the target summaries. this paper investigates techniques for generating lay summaries and readability-controlled abstracts for long biomedical documents. we focus on their effectiveness in maintaining relevance, factuality, and readability. the ultimate goal is to facilitate knowledge dissemination and empower diverse audiences to engage with complex scientific information @xcite . a multistep approach involving bidirectional and auto-regressive transformers (bart) @xcite and multilingual unsupervised sentence simplification (muss) @xcite obtains the highest readability scores in the first subtask of generating lay summaries at the cost of lower relevance and factuality. in the second subtask, an approach based on local, sparse and global (lsg) attention @xcite obtains the highest readability and joint highest overall scores. ## related work in recent years, biomedical document summarization has benefitted from advancements in deep learning and language models @xcite . @xcite investigated bart @xcite for summarization of randomized controlled trials. sotudeh et al. @xcite improved radiology report summarization by incorporating medical ontology into a sequence-tosequence model. domain-specific corpora using abstracts as summaries @xcite , has also contributed to the field. deyoung et al. @xcite examined the summarization of systematic reviews based on cited clinical trials. in contrast, guo et al. @xcite combined summarization and simplification to generate plain language summaries from abstracts of systematic reviews. ## data the shared task leverages data from two principal sources, the public library of science (plos) @xcite @xcite and elife @xcite . each of these datasets comprises biomedical research articles, paired with their technical abstracts and lay summaries written by experts. as discussed in the preceding section, the utility of each type of summary differs depending on the subtask. moreover, the lay summaries in each dataset exhibit several distinct characteristics; for more comprehensive details, readers are referred to @xcite @xcite . the plos dataset, the larger of the two, contains 24,773 instances designated for training and 1,376 instances for validation, applicable to both subtasks. in contrast, the elife dataset consists of 4,346 training instances and 241 validation instances. the test data for subtask 1 includes 142 articles each from plos and elife. meanwhile, the test data for subtask 2 comprises 142 plos articles, distinct from those used in subtask 1. it's important to note that the test sets allocated for both subtasks are distinct from those published in any of the cited papers, and are made accessible via the codalab pages dedicated to the shared task. this configuration fosters a rich and diverse dataset, providing comprehensive support for the objectives of the shared task. ## lay summarization the generation of lay summaries was executed through a two-stage method. prior to delving into this method, it's important to outline the baseline approach from which it sprang. this involved using a base bart model @xcite , pretrained on the cnn daily mail @xcite , with an input and output token length of 1024 1 . any articles exceeding 1024 tokens in length were truncated. this approach was 1 facebook/bart-large-cnn 2 ccdv/lsg-bart-base-4096 3 mrm8488/t5-base-finetuned-summarize-news the most successful of the three proposed methods when generating abstractive lay summaries, achieving the highest overall scores, primarily due to its superior relevance score, which is consistent with the findings of goldsack et al. @xcite . this approach is referred to as pd 1 (pathology dynamics) in tables 1 and 2 . however, this bart-based approach fell short in terms of readability scores. to remedy this, the twostage method was implemented. in the first stage, a large language model such as bart @xcite or lsg @xcite was trained to generate lay summaries based on the articles, which incorporated abstracts, containing the most important information in a concise format. this eliminated the need for longer input token lengths required in subtask 2, which did not include the abstract in the input. in the second stage, the two-stage approach used a zero-shot sentence simplification method called muss @xcite to enhance the readability of the generated lay summaries. this approach, referred to as pd 2 in tables 1 and 2 , achieved the best readability score among all the participating teams in the subtask, attesting to the effectiveness of integrating muss @xcite into the process. despite this success, figure 1 indicates certain limitations with muss @xcite . while readability was improved, certain aspects of clarity were compromised, such as the detachment of the abbreviation opc from its full form, oropharyngeal candidiasis. as a result, the summary's relevance and factuality scores dipped below that of the base model, emphasizing the need for a careful balance between simplification techniques and the maintenance of essential information. the final method involved the use of an lsg attention model @xcite with an increased input token length of 4096 for the encoder to generate lay summaries, while keeping upon manual review and metric evaluation using the validation set, it is noted that the output from stage 2 diverges from the target lay summary. as indicated by the arrows, the output from stage 1 therefore acts as both the lay summary and the abstract for the purposes of evaluation. the decoder's output token length at 1024 2 . this approach, denoted as pd 3 in ## readability controlled summarization this subtask was designed to yield lay summaries and abstracts from other sections of the article using a singular model. in order to address this intricate task, we initiated our methodology with a pre-processing phase designed to create a synthetic dataset. this was achieved by duplicating each article and correlating it with the corresponding abstract and lay summary as outputs. as a result, this tactic effectively doubled the number of articles available for training in our dataset, representing a significant enhancement over the volume offered by the original dataset. following the augmentation of our dataset, we utilized this expanded synthetic dataset to train a model based on lsg @xcite . this particular model was adept at generating a summary that seamlessly integrated components from both the lay summary and the article. one notable characteristic of this subtask was the absence of an abstract in the input articles. this invariably meant that crucial information was scattered throughout the entirety of the article, thereby calling for a greater input token length. to account for this, we engaged an lsg-based model @xcite , specifically configured with an encoder that accommodated 4096 input tokens and a decoder that handled 1024 output tokens 2 . this specialized setup surpassed the performance metrics exhibited by t5 3 (raffel and 4 . moving forward, we incorporated the two-stage modeling approach, as described in the previous task, into this current subtask. in the initial stage, the lsg-based model @xcite was tasked with generating a summary intended to act as the abstract. the subsequent stage hinged on the use of muss @xcite to produce the lay summary. however, an in-depth inspection of the generated summaries coupled with an analysis of the validation set scores led us to the realization that the output from the first stage of the model served as a more effective lay summary than that produced by muss. we found that the muss output was excessively simplified when juxtaposed against the target lay summary. due to this observation, we opted for the utilization of the output from the first stage to fulfill the dual roles of both the abstract and the lay summary. the objective of this task was to achieve readability scores that more closely aligned with the target values, as opposed to pursuing the lowest possible scores. given this context, the application of muss resulted in a decline in readability scores for this subtask. figure 2 presents the opening sentences of the outputs from the first and second stages for a representative article. this visual comparison highlights the propensity of muss @xcite to oversimplify the summary, causing it to diverge from the intended lay summary. as a result, the utilization of the same output from the first stage for both the abstract and lay summary culminated in superior scores, even in terms of readability. this is because the readability score for this subtask involved a comparison between the generated lay summary and the original version. this points to the inherent challenge of striking a balance between readability and the retention of crucial content when employing simplification strategies for the generation of lay summaries. ## conclusion and future work in conclusion, this paper explored the intricate trade-offs between readability, relevance, and factuality in lay summarization. we have highlighted the inherent challenges associated with transforming complex scientific information into an accessible format for non-expert audiences. our proposed two-stage framework attains state-of-the-art readability metrics; however, this is achieved at the cost of reduced relevance and factuality. these findings emphasize the necessity of striking a balance between readability and content preservation when creating lay summaries. future work will focus on integrating simplification and summary generation within a joint optimization framework. this approach aims to overcome the trade-offs identified in our study and enable the generation of high-quality lay summaries without sacrificing readability, relevance, or factuality. more effective lay summarization methods can be developed by considering both simplification and summarization as complementary processes.