Ctyun AI at B io L ay S umm: Enhancing Lay Summaries of Biomedical Articles Through Large Language Models and Data Augmentation (2024)

Abstract: Lay summaries play a crucial role in making scientific research accessible to a wider audience. However, generating lay summaries from lengthy articles poses significant challenges. We consider two approaches to address this issue: Hard Truncation, which preserves the most informative initial portion of the article, and Text Chunking, which segments articles into smaller, manageable chunks. Our workflow encompasses data preprocessing, augmentation, prompt engineering, and fine-tuning large language models. We explore the influence of pretrained model selection, inference prompt design, and hyperparameter tuning on summarization performance. Our methods demonstrate effectiveness in generating high-quality, informative lay summaries, achieving the second-best performance in the BioLaySumm shared task at BioNLP 2024.

## introduction biomedical publications serve as a critical channel for disseminating cutting-edge research findings on a wide range of health-related topics. while biomedical publications are essential for advancing medical knowledge and public health awareness, the technical terminology and lack of backgroud information often render them inaccessible to non-expert audiences @xcite . the biolaysumm shared task addresses this need by developing effective models to generate lay summaries of biomedical articles aimed at non-expert audiences @xcite . the challenge in the biolaysumm shared task is to distill complex biomedical content into lay summaries that are both comprehensible and engaging to non-expert audiences. large language models (llms) have shown remarkable capabilities in generating coherent and contextually accurate texts @xcite , which could refor-plos lay summary 17 674 268 270 article 1,046 37,770 10,289 10, 029 to tackle the challenge of lengthy articles in the biolaysumm shared task, we consider two approaches: hard truncation and text chunking. we preprocess the data using these methods, apply data augmentation and prompt engineering, and finetune large language models on the task-specific data. we explore the effect of pretrained models, inference prompts, and hyperparameters on the quality of the generated lay summaries. our experiments show that our approach effectively extracts key information and produces informative, easy-to-understand summaries. ## metrics to thoroughly evaluate the quality of the generated lay summaries, we use a diverse set of metrics that capture various aspects of the summarization task: relevance: we use rouge (1, 2, and l) @xcite and bertscore @xcite to evaluate the relevance of the generated summaries to the original articles. higher scores indicate better performance for these metrics. readability: to assess the readability of the generated summaries, we utilize several widely-used metrics: flesch-kincaid grade level (fkgl) @xcite , dale-chall readability score (dcrs) (chall and dale, 1995), coleman-liau index (cli) @xcite , and lens @xcite . for fkgl, dcrs, and cli, lower scores indicate better readability, while for lens, higher scores are preferable. factuality: ensuring the factual correctness of the generated summaries is crucial in the biomedical domain. we employ alignscore @xcite and summac @xcite to measure the factual consistency between the generated summaries and the source articles. higher scores on these metrics indicate better factual alignment. ## experiments we conduct a series of experiments to investigate the impact of various factors on our lay summarization model's performance. due to the plos validation set's size, we use the first 142 entries as our validation subset. ## discussion this paper introduces two methods for handling long input sequences in the biolaysumm task and future research could explore larger pretrained models and more sophisticated strategies for handling lengthy inputs. section-specific summarization techniques could also improve performance. carefully designing inference prompts and selecting appropriate hyperparameters are crucial when fine-tuning pretrained language models for specific tasks. we hope our work inspires further research and contributes to developing effective tools for making scientific knowledge more accessible. ## limitation in this study, we conducted a comprehensive analysis of various factors influencing model performance, including pre-trained models, hyperparameters, and data processing techniques. our investigation, however, did not extend to examining the differential impact of distinct article sections on summary generation. this aspect warrants further exploration, as the introduction and conclusion sections often encapsulate the core content of an article and may hold greater significance for summarization, while body sections typically provide more granular details. additionally, to enhance the model's proficiency in specialized biological domains, future work could investigate the efficacy of incremental pretraining. this approach may potentially improve the model's ability to elucidate technical terminology in more accessible language, thereby enhancing the overall quality and comprehensibility of generated summaries. these unexplored avenues present promising directions for future research, aimed at refining and advancing the performance of summarization models in specialized scientific domains, particularly in the field of biology.