Team YXZ at B io L ay S umm: Adapting Large Language Models for Biomedical Lay Summarization (2024)

Abstract: Biomedical literature are crucial for disseminating new scientific findings. However, the complexity of these research articles often leads to misinterpretations by the public. To address this urgent issue, we participated in the BioLaySumm task at the 2024 ACL BioNLP workshop, which focuses on automatically simplifying technical biomedical articles for non-technical audiences. We conduct a systematic evaluation of the SOTA large language models (LLMs) in 2024 and found that LLMs can generally achieve better readability scores than smaller models like Bart. Then we iteratively developed techniques of title infusing, K-shot prompting , LLM rewriting and instruction finetuning to further boost readability while balancing factuality and relevance. Notably, our submission achieved the first place in readability at the workshop, and among the top-3 teams with the highest readability scores, we have the best overall rank. Here, we present our experiments and findings on how to effectively adapt LLMs for automatic lay summarization. Our code is available at https://github.com/zhoujieli/biolaysumm.

## introduction the biomedical literature is one of the most important information sources for researchers to share their latest discoveries. however, the increasing volume of information has become overwhelming, e.g. pubmed alone hosts over 36 million papers, with more than one million new articles added annually @xcite . this information deluge makes it challenging for even specialized biomedical experts to keep up with the latest research, let alone the general public. general public, although having a keen interest in biomedical research due to its relevance to everyday life, may be prohibited by the difficult biomedical terminology, experimental setups, or the metric abbreviations. currently, media outlets play a crucial role in bridging the gap between scientific literature and public understanding @xcite . however, media often lack the necessary context when reporting new studies, which can lead to exaggerated claims. for instance, reports may claim that certain diets promote longevity foot_0 , but closer inspection of the literature reveals that these claims are typically based on preliminary animal studies and lack robust support in human studies @xcite @xcite . in recent years, with the rapid advancements of transformer-based natural language processing algorithms, many intelligent systems have been developed to automate the process of explaining and summarizing research papers to lay people. @xcite collected 100,000 webpages from science daily, a popular press release websites for research papers, and finetuned a bert model to produce automatic science journalism; @xcite developed a hierarchical disource-aware attention model to effectively summarize the long and hierarchical research paper. interactive research paper summarization systems like scisummary foot_1 , scholarcy foot_2 and scispace foot_3 can help researchers quickly get the gist from the literature and develop a map of connected research @xcite . in particular, biomedical literature contains many domain jargons which lack readable explanations; background stories are frequently omitted, and findings are hidden in obscure metrics names without direct discussions. in essense, the literature cannot "talk" by itself and answer general public's questions. to advance the research in automatic lay summary generation for biomedical literature, the acl bionlp task of biolaysumm was started in 2023 @xcite . accompanying this task, a lay summary dataset was curated to facilitate model training and evaluation @xcite . the dataset contains 31,020 articlesummary pairs 5 from two journal series, public library of science (plos) and elife. each articlesummary pair contains the paper full text and the corresponding lay summaries written by paper authors (plos) or journal editors (elife). due to the different sources of lay summaries, the characteritics of these summaries vary, e.g. plos lay summaries are on average 175.6 words and elife summaries are on average 347.6 words 6 . the task of biolaysumm is then given the full article text, a system should automatically output a lay summary, which will be measured in 10 metrics of factuality, readability and relevance. in the previous iteration of biolaysumm, we observe that the best performing teams used advanced large language models (llms) like gpt-3.5 to produce zero-shot lay summaries @xcite and augment training data @xcite . in addition, we have frequently used tools like gpt4, kimi.ai, and chatdoc.com to facilitate rapid understanding of research papers. in this work, we make several key contributions. firstly, we conduct a comprehensive set of experiments demonstrating that while directly prompting large language models (llms) can enhance the readability of lay summaries, it often compromises their factuality and relevance. secondly, we have developed a suite of adaptation techniques, including title infusion, k-shot prompting, llm rewriting, and instruction fine-tuning, which enable llms to generate lay summaries that are well-balanced in terms of factuality, relevance, and readability. notably, our approach achieved first place in readability in the 2024 biolaysumm competition, while also maintaining a strong balance across the other evaluated metrics. ## dataset and evaluation metrics the dataset @xcite of 31,020 article-summary pairs is divided into three parts, train, validation and test, where the train and validation sets are given for model development, and the lay summaries of the test set are hidden. the distribution of the data is shown in relevance: rouge or recall-oriented understudy for gisting evaluation (1, 2, and l) @xcite and bertscore @xcite are the relevance metrics which uses lexical or embedding based methods to measure the overlap between the generated summaries to the reference ones. the higher the scores the more relevant the summaries. readability: flesch-kincaid grade level (fkgl) @xcite , dale-chall readability score (dcrs)(chall and dale, 1995), coleman-liau index (cli) @xcite are lightweight readability metrics that predict the us grade level of education needed to understand the generated summaries, so the lower the scores, the more readable the summaries. whereas lens (learnable evaluation metric for simplification) @xcite , a readability metric trained on human judgment data, aligns more closely with human preferences, the goal is to achieve a higher lens score. factuality: alignscore @xcite is an automatic factual consistency metric for checking whether all information in the summary is contained in the reference. similarly, summac (laban et al., 2022) or summary consistency is a natural language inference (nli) method that measures the factual consistency between generated summary and reference sentence-wise. the goal is to maximize these two factuality metrics. ## method and results following previous work @xcite , we mainly focus on using llms in this work. in our settings, the input is the article text and relevant metadata, and different llms are prompted along with a system prompt to generate the lay summary. in this section, we explain our methods and the results in detail. ## results and conclusion we constructed our final submission based a combination of three techniques: title infusing, instruction finetuning and k-shot prompting. overall, our submission achieved the 1st place in the readability catorgy, and had the better overall score compared with the top-3 team in the readability category (29th vs. 36th and 40th) in this work, through systematic experiments on the capabilities and limitatons of the sota large language models, we developed strategies to adapt llms for the task of biolaysumm, and achieved the best result in readability while balancing other metrics. we provide experiment details and findings for researchers to keep advancing in this field.