Understanding Faithfulness and Reasoning of Large Language Models on Plain Biomedical Summaries (2024)

Abstract: Generating plain biomedical summaries with Large Language Models (LLMs) can enhance the accessibility of biomedical knowledge to the public. However, how faithful the generated summaries are remains an open yet critical question. To address this, we propose FaReBio, a benchmark dataset with expert-annotated Faithfulness and Reasoning on plain Biomedical Summaries. This dataset consists of 175 plain summaries ($,445 sentences) generated by seven different LLMs, paired with source articles. Using our dataset, we identify the performance gap of LLMs in generating faithful plain biomedical summaries and observe a negative correlation between abstractiveness and faithfulness. We also show that current faithfulness evaluation metrics do not work well in the biomedical domain and confirm the over-confident tendency of LLMs as faithfulness evaluators. To better understand the faithfulness judgements, we further benchmark LLMs in retrieving supporting evidence and show the gap of LLMs in reasoning faithfulness evaluation at different abstractiveness levels. Going beyond the binary faithfulness labels, coupled with the annotation of supporting sentences, our dataset could further contribute to the understanding of faithfulness evaluation and reasoning.

## introduction generating plain text summaries-summarising technical articles in plain language-helps facilitate public access to biomedical knowledge and has been an important topic in the biomedical domain @xcite @xcite . despite the overall performance achieved by llms @xcite @xcite , the faithfulness, which is to what extent the generated text is consistent with the source articles, and factuality, that is to what extent the generated 1 dataset available at https://data.csiro.au/ . text is consistent with the external knowledge, have been known problems of llms @xcite and have not been well-studied in the biomedical domain @xcite . apart from labor-intensive and costly manual examination, prior work @xcite @xcite has proposed various metrics to automatically evaluate the faithfulness of the generated text. however, these metrics are mainly designed to measure faithfulness in the general domain, such as news. to what extent it can be used in domain-specific areas, such as the biomedical domain, remains an open question @xcite . additionally, current research @xcite has shown that, although llm-based evaluators achieve promising alignments with human judgment, they do not always provide correct reasoning for their decisions, e.g., correct rationales or supporting sentences from text. examining to what extent llms can provide correct reasoning for their decisions could help better understand the reasoning behind llms, especially in the biomedi-cal domain where accurate evidence is paramount. to address these problems, we propose a benchmark dataset, farebio, on evaluating faithfulness and reasoning of llms on plain biomedical summaries in section 3. specifically, as shown in figure 1 , we enlist medical doctors to manually evaluate the faithfulness of plain summaries from seven representative llm-based summarisation systems and highlight the corresponding supporting sentences from source articles. in section 4, we answer the four major research questions on the faithfulness evaluation of llms on plain biomedical summaries: (1) how faithful are generated summaries across current llms? (2) how abstractive and readable are plain biomedical summaries, and how do they correlate with faithfulness? (3) how do current automatic faithfulness evaluators align with human judgment? (4) do llms consider their own generation more faithful than other llms in zero-shot evaluation? we further evaluate capability of llms in providing correct reasoning for their judgment in section 5. specifically, we focus on the reasoning as extracting the supporting sentences and address three major research questions: (5) can llms identify the supporting sentences from the source article when they are instructed to evaluate the faithfulness of generate summaries? (6) how does the abstractiveness of the summary impact the identification of supporting sentences? (7) do llms perform better when identifying supporting sentences for their own generated summaries? to the best of our knowledge, our study is the first publicly available benchmark dataset investigating faithfulness with the identification of supporting sentences for plain biomedical summaries. we find that generated summaries demonstrate a high degree of non-faithful hallucination and the level of abstractiveness shows a negative correlation with the faithfulness of plain summaries. current faithfulness evaluators tailored for the general domain do not directly transfer well to the biomedical domain. we also observe a tendency where llms, as evaluators, favour their generation when evaluating faithfulness. however, the construction of the prompt could also impact such a tendency. additionally, we identify the performance gap of llms in retrieving supporting sentences from source articles, either for high or low abstractiveness summaries. we hope our dataset would enable further studies on understanding the faithfulness evaluation of llms as well as its rea-soning. ## related work faithfulness, where the generated summary is consistent with the source @xcite , has been a known challenge in text generation @xcite . current faithfulness research on llms mostly focuses on the general domain, with a particular interest in news articles @xcite @xcite . some studies evaluated faithfulness and factuality-factual consistency with the source and external knowledge, respectively-in the biomedical domain. for instance, @xcite measured the factuality of zero-shot summaries from gpt-3.5 @xcite and flan-t5-xl @xcite . @xcite was proposed to measure the factuality of gpt-4 @xcite , llama-2-chat @xcite , and alpaca @xcite under the pico framework @xcite . @xcite proposed an evaluation framework for plain language summarisation and measured the faithfulness of gpt-based text simplification on biomedical abstracts. current research has proposed various metrics based on different frameworks to evaluate the faithfulness of generated text for the general domain: apart from solely evaluating the binary faithfulness label of the generated summary, a natural extension is to provide the reasoning for the judgment, e.g., supporting sentences from the source. for faithfulness reasoning, the fever dataset @xcite annotated the factuality of claims based on wikipedia articles and provided extracted facts from corresponding sources. wadden et al. (2020) created scifact, a dataset of 1.4k expert-written scientific claims paired with the abstracts from s2orc @xcite , annotating with labels and rationales. similarly, factpico @xcite ) contains expert-written rationales for factuality annotations. @xcite proposed a shared task in identifying all grounding context from the scholarly paper discussing methodological details in the claim. 2 3 dataset creation ## supporting sentence identification in section 4, we investigate the capability of llms in predicting faithful sentences. one following question is whether the models have correct reasoning to support their judgment. in our annotation, we ask the annotators to highlight the supporting evidence from the source article. this enables us to understand if the llms can identify the evidence from scientific literature. specifically, we consider a subset of our dataset where sentences are labeled as faithful and supporting evidence is provided. the statistic of this subset is shown in for the baseline, we consider okapi bm25 @xcite , a ranking model based on the term and document frequency. we select the most relevant sentence from the document as the supporting evidence. 13 it is worth mentioning that, as discussed in section 3.3, annotators might overlook the supporting evidence. although we provide precision, recall, and f1 on extracted sentence matching, we focus on recall, i.e., the coverage of the annotated sentence, for our analysis, to understand if models can retrieve comprehensive evidence from the source. rq5. can llms identify the supporting sentences from the source article? we further perform an error analysis on the extracted supporting evidence from llms. we randomly sample 50 summaries from our dataset. as shown in rq6. does abstractiveness impact the identification of supporting evidence? as shown in figure 3a , summaries from different llms demonstrate different levels of abstractiveness. high abstractive sentences might require a deeper understanding of the text in order to identify the supporting sentences. to study the impact, we further separate the generated summaries into two groups: (1) high abstractiveness, i.e., gpt-4, claude-3, gemini-1.5, and llama-3; and (2) low abstractiveness, i.e., flan-t5, finetuned-llama-3, and finetuned-flan-t5. rq7. do llms perform better when extracting evidence for their generated summaries? that is, assuming llms generate summaries based on their reasoning, would llms be able to retrieve evidence for their generation, as they might follow a similar reasoning process? we plot the heatmap of llms' retrieval performance across different subsets of summaries generated by different llms in figure 5 . overall, gemini-1.5 archives higher precision across all summary subsets (figure 5a , third column). gpt-4 consistently achieves the highest recall among the generated summaries (figure 5b , first column). among the generated text, llms achieve higher precision based on the summaries from gpt-4 (figure 5a , first row) and higher recall from claude-3 (figure 5b , second row). however, we do not observe that the model outperforms the others when identifying the supporting evidence from its generation, i.e., llms do not necessarily outperform other models when reasoning its generation over others. the retrieval performance might be affected by other factors, e.g., abstractiveness. ## conclusions we create a benchmark dataset, farebio, with expert-annotated faithfulness evaluation and reasoning for plain biomedical summaries, consisting of 175 summaries and 1445 sentences from 7 different llms. we use this dataset to evaluate the faithfulness of prevalent llms and measure the transferability of current faithfulness metrics to the biomedical domain. we observe a positive correlation between abstractiveness and nonfaithful hallucination and find that the construction of prompts could also affect faithfulness prediction preferences. we further benchmark the capability of llms in retrieving supporting sentences for the plain summaries at different levels of abstractiveness. by going beyond the binary faithfulness labels, equipped with annotations of faithfulness and reasoning, our dataset could further deepen the study of faithfulness in better understanding the reasoning behind llms for their faithfulness judgment. we expect our research would enable further studies on understanding the faithfulness evaluation of llms in its reasoning.