F a M e S umm: Investigating and Improving Faithfulness of Medical Summarization (2023)

Abstract: Summaries of medical text shall be faithful by being consistent and factual with source inputs, which is an important but understudied topic for safety and efficiency in healthcare. In this paper, we investigate and improve faithfulness in summarization on a broad range of medical summarization tasks. Our investigation reveals that current summarization models often produce unfaithful outputs for medical input text. We then introduce FaMeSumm, a framework to improve faithfulness by fine-tuning pre-trained language models based on medical knowledge. FaMeSumm performs contrastive learning on designed sets of faithful and unfaithful summaries, and it incorporates medical terms and their contexts to encourage faithful generation of medical terms. We conduct comprehensive experiments on three datasets in two languages: health question and radiology report summarization datasets in English, and a patient-doctor dialogue dataset in Chinese. Results demonstrate that FaMeSumm is flexible and effective by delivering consistent improvements over mainstream language models such as BART, T5, mT5, and PEGASUS, yielding state-of-the-art performances on metrics for faithfulness and general quality. Human evaluation by doctors also shows that FaMeSumm generates more faithful outputs. Our code is available at https://github.com/psunlpgroup/FaMeSumm.

## introduction summarizing medical text is a key step towards improving the efficiency in healthcare @xcite , including applications in summarizing medical dialogues @xcite , clinical notes @xcite , health questions @xcite , and radiology reports @xcite . an important but understudied issue of medical text summarization is faithfulness, as defined in recent literature @xcite : a summary is unfaithful if it contains intrinsic error (the fact that contradicts the source) or extrinsic error (the fact that cannot be directly inferred from the source text), while a faithful summary should be free from both. unfaithful summaries will pose significant healthcare risks by misleading patients and medical providers. however, to the best of our knowledge, only a few papers investigate the faithfulness of medical summarization by providing systematic categorization of errors for medical domain @xcite . additionally, recent investigation and faithfulness improvement @xcite works are limited to certain types of medical text such as radiology reports. moreover, as many summarization approaches are based on language models that are pretrained on general domain text, they are inadequate to generate faithful summaries due to the lack of medical domain knowledge. for example, although @xcite and @xcite leveraged pretrained models such as t5 @xcite and bart @xcite , their performance is suboptimal because they did not incorporate medical knowledge into their models. in this work, we aim to both investigate and improve the faithfulness of medical summarization. we first provide a taxonomy of faithfulness error types appearing in medical summaries and analyze several competitive baselines. our investigation in ## investigating faithfulness of medical summarization comprehensive error analysis can improve the understanding of the severity of faithfulness issues, and help researchers better design a model for improving faithfulness. however, little research has studied the faithfulness issues in medical summarization. therefore, we provide a systematic taxonomy of faithfulness errors. specifically, we first randomly pick 100 samples from hqs, rrs, and mds datasets (section 4.1), respectively, forming 300 samples in total. we use fine-tuned transformer models and the pointer-generator network as baselines to generate summaries for each sample, and we manually check and categorize the faithfulness errors of the summaries. inspired by @xcite , we first classify the faithfulness errors found in the 300 samples into intrinsic errors and extrinsic errors. then, we further cluster the error cases in a finer granularity based on their main characteristics: (1) the intrinsic error includes entity error, entity relationship error, negation error, question error, and modifier error, and (2) the extrinsic error contains template error, low specificity error, and extraneous fact error. complementing the error types in @xcite with new types that are specific to the medical domain (e.g., low specificity error), ## improving faithfulness of medical summarization we propose famesumm, a faithfulness optimization framework guided by medical domain knowledge for pretrained language models. as shown in figure 1 , the first objective l cl uses contrastive learning (cl) over a set of curated faithful and unfaithful summaries created by perturbing medical entities and concepts (section 3.1); the second objective l mki explicitly performs medical knowledge incorporation (mki) and encourages generating accurate medical terms from source texts (section 3.2). these two strategies complement each other during the fine-tuning stage in addition to the cross-entropy loss l ce (section 3.3). famesumm is flexible because it is widely applicable to various backbone pretrained language models. ## results and analysis all performance scores in this section are evaluated on the test set of each dataset. our results and analysis aim to answer the following research questions: ## related work medical summarization there are different medical summarization datasets on medical dialogues @xcite @xcite @xcite and multidocuments @xcite . some of them are private. mediqa 2021 shared tasks @xcite tackle three summarization tasks: consumer health question summarization (hqs), multianswer summarization (mas), and radiology report summarization (rrs). as for medical summarization models, @xcite and @xcite constructed pointer-generator networks, so they did not benefit from the success of pre-trained transformer models. @xcite fine-tuned bart and developed a multistage approach to handle long conversations. the length of conversations is not the concern of our work. the focus of @xcite is to generate soap notes (subjective information, objective observation, assessments made by doctors, and plan for future care), and their work aims to have summaries divided into at most 15 sections. by contrast, famesumm serves as a general-purpose fine-tuning strategy for abstractive summarization in healthcare with wide applicability. ## conclusion in this paper, we investigate faithfulness issues on abstractive medical summarization in current summarization models. we propose famesumm consisting of two fine-tuning objectives that can be applied to mainstream pretrained language models across many datasets. our results based on automatic metrics and doctor evaluation show that our method mitigates the faithfulness concern in medical summarization and delivers state-of-the-art performances on several benchmarks.