Learning to Revise References for Faithful Summarization (2022)

Abstract: In real-world scenarios with naturally occurring datasets, reference summaries are noisy and may contain information that cannot be inferred from the source text. On large news corpora, removing low quality samples has been shown to reduce model hallucinations. Yet, for smaller, and/or noisier corpora, filtering is detrimental to performance. To improve reference quality while retaining all data, we propose a new approach: to selectively re-write unsupported reference sentences to better reflect source data. We automatically generate a synthetic dataset of positive and negative revisions by corrupting supported sentences and learn to revise reference sentences with contrastive learning. The intensity of revisions is treated as a controllable attribute so that, at inference, diverse candidates can be over-generated-then-rescored to balance faithfulness and abstraction. To test our methods, we extract noisy references from publicly available MIMIC-III discharge summaries for the task of hospital-course summarization, and vary the data on which models are trained. According to metrics and human evaluation, models trained on revised clinical references are much more faithful, informative, and fluent than models trained on original or filtered data.

## introduction the tendency of abstractive systems to produce unfaithful summaries is well-studied @xcite , yet less attention is paid to the role of the data on which the models are trained. this is problematic for two reasons: (1) many corpora are naturally occurring-not created for training modelsand, as such, are noisy @xcite and without "inherent quality guarantees" @xcite ; (2) noisy data is detrimental to training faithful models @xcite . starting steroid taper.. a. copd flare...s when she arrived on the floor, she was on bipap and had a cough. ## data the publicly available mimic-iii dataset contains de-identified clinical records from patients admitted to beth israel deaconess medical center @xcite . hospital-admission summarization is a challenging task: in at most a paragraph, a summary must discuss what happened to the patient during the admission, why it happened, and what needs to happen next. to create a dataset for there has been interval removal of the right chest tube. unsupported entities bertscore precision 1/2 aligned source context 88.0 reference 41 year old woman with recent pregnancy, s p stillbirth. ## building source-reference alignments we link each reference sentence to a subset of source sentences to identify the minimal context necessary for revision and determine which sentences need to be revised. we select no more than five sentences because @xcite find that reference sentences tend to reflect content from very few source sentences. we follow their approach to greedily select sentences with high rouge @xcite overlap and minimize redundancy by removing covered tokens after each step. yet, given high levels of abstraction (abbreviations @xcite , misspellings), we increase semantic coverage by replacing rouge with bertscore precision @xcite and adding additional sentences, as needed, to cover all supported entities. please refer to appendix b for intuition, notation, and an example. classifying references. we treat reference sentences with 0 unsupported entities and a bertscore precision with respect with its aligned source evidence @xmath0.75 as supported. the remaining are unsupported. 417,318 (41%) reference sentences qualify as supported and the remaining 595,300 (59%) unsupported: 47% fail both thresholds (280,839), 38% have hallucination(s) with a high bertscore (225,423), and 15% have no hallucinations but a low bertscore (88,189). figure 2 reveals why both bertscore and entity overlap are needed to identify full coverage. the first sentence has unsupported entities and poor embeddingbased scores. the second is semantically similar yet missing a critical concept (complications) which cannot be inferred from the context. the third has full entity coverage (iufd is a term for stillbirth) yet bertscore is low because there is no mention of social work. only the final sentence is covered by both metrics and treated as supported. ## learning to revise unsupported text the goal is to re-write these unsupported reference sentences such that they are supported, i.e., covered by the source notes. to learn this revision task without a gold standard ground-truth, we take each supported reference sentence, inject noise to create unsupported, yet realistic, alternatives ( §4.1), and then use this mix of real and synthetic data to create a supervised set of positive and negative revisions to support a contrastive learning objective ( §4.2) foot_3 . ## experimental setup we design experiments around our central hypothesis: for a setting (long form hospital-course summarization), in which high-quality reference data is limited, reference revision is the best data-centric intervention to improve model faithfulness. as such, we restrict the set of comparison methods to model-agnostic methods which explicitly address data quality. based on our thorough literature review, we consider two classes of baselines: those which filter low quality data @xcite @xcite @xcite , and those which control for it @xcite . ## results please refer to appendix for basic statistics on the revised training datasets ( §f), bart summariza2021 ) rely on entities. 7 they use the dependency-arc entailment (dae) model @xcite to identify inconsistent spans. without such a model for clinical text, we use unsupported entities. ## limitations our sentence-level revision strategy does not consider the impact of changing one sentence on the reference as a whole, which can impact coherence. this can be addressed in future work by exploring summary-level revision (especially for corpora with short summaries) or incorporating coherence into revision re-scoring and/or contrast set creation. given the complexity and expertise required for the task, the length of both source notes and summaries, and a mandatory license for data access, only 30 human ratings were procured from a single annotator. the assessment still took four full days. ## conclusion we propose a new approach to mitigating the downstream impact of noisy references on summarization models. we learn a model to improve the coverage of existing references from a large clinical corpus of ehr notes, and re-train models on revised references. results show that reference revision is a more effective intervention for improving faithfulness than quality filtering for this task. intended use & failure modes. the goal of this paper is to make progress toward automatic summarization of a patient's hospital admission. deploy such a system in a real-world clinical setting has its own set of ethical and procedural concerns. robustness, fairness, and trust is vital to any nlp system, especially one deployed in a hospital setting where lives are at risk. as with many nlp datasets, our mimic-iii dataset likely contains biases, which may be perpetuated by its use. it is important to analyze the underlying population to identify demographic, social, and economic discrepancies vis-a-vis the broader population of interest. model-generated errors could be harmful to patient safety and even negatively affect outcomes. there are lessons to be learned from existing clinical decision support tools @xcite . furthermore, there is a moral hazard from deploying clinical systems, in which clinicians start to over-rely on a system at the expense of their own judgment @xcite . ehrs are also living systems and deploying a summarization system within it necessitates evolving with the ehr and the underlying population. finally, we compute the tf-idf overlap (t f _idf (t x , t y )) to compensate for occasional noise in embedding space. we define the aggregate score as the average of embed, code, and tf-idf overlaps. for entity types with no available codes (treatments, procedures, and tests), we only examine mention overlap. based on validation on a manually labeled held-out set, we classify e identifying correlates we collect a wide-range of visit-level statistics and compute the pearson correlation coefficient with respect to the hallucination rate, defined as the fraction of entities in the reference text not present in the available source documentation. unsurprisingly, lexical overlap (unigram coverage) is highly correlated (88%) with the hallucination rate. examples with well-supported references contain more source notes and distinct note types. the mimic-iii notes do not cover time a patient spent outside the icu. interestingly enough, however, the number of days spent outside the icu has zero direct correlation with the hallucination rate. distribution of hallucinations. we examine the example and sentence-level distribution of hallucinations before devising a revision strategy. figure 6 reveals a degree of uniformity in the examplelevel hallucination rate and, to a lesser extent, sentence-level foot_5 . the figure indicates that the faithfulness issue is not concentrated to just a few very low coverage examples, or sentences. as such, example-level quality filtering is noisy since there is no clear coverage boundary and relatively few references contain zero entity hallucinations (< 2k). these two basic plots inform two key design choices: (1.) to address quality at the sentencelevel rather than the reference; (2.) to enforce diversity of faithfulness in synthetic hallucinations.