Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports (2020)

Abstract: Neural abstractive summarization models are able to generate summaries which have high overlap with human references. However, existing models are not optimized for factual correctness, a critical metric in real-world applications. In this work, we develop a general framework where we evaluate the factual correctness of a generated summary by fact-checking it automatically against its reference using an information extraction module. We further propose a training strategy which optimizes a neural summarization model with a factual correctness reward via reinforcement learning. We apply the proposed method to the summarization of radiology reports, where factual correctness is a key requirement. On two separate datasets collected from hospitals, we show via both automatic and human evaluation that the proposed approach substantially improves the factual correctness and overall quality of outputs over a competitive neural summarization system, producing radiology summaries that approach the quality of human-authored ones.

## introduction neural abstractive summarization systems aim at generating sentences which compress a document while preserving the key facts in it @xcite @xcite . these systems are potentially useful in many realworld applications. for example, @xcite have shown that customized neural abstractive summarization models are able to generate radiology summary statements with high quality by summarizing textual findings written by radiologists. this task has significant clinical value because of its potential to accelerate the radiology workflow, reduce repetitive human labor, and improve clinical communications @xcite . however, while existing abstractive summarization models are optimized to generate summaries that highly overlap with human references @xcite , this does not guarantee factually correct summaries, as shown in figure 1 . therefore, maintaining factual correctness of the generated summaries remains a critical yet unsolved problem. for example, @xcite found that about 30% of the outputs from a radiology summarization model contain factual errors or inconsistencies. this has made such a system unusable in practice, as factual correctness is critically important in this domain to prevent medical errors. existing attempts at improving the factual correctness of abstractive summarization models have seen very limited success. for example, @xcite augmented the attention mechanism of neural models with factual triples extracted with open information extraction systems; @xcite studied using natural language inference systems to rerank generated summaries based on their factual consistencies; @xcite proposed to verify factual consistency of generated summaries with a weakly-supervised model. despite these efforts, none of the existing work has focused explicitly on optimizing an abstractive summarization system with a correctness objective. as a result, even state-of-the-art systems trained with ample data still produce summaries with a substantial number of factual errors @xcite . in this work we aim to optimize the factual correctness of existing neural summarization systems, with a focus on summarizing radiology reports. this task has several key properties that make it ideal for studying factual correctness in summarization models. first, the clinical facts or observations present in radiology reports have less ambiguity compared to open-domain text, which allows objective comparison of facts. second, radiology reports involve a relatively limited space of facts, which makes automatic measurement of factual correctness in the generated text approachable. lastly, as factual correctness is a crucial metric in this domain, improving factual correctness will directly lead to an ability to use the system. to this end, we design a framework where an external information extraction system is used to extract information in the generated summary and produce a factual accuracy score by comparing it against the human reference summary. we further develop a training strategy where we combine a factual correctness objective, a textual overlap objective and a language model objective, and jointly optimize them via reinforcement learning (rl). on two datasets of radiology reports collected from different hospitals, we show that our training strategy substantially improves the factual correctness of the summaries generated by a competitive neural summarization system. moreover, we observe for the first time that, even in the absence of a factual correctness objective, optimizing a textual overlap-based metric substantially improves the factual correctness of the resulting system compared to maximum likelihood training. we further show via human evaluation and analysis that our training strategy leads to summaries with higher overall quality and correctness and which are closer to the human-written ones. our main contributions are: (i) we propose a general framework and a training strategy for improving the factual correctness of summarization models by optimizing a multi-part objective via rl; factual correctness in summarization. our work is closely related to recent work that studies factual correctness in summarization. @xcite proposed to improve summarization models by attending to fact triples extracted using open information extraction systems. @xcite compared different information extraction systems to evaluate the factual accuracy of generated text. @xcite explored using natural language inference systems to evaluate the correctness of generated summaries, and found models trained on existing datasets to be inadequate. @xcite proposed to evaluate factual consistencies in the generated summaries using a weakly-supervised fact verification model. despite these efforts, none of this work has shown success in directly optimizing a summarization system for factual correctness, and to our knowledge our work represents the first attempt in this direction. while our work is focused on improving neural summarization models, we note that the idea of using information extraction systems to evaluate the fidelity of generated text has also been explored for data-to-text generation @xcite . summarization of radiology reports. @xcite first studied the problem of automatic generation of radiology impressions by summarizing textual radiology findings, and showed that an augmented pointer-generator model achieves high overlap with human references. @xcite extended this model with an ontologyaware pointer-generator and showed improved summarization quality. li et al. (2019) and liu et al. (2019) studied generating textual descriptions of radiology findings from medical images, and proposed rl-based approaches to tackle this problem. while @xcite found that about 30% of the radiology summaries generated from neural models contain factual errors, improving factual correctness in radiology summarization remains unstudied. ## task & baseline pointer-generator we start by briefly introducing the task of summarizing radiology findings. given a passage of radiology findings represented as a sequence of tokens @xmath0, x 2 , . . . , x n }, with n being the length of the findings, the task involves finding a sequence of tokens @xmath1, y 2 , . . . , y l } that best summarizes the salient and clinically significant findings in x. in routine radiology workflow, an output sequence y is produced by the radiologist, which we treat as a reference summary sequence. 1 to model the summarization process, we use the background-augmented pointer-generator network @xcite as the backbone of our method. this abstractive summarization model extends a pointer-generator @xcite with a separate background section encoder and is shown to be effective in summarizing radiology notes with multiple sections. we briefly describe this model and refer readers to the original papers for details. at a high level, this model first encodes the input sequence x into hidden states with a bi-directional long short-term memory (bi-lstm) network, and then generates an output sequence y with a separate lstm decoder. to make the input information available at decoding time, an attention mechanism @xcite over the input hidden states is also added to the decoder. the baseline pointer-generator model by @xcite adds two augmentations to this attentional encoder-decoder model to make it suitable for summarizing radiology findings: copy mechanism. to enable the model to copy words from the input, a copy mechanism @xcite is added to calculate a generation probability at each step of decoding. this generation probability is then used to blend the original output vocabulary distribution and a copy distribution to generate the next word. background-guided decoding. as shown in figure 1 , radiology reports often consist of a background section which documents the crucial study background information (e.g., purpose of the study, patient conditions), and a findings section which documents clinical observations. while words can be copied from the findings section to form the summary, @xcite found it worked better to separately encode the background section, and inject the representation into the decoding process by concatenating it with the input. ## fact checking in summarization summarization models such as the one described in section 3 are commonly trained with the teacherforcing algorithm @xcite ) by maximizing the likelihood of the reference, humanwritten summaries. however, this training strategy results in a significant discrepancy between what the model sees during training and test time, often referred to as the exposure bias issue @xcite , leading to degenerate output at test time. an alternative training strategy is to directly optimize standard metrics such as rouge scores @xcite with rl and this was shown to improve summarization quality @xcite . nevertheless, this method still provides no guarantee that the generated summary is factually accurate and complete, since the rouge scores merely measure the superficial text overlap between two sequences and do not account for the factual alignment between them. to illustrate this, a reference sentence pneumonia is seen and a generated sentence pneumonia is not seen have substantial text overlap and thus the generated sentence would achieve a high rouge score, however the generated sentence conveys an entirely opposite fact. in this section radiographs show severe cardiomegaly with plural effusions. @xmath2 severe cardiomegaly is seen. ## experiments we collected two real-world radiology report datasets and describe our experiments using them as our main training and evaluation corpora. ## results we first present our automatic evaluation results on the two collected datasets. we then present a human evaluation with board-certified radiologists where we compare the summaries generated by humans, the baseline and our proposed model. ## analysis & discussion fluency and style of summaries. our human evaluation results in section 6.2 suggest that in terms of fluency our model output is less preferred than human reference and baseline output. to further understand the fluency and style of summaries from different models at a larger scale, we trained a neural language model (lm) for radiology summaries following previous work @xcite . intuitively, radiology summaries which are more fluent and consistent with humans in style should be able to achieve a lower perplexity under this in-domain lm, and vice versa. to this end, we collected all human-written summaries from the training and dev split of both datasets, which in total gives us about 222,000 summaries. we then trained a strong mixture of softmaxes lm @xcite on this corpus, and evaluated the perplexity of test set outputs for all models. the results are shown in limitations. while we showed the success of our proposed method on improving the factual correctness of a radiology summarization model, we also recognize several limitations of our work. first, our proposed training strategy crucially depends on the availability of an external ie module. while this ie module is relatively easy to implement for a domain with a limited space of facts, how to generalize this method to open-domain summarization remains unsolved. second, our study was based on a rule-based ie system, and the use of a more robust statistical ie model can potentially improve the results. third, we mainly focus on key factual errors which result in a flip of the binary outcome of an event (e.g., presence of disease), whereas factual errors in generated summaries can occur in other forms such as wrong adjectives or coreference errors @xcite . we leave the study of these problems to future work. ## conclusion in this work we presented a general framework and a training strategy to improve the factual correctness of neural abstractive summarization models. we applied this approach to the summarization of radiology reports, and showed its success via both automatic and human evaluation on two separate datasets collected from hospitals. our general takeaways include: (1) in a domain with a limited space of facts such as radiology reports, a carefully implemented ie system can be used to improve the factual correctness of neural summarization models via rl; (2) even in the absence of a reliable ie system, optimizing the rouge metrics via rl can substantially improve the factual correctness of the generated summaries. we hope that our work draws the community's attention to the factual correctness issue of abstractive summarization models and inspires future work in this direction.