Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards (2022)

Abstract: Neural image-to-text radiology report generation systems offer the potential to improve radiology reporting by reducing the repetitive process of report drafting and identifying possible medical errors. These systems have achieved promising performance as measured by widely used NLG metrics such as BLEU and CIDEr. However, the current systems face important limitations. First, they present an increased complexity in architecture that offers only marginal improvements on NLG metrics. Secondly, these systems that achieve high performance on these metrics are not always factually complete or consistent due to both inadequate training and evaluation. Recent studies have shown the systems can be substantially improved by using new methods encouraging 1) the generation of domain entities consistent with the reference and 2) describing these entities in inferentially consistent ways. So far, these methods rely on weakly-supervised approaches (rule-based) and named entity recognition systems that are not specific to the chest X-ray domain. To overcome this limitation, we propose a new method, the RadGraph reward, to further improve the factual completeness and correctness of generated radiology reports. More precisely, we leverage the RadGraph dataset containing annotated chest X-ray reports with entities and relations between entities. On two open radiology report datasets, our system substantially improves the scores up to 14.2% and 25.3% on metrics evaluating the factual correctness and completeness of reports.

## introduction an important medical application of natural language generation (nlg) is to build assistive systems that take x-ray images of a patient and generate a textual report describing clinical observations in the images @xcite @xcite . this is a clinically important task, offering the potential to reduce radiologists' repetitive work and generally improve clinical communication @xcite . recently, a lot of attention has been given to new architectures @xcite @xcite and how the structure of data could be input into the system @xcite . these systems have achieved promising performance as measured by widely used nlg metrics such as bleu @xcite and cider @xcite . however, these studies face important limitations. first, they present an increased complexity in architecture that offers only marginal improvements on nlg metrics. secondly, these systems that achieve high performance on nlg metrics are not always factually complete or consistent due to both inadequate training and evaluation of these systems. @xcite have shown that existing systems are inadequate in factual completeness and consistency, and that an image-to-text radiology report generation (rrg) system can be substantially improved by replacing the widely used nlg metrics with "factually-oriented" methods encouraging 1) the generation of domain entities consistent with the reference and 2) describing these entities in inferentially consistent ways. so far, these new methods rely on weakly-supervised approaches (rule-based) to construct nli models for radiology reports and biomedical named entity recognition systems @xcite that are not specific to chest x-rays. despite these "factually-oriented" methods being weakly supervised or being limited to generic biomedical entities, their use showed substantial improvements on a wide range of metrics and board-certified radiologists' evaluations. these findings motivate us to propose a new method to further improve the factual completeness and correctness of generated radiology reports. more precisely, we leverage radgraph @xcite , a dataset annotated by radiologists containing chest x-ray radiology reports along with annotated entities and relations. these annotations allow us to create two semantic graphs, one for the generated and one for the reference report. we then introduce three simple rewards that score the differences between the two graphs in terms of entities and relations. these rewards can be directly optimized using reinforcement learning (rl) to further improve the quality of the generated report by our systems. by doing so, we show on two popular chest x-ray datasets that our models are able to maximize the defined rewards but also outperform the previous works on various nlg and factuallyoriented metrics. in summary our contributions are: the paper is structured as follows: first, we describe our factually-oriented graph-based rewards ( §2). more precisely, we begin by examining the radgraph dataset ( § 2.1) and how we leveraged the annotations to create our rewards ( § 2.2). then, we explain the architecture of the model ( § 3) that we used to generate reports and how we trained it using negative log-likelihood (nll) and rl. the sections that follow afterwards are dedicated to the datasets used for the experiments ( § 4) and the metrics ( § 5) chosen to evaluate the generation of reports. this latter section is divided into two groups: the classic nlg metrics ( § 5.1) and the factually-oriented metrics ( § 5.2). finally, we present the results ( § 6) and end this paper with a section addressing related works ( § 7). ## factually-oriented graph-based reward in this section, we present a new semantic graphbased reward, called the radgraph reward, used throughout our experiments. we first start by explaining in section 2.1 the radgraph dataset and how we get the annotations that shape our reward. in section 2.2, we explain how we construct the radgraph reward and its different variants. radiologist annotations on reports from mimic-cxr and chexpert (smit et al., 2020) datasets (50 reports each). ## model architecture to encode an x-ray image, we extract convolutional visual features d of size 49 × 1024 using a densenet-121 @xcite . to generate language, we use a one-layered bert @xcite with cross-attention over the visual features. more formally, the cross-attention of the transformer layer is written: training if we denote θ as the model parameters, then θ is learned by maximizing the likelihood of the observed report @xmath0, y 2 , • • • , y n ) or in other words by minimizing the negative loglikelihood. the objective function is given by: after the nll training, we start a rl training that optimizes one of our factually-oriented rewards. the loss function in equation 2 is now given by: here r( ȳ ) acts as a baseline @xcite to reduce the variance of r(y ). in our case, r( ȳ ) is the expected reward by sampling from the model during training. hyper-parameters our model consists of 1 transformer block of size 768 with a feed-forward layer of size 3072. as optimizer, we pick adam (kingma and ba, 2015) with a learning rate of 3e -4 and mini-batch size of 128. we decode with a beam-search of size 3. ## datasets to carry out our experiments, we use two chest x-ray datasets: mimic-cxr @xcite and open-i chest x-ray (indiana university, demner-fushman et al. ( 2012 )). for both datasets, we use the official splits but we discard the reports that do not contain a findings section. mimic-cxr thus consists of 152, 173 training samples, 1, 196 for validation and 2, 347 for testing; similarly, the indiana dataset, originally containing 5, 935 training images, 740 for validation and 740 for testing, consists of only 3, 335 reports in total if we do not count the multiplicity of images in each study and if we discard the reports without findings. since this renders the dataset too small to both train and test a rrg model, open-i dataset is only used for testing purposes. for each sample of both datasets, we generate what we consider the ground-truth radiology diagnostic by running the chexbert labeler @xcite on the ground-truth findings section. this creates for each ground-truth report the associated diagnostic label, which describe for 14 possible observations the degree of presence (e.g. consolidation or edema for which the report can be positive, negative, uncertain or unspecified). this label is used to compute the f1 chexbert score (see section 5.2). ## results in this section, we discuss the results displayed in ## related work first, we describe the studies that presented architectural novelties. usually, they focus on improving the widely used nlg metrics such as bleu and rouge. we then proceed to go over the previous works that used reinforcement learning (rl) to optimize nlg or factually-oriented metrics. finally, we quickly mention a few projects that do not fall into the first two categories. ## conclusion in this paper, we leveraged the radgraph dataset containing annotated chest x-ray reports with entities and relations between entities to design a new reward that qualitatively evaluate the factual correctness and completeness of the generated reports. we showed on two datasets that directly optimizing these rewards outperforms previous approaches that prioritize traditional nlg metrics or leverage unsupervised out-of-domain systems as factual-oriented metrics. our best model reports up to +14.2% improvements on these factual metrics on the mimic-cxr and +25.3% on open-i.