Neural image-to-text radiology report generation systems offer the potential to accelerate clinical processes by saving radiologists from the repetitive labor of drafting radiology reports and preventing medical errors. However, existing report generation systems, despite achieving high performances on natural language generation metrics such as CIDEr or BLEU, still suffer from incomplete and inconsistent generations , rendering these systems unusable in practice. In this work, we aim to overcome this problem by proposing two new metrics that encourage the factual completeness and consistency of generated radiology reports. The first metric, the Exact Entity Match score, evaluates a generation by its coverage of radiology domain entities against the references. The second metric, the Entailing Entity Match score, augments the first metric by introducing a natural language inference model into the entity match process to encourage consistent generations that can be entailed from the references. To achieve this, we also developed an in-domain NLI model via weak supervision to improve its performance on radiology text . We further propose a report generation system that optimizes these two new metrics via reinforcement learning. On two open radiology report datasets, our system not only achieves the best performance on these two metrics compared to baselines, but also leads to as much as +2.0 improvement on the F1 score of a clinical finding metric. We show via analysis and examples that our system leads to generations that are more  complete and consistent compared to the baselines.
Neural image-to-text radiology report generation systems offer the potential to improve radiology reporting by reducing the repetitive process of report drafting and identifying possible medical errors. However, existing report generation systems, despite achieving high performances on natural language generation metrics such as CIDEr or BLEU, still suffer from incomplete and inconsistent generations . Here we introduce two new simple rewards to encourage the generation of factually complete and consistent radiology reports: one that encourages the system to generate radiology domain entities consistent with the reference, and one that uses natural language inference to encourage these entities to be described in inferentially consistent ways. We combine these with the novel use of an existing semantic equivalence metric (BERTScore) . We further propose a report generation system that optimizes these rewards via reinforcement learning. On two open radiology report datasets, our system substantially improved the F1 score of a clinical information extraction performance by +22.1 (Delta +63.9\%). We further show via a human evaluation and a qualitative analysis that our system leads to generations that are more factually complete and consistent compared to the baselines.
R
accelerate clinical processes by saving radiologists from the repetitive labor of drafting radiology reports and preventing
improve radiology reporting by reducing the repetitive process of report drafting and identifying possible
R
, rendering these systems unusable in practice. In this work, we aim to overcome this problem by proposing two new metrics that encourage the factual completeness and consistency of generated radiology reports. The first metric, the Exact Entity Match score, evaluates a generation by its coverage of
. Here we introduce two new simple rewards to encourage the generation of factually complete and consistent radiology reports: one that encourages the system to generate
R
against the references. The second metric, the Entailing Entity Match score, augments the first metric by introducing a
consistent with the reference, and one that uses
R
model into the entity match process to encourage consistent generations that can be entailed from the references. To achieve this, we also developed an in-domain NLI model via weak supervision to improve its performance on radiology text
to encourage these entities to be described in inferentially consistent ways. We combine these with the novel use of an existing semantic equivalence metric (BERTScore)
R
two new metrics
rewards
R
not only achieves the best performance on these two metrics compared to baselines, but also leads to as much as +2.0 improvement on the
substantially improved the
R
finding metric. We show via analysis and examples
information extraction performance by +22.1 (Delta +63.9\%). We further show via a human evaluation and a qualitative analysis
A
factually