GREEN : Generative Radiology Report Evaluation and Error Notation (2024)

Abstract: Evaluating radiology reports is a challenging problem as factual correctness is extremely important due to its medical nature. Existing automatic evaluation metrics either suffer from failing to consider factual correctness (e.g., BLEU and ROUGE) or are limited in their interpretability (e.g., F1CheXpert and F1RadGraph). In this paper, we introduce GREEN (Generative Radiology Report Evaluation and Error Notation), a radiology report generation metric that leverages the natural language understanding of language models to identify and explain clinically significant errors in candidate reports, both quantitatively and qualitatively. Compared to current metrics, GREEN offers: 1) a score aligned with expert preferences, 2) human interpretable explanations of clinically significant errors, enabling feedback loops with end-users, and 3) a lightweight open-source method that reaches the performance of commercial counterparts. We validate our GREEN metric by comparing it to GPT-4, as well as to error counts of 6 experts and preferences of 2 experts. Our method demonstrates not only higher correlation with expert error counts, but simultaneously higher alignment with expert preferences when compared to previous approaches.

## introduction machine learning has enabled great progress in the automatic interpretation of images, where vision language models (vlms) translate features of images into text @xcite . in the medical domain, patient images are interpreted by radiologists, which is referred to as radiology report generation (rrg). automated and high-quality rrg has the potential to greatly reduce the workload of radiologists alleviating burdens arising from shortage of radiologists, generally improve clinical communication @xcite , and increase the accuracy of radiology reports @xcite . commonly used evaluation metrics in rrg literature @xcite @xcite seek to evaluate a generated radiology report against a reference report written by a radiologist by leveraging simple n-grams overlap, general language similarity, pathology identification within specific imaging modalities and disease classes, and commercially-available large language models. to achieve performance on par with radiologists, evaluation metrics must be adept with the radiology language in order to accurately assess factual correctness and levels of uncertainties. additionally, rrg metrics should be interpretable in a scalable fashion to enable a feedback loop between the generated reports and the experts who review them. moreover, these metrics should be open-source to allow for assessment of private datasets that require the safeguarding of patient information. current rrg metrics fall short of capturing the nuanced and multifaceted nature of radiology reports. to mitigate the current gaps in appropriate metrics for rrg, we introduce green (generative radiology evalua-rouge-l evaluation 0.57 0.57 candidate 2: pleural effusion not present. candidate 1: pleural effusion is present. bleu evaluation 0.75 0.75 candidate 2: pleural effusion not present. candidate 1: pleural effusion is present. = = f1radgraph evaluation 1.0 0.5 candidate 2: pleural effusion not present. candidate 1: pleural effusion is present bertscore evaluation 0.85 0.75 candidate 2: pleural effusion not present. candidate 1: pleural effusion is present. our proposed green evaluation 1.0 0.0 candidate 2: pleural effusion not present. candidate 1: pleural effusion is present. ## related work the literature demonstrates various advances in generating radiology reports from medical images @xcite @xcite @xcite @xcite . for instance, a set of evaluation metrics are commonly utilized to assess the quality of the generated reports and focus on lexical similarity (e.g., rouge-l @xcite and bleu @xcite ) and factual correctness (e.g., f1chexbert @xcite and f1radgraph @xcite ). f1chexbert assesses the accuracy of identified disease labels in reports against a narrow reference, covering only 14 chexbert classes of common-but-specific chest x-ray findings. f1radgraph enhances factual correctness evaluations by comparing the agreement on anatomical and observational entities between candidate reports and reference reports, using a graph model trained on human annotations. however, the correlation of f1radgraph with manual evaluations by radiologists is low, leading to the development of more closely-aligned metrics such as radcliq @xcite . radcliq consists of an ensemble of rouge, bleu, chexbert embedding similarities, and radgraph to form a composite metric which aims to match expert-generated error counts. while radcliq is effective in mirroring these error counts, it has low interpretability as the individual metric weights are unknown and the single numerical score is inadequate for clinical integration (figure 1 ). our approach stands out from previous metrics by emphasizing clinical relevance and interpretability, showing higher alignment with expert error counts and expert preferences while still leveraging an open-@xmath0. ## multimodality generalizability we now demonstrate how this method can be applied to various other imaging modalities. ## conclusion in this study, we introduced green (generative radiology report evaluation and error notation), a novel metric aimed at enhancing the evaluation of radiology reports. green outperforms existing metrics by aligning closely with the nuanced requirements of medical diagnostics through its precise assessment of factual correctness and uncertainties. the score's high correlation with expert evaluations underscores its effectiveness. the open-source nature of green supports widespread use and collaborative improvements without compromising data privacy. its lightweight design ensures practicality across diverse settings, reducing computational demands. additionally, green's adaptability across different imaging modalities and extensive datasets encourage broader applicability and research in medical artificial intelligence. the green metric's ability to maintain robust performance on ood data further signifies its versatility and potential as a standard for future developments in automated radiology reporting.