R a TES core: A Metric for Radiology Report Generation (2024)

Abstract: This paper introduces a novel, entity-aware metric, termed as Radiological Report (Text) Evaluation (RaTEScore), to assess the quality of medical reports generated by AI models. RaTEScore emphasizes crucial medical entities such as diagnostic outcomes and anatomical details, and is robust against complex medical synonyms and sensitive to negation expressions. Technically, we developed a comprehensive medical NER dataset, RaTE-NER, and trained an NER model specifically for this purpose. This model enables the decomposition of complex radiological reports into constituent medical entities. The metric itself is derived by comparing the similarity of entity embeddings, obtained from a language model, based on their types and relevance to clinical significance. Our evaluations demonstrate that RaTEScore aligns more closely with human preference than existing metrics, validated both on established public benchmarks and our newly proposed RaTE-Eval benchmark.

## introduction with the general advancement in natural language processing @xcite @xcite and computer vision @xcite @xcite @xcite , the pursuit of generalist medical artificial intelligence has grown increasingly promising and appealing @xcite @xcite @xcite , leading to the development of free-text generative foundation models capable of understanding and interpreting radiology imaging studies. yet, the complexity and specialized nature of clinical free-form texts, such as radiology reports and discharge summaries, present substantial challenges in evaluating model performance. there is an urgent need for a robust and lightweight free-text evaluation metric to better monitor the development of medical generative foundation models and drive advancements in the field of generalist medical artificial intelligence. in the existing literature, four main types of metrics have been adopted to assess the similarity between free-form texts in medical scenarios, as shown in figure 1 . these include: (i) word overlap metrics, such as bleu @xcite and rouge @xcite . while intuitive, these metrics fail to capture negation or synonyms in sentences, thus neglecting the semantic factuality; (ii) embedding similarity metrics, like bertscore @xcite , provide better semantic awareness but fail to emphasize key medical terms, leading to overlooking errors in critical conclusions; (iii) metrics based on named entity recognition (ner), such as radgraph f1 @xcite and medcon @xcite . al-though tailored for the medical domain, they struggle with synonym unification and are typically restricted to analyzing chest x-ray reports; (iv) metrics relying on large language models (llms) @xcite . while these metrics are better aligned with human preferences, they suffer from potential subjective biases and are prohibitively costly for large-scale evaluation. in this study, we aim to develop a metric that prioritizes key medical entities such as diagnostic outcomes and anatomical features, while exhibiting robustness against complex medical synonyms and sensitivity to negation expressions. we present two contributions: first, we introduce ratescore, a novel evaluation metric specifically designed for radiology reports. this metric emphasizes entitylevel assessments across various imaging modalities and body regions. specifically, it starts by identifying medical entities and their types (e.g., anatomy, disease) from each sentence. to address the challenges associated with medical synonyms, we compute entity embeddings using a synonym disambiguation module and assess their cosine similarities. ratescore then calculates a final score based on weighted similarities that emphasize the clinical importance of the entity types involved. second, we have developed a comprehensive medical named-entity recognition (ner) dataset, rate-ner, which encompasses data from 9 modalities and 22 anatomical regions, derived from mimic-iv and radiopaedia. in addition, we introduce rate-eval, a novel benchmark designed to assess metrics across various clinical texts. this benchmark is structured around three sub-tasks: sentence-level human rating, paragraph-level human rating, and comparison of synthetic reports, each targeting specific evaluation challenges. both the rate-ner dataset and the rate-eval benchmark have been made publicly available, aiming to foster the development of more effective evaluation metrics within the field of medical informatics. our extensive experiments demonstrate the superiority of ratescore. we initially tested our metric against others on the public dataset rexval @xcite , and it shows superior alignment with human preference. considering rexval's limitation to chest x-ray reports, further testing was conducted on the diverse sub-tasks of rate-eval, where ratescore consistently outperformed other metrics. we also conduct ablation studies to validate the effectiveness of different individual components in our pipeline. ## methods in this section, we start by properly formulating the problem, and introducing the pipeline of our metric (sec. 2.1). then, we detail each of the modules for our metric computation, i.e., medical named entity recognition (sec. 2.2), synonym disambiguation encoding (sec. 2.3), and the final scoring procedure (sec. 2.4). lastly, we present the implementation details at each stage. ## rate-eval benchmark to effectively measure the alignment between automatic evaluation metrics and radiologists' assessments in medical text generation tasks, we have established a comprehensive benchmark, rate-eval, that encompasses three tasks, each with its official test set for fair comparison, as detailed below. the comparison of rate-eval benchmark number type scoring principle data source modality anatomy rexval dataset 200 sent. + para. error count mimic_cxr 1 (x-ray) 1 (chest) ours sent. level 2215 sent. error count / potential errors mimic_iv 9 22 para. level 1856 para. 5-point scoring system sim. report 847 sent. mistral 8*7b sentences-level human rating. existing studies have predominantly utilized the rexval dataset @xcite , which requires radiologist annotators to identify and count errors in various potential categories. the metric's quality is assessed by the kendall correlation coefficient between the total number of errors and the result from the automatic metric. the possible error categories are as follows: building on this framework, we introduce two improvements to enhance the robustness and applicability of our benchmark: (1) normalization of error counts: recognizing that a simple count of errors may not fairly reflect the informational content in sentences, we have adapted the scoring to annotate the number of potential errors. specifically, we computed the sum of both correct and incorrect findings in the reference sentence. this sum represents the total number of potential errors that could occur. this approach normalizes the counts, ensuring a more balanced assessment across varying report complexities. (2) diversification of medical texts: unlike existing benchmarks that are limited to chest x-rays from the mimic-cxr dataset @xcite , our dataset includes 2215 reports spanning 9 imaging modalities and 22 anatomies from the mimic-iv dataset @xcite , the involving imaging modalities and anatomies are listed in appendix a.3. specifically, our annotation process is as follows: first, we divide the mimic-iv dataset into 49 subsets based on modality and anatomy. to conduct sentence-level evaluation, we split the report paragraphs into individual sentences by periods and remove the duplicates. next, we randomly sample 25 sentences from each subset to create a reference report list and sample another 1000 reports to form a candidate report list. subsequently, we use several evaluation metrics-bleu, rouge, bertscore, cider, and our proposed ratescore to identify the most similar one from the candidate report list for each report in the reference report list. we take the union of all the metric results as the report pairs for manual annotations. finally, each case in the annotation reports was annotated by two experienced radiologists with over five years of clinical practice. for each candidate report and the corresponding reference report, they are required to count errors in six provided categories as well as the number of potential errors, where the error refers to the candidate report's error based on the reference report. the final human rating result is computed by dividing the total error count by the the number of potential errors. the final sentence-level benchmark is composed of 2215 reference report sentences, candidate report sentences and their rating score. for parameter search (sec. 2.5), we divided all reports into a training set and a test set at an 8:2 ratio, to identify the most effective parameters that align with human scoring rules. paragraph-level human rating. given that medical imaging interpretation commonly involves the evaluation of lengthy texts rather than isolated sentences, we also incorporate paragraph-level assessments into our analysis, from the mimic-iv reports. however, as it is challenging for humans to completely count all errors in long paragraphs accurately, we established a 5-point scoring system for our evaluations, following the rad-peer @xcite -an internationally recognized standard for radiologic peer review. they defines "concur with interpretation" as "correct diagnosis". in our annotation process, due to the difficulty of counting errors in paragraph-level reports (too detailed and some errors are not of great clincial significance), we instructed the radiologists to approximate the ratio of correct diagnosess based on their clinical judgement, following the 'radpeer' standard, which is more aligned with the human rating system for report writing in clinical. the scores range from 5, denoting a perfectly accurate report, to 0, that indicates the report without any correct observations. detailed scoring criteria are provided in appendix a.4, guiding radiologists on how to assign scores at different levels. specifically, our annotation process is as follows: first, we divide the mimic-iv dataset into 49 subsets based on modality and anatomy. next, we sample 20 reports from each subset to create a reference list and 500 reports to form a candidate list. the report selection process is the same as sentence-level human rating. for each candidate report and the corresponding reference report, the radiologists are required to give a 5-point score. the final benchmark in paragraph-level is composed of 1856 reference reports, candidate reports and their rating score. similarly, for parameter search (sec. 2.5), we also divide all reports into training set and a test set at an 8:2 ratio. ## experiments in this section, we start by introducing the baseline evaluation metrics. later, we compare the different metrics with our proposed ratescore on both rexval and rate-eval benchmarks. lastly, we present details for the ablation studies. ## conclusion in this work, we propose a new lightweight, explainable medical free-form text evaluation metric, ratescore, by comparing two medical reports on the entity level. in detail, first, we build up a new medical ner dataset, rate-ner targeting a wide range of radiological report types and train a ner model on it. then, we adopt this model to simplify the complex radiological reports and compare them on the entity embedding level leveraging an extra synonyms disambiguation encoding model. our final ratescore correlates strongly with clinicians' true preferences, significantly outperforming previous metrics both on the former existing benchmark and our new proposed rate-eval, while maintaining computational efficiency and interpretability.