CLEAR : A Clinically Grounded Tabular Framework for Radiology Report Evaluation (2025)

Abstract: Existing metrics often lack the granularity and interpretability to capture nuanced clinical differences between candidate and ground-truth radiology reports, resulting in suboptimal evaluation. We introduce a **Cl**inically grounded tabular framework with **E**xpert-curated labels and **A**ttribute-level comparison for **R**adiology report evaluation (**CLEAR**). CLEAR not only examines whether a report can accurately identify the presence or absence of medical conditions, but it also assesses whether the report can precisely describe each positively identified condition across five key attributes: first occurrence, change, severity, descriptive location, and recommendation. Compared with prior works, CLEAR’s multi-dimensional, attribute-level outputs enable a more comprehensive and clinically interpretable evaluation of report quality. Additionally, to measure the clinical alignment of CLEAR, we collaborated with five board-certified radiologists to develop **CLEAR-Bench**, a dataset of 100 chest radiograph reports from MIMIC-CXR, annotated across 6 curated attributes and 13 CheXpert conditions. Our experiments demonstrated that CLEAR achieves high accuracy in extracting clinical attributes and provides automated metrics that are strongly aligned with clinical judgment.

## introduction evaluation is becoming increasingly challenging in the era of large language models (llms). while models continue to rapidly hill-climb on benchmarks rapidly @xcite @xcite @xcite , it remains unclear whether these reported metrics match task-specific needs @xcite @xcite . in the context of radiology, the pursuit of generalist foundation models has achieved promising progress @xcite , but do these "appealing" automated metrics truly capture clinically aligned qualities @xcite ? in the existing literature, three main types of metrics have been proposed to assess the quality of generated radiology reports, as illustrated in figure 1 : (i) lexical metrics measure surface-level similarity between the generated and ground-truth reports @xcite @xcite . while straightforward and easy to compute, these metrics struggle to capture nuanced semantics and domain-specific terminology, leading to poor sensitivity for capturing clinically significant errors. (ii) clinical efficacy metrics evaluate the correctness of medical entities and their relationships @xcite @xcite , typically through structured extraction-based comparisons. although more clinically informed than lexical metrics, they lack the resolution to assess fine-grained attributes such as severity, temporal progression, or treatment recommendations. (iii) llm-based metrics @xcite @xcite represent the latest direction, often leveraging the pipeline of llm-as-a-judge @xcite with pre-defined taxonomies such as the six error categories from the rexval dataset @xcite . while getting closer to expert judgment compared with the previous two types, these methods may still lack comprehensive structured attribution and condition-level interpretability. ## clear framework we introduce the clear framework, a hierarchical and fine-grained system for evaluating the clinical accuracy of radiology reports. clear addresses both high-level diagnostic correctness and the descriptive quality of positive findings. as shown in figure 2 , clear includes three sequential stages: label extraction, description extraction, and structured scoring. specifically, given a ground-truth and a candidate report pair, clear first identifies whether the candidate correctly recognizes the presence or absence of specific medical conditions (stage 1). it then examines, for each positively identified condition, whether the ground-truth and candidate reports are aligned across a set of expert-curated descriptive dimensions (stage 2). finally, it aggregates these evaluations into standardized, multidimensional metrics (stage 3). ## clear-bench: attribute-level expert alignment dataset in this section, we introduce clear-bench, an expert-curated, attribute-level dataset in collaboration with five radiologists. inspired by recent expert evaluation datasets for chest x-ray reports @xcite @xcite , clear-bench is specifically designed to assess how well automated evaluators such as clear align with radiologist judgments. clear-bench consists of two annotation subsets: expert ensemble labels and expert-curated attributes. we defer full details of the instruction criteria, interface design, and annotation workflow to appendix b. expert ensemble labels. these provide the ground-truth labels for the presence attribute. we randomly selected 100 studies from the validation and test sets of mimic-cxr-jpg (johnson et al., 2024), excluding any training samples and normal studies. each report was independently annotated from scratch by three boardcertified radiologists. during annotation, the radiologists categorized each of 13 chexpert conditions @xcite into one of five categories (confidently absent, likely absent, neutral, likely present, and confidently present) based on their best interpretation of the report. after the initial round of annotations, we merged confidently present and likely expert-curated attributes. these cover the remaining five report attributes: first occurrence, change, severity, descriptive location, and recommendation. we began by preparing two sets of model-generated attributes, one from llama-3.1-8b-instruct and the other from gpt-4o, for each positive condition identified in the expert ensemble labels. these two sets were merged and then randomly split into two review sets, each with 50 samples from llama and 50 samples from gpt-4o. each set was independently reviewed by separate radiologists. during curation, each radiologist first rated each attribute as incorrect, partially correct, or correct. for non-correct attributes, the radiologist also provided a revised version, which was used to construct the ground-truth attribute set. llm-based labeler achieves substantial gains over existing labelers. we begin with evaluating the performance of the label extraction module. as shown in ## conclusion we present clear, the first clinically grounded, attribute-level evaluation framework that leverages llms to convert free-text radiology reports into a structured tabular format. clear consists of three components: (1) a label extraction module to assess the accurate identification of medical conditions;