Finding-Centric Structuring of J apanese Radiology Reports and Analysis of Performance Gaps for Multiple Facilities (2025)

Abstract: This study addresses two key challenges in structuring radiology reports: the lack of a practical structuring schema and datasets to evaluate model generalizability. To address these challenges, we propose a “Finding-Centric Structuring,” which organizes reports around individual findings, facilitating secondary use. We also construct JRadFCS, a large-scale dataset with annotated named entities (NEs) and relations, comprising 8,428 Japanese Computed Tomography (CT) reports from seven facilities, providing a comprehensive resource for evaluating model generalizability. Our experiments reveal performance gaps when applying models trained on single-facility reports to those from other facilities. We further analyze factors contributing to these gaps and demonstrate that augmenting the training set based on these performance-correlated factors can efficiently enhance model generalizability.

## introduction a radiology report documents abnormal findings and suspected diseases observed in medical images. radiology reports contain expert insights; however, they are often recorded in free-text format, limiting their secondary application. structuring these reports through information extraction (ie) can support a wide range of applications, such as report generation @xcite and multimedia reports @xcite . despite advancements in ie from radiology reports @xcite @xcite , two critical challenges hinder the practical application of structured reports: the lack of a well-designed structuring schema for practical use and datasets suitable for evaluating the generalizability of structuring models. we propose finding-centric structuring (fcs), which organizes reports around individual findings to address the first challenge. figure 1 shows an overview of fcs. our approach structures reports into individual findings along with related attributes such as characteristics and diagnoses. structured data created by fcs can be useful for a variety of applications. for example, fcs can be applied to medical visual grounding @xcite , which aligns sentences in reports with corresponding objects in images. by decomposing these reports into finding-centric data, fine-grained medical visual grounding for individual findings is promoted. furthermore, fcs allows radiologists to efficiently track changes in the size of each finding and monitor the effectiveness of treatments. fcs enables us to go beyond existing secondary uses such as report retrieval, supporting applications focused on individual findings. the second challenge involves assessing the generalizability of structuring models. @xcite reports that radiologists use diverse terminologies. for example, they may describe subsolid nodules using synonyms such as "ggn." this variability raises concerns about the ability of the model to accurately structure reports with varied writing styles and across facilities. most existing studies on structuring reports @xcite @xcite use reports from a single facility or focus on specific diseases to validate their models, limiting the evaluation of model generalizability. we construct jradfcs, a large-scale dataset annotated with nes and relations based on our schema, comprising 8,428 japanese ct reports from seven facilities, to address second challenge. jradfcs includes a wide variety of reports covering different organs and diseases by collecting all reports written during a specific period. this diversity makes jradfcs suited for evaluating the generalizability of models across various reports. in developing a model for practical use, it is difficult to use data from multiple facilities as a training set due to contractual and cost constraints. therefore, as a more practical setting, we evaluate the performance of a model trained on single-facility reports when applied to reports from other facilities. we evaluate various bert @xcite models, including our bert for the radiology domain and a large language model (llm), revealing large performance gaps between facilities. additionally, to identify the factors contributing to these performance gaps, we analyze the relationship between metrics indicating the complexity of reports, such as the length of the report, and model performance. furthermore, we demonstrate that training set augmentation based on the identified complexity metrics can efficiently improve performance on reports from other facilities. ## related work various annotation schemes for radiology reports have been proposed. @xcite propose a schema for nes, which has been applied in various studies @xcite @xcite . this scheme treats multiple findings such as "結節と網状影 (nodule and reticular shadows)" as a single ne. @xcite and @xcite annotate multiple segments such as "左第7、8肋骨 (left 7th, 8th ribs)" as a single ne. these schemas define coarse-grained nes, which hinder fcs and limit applications requiring precise statistics. radgraph @xcite and its extension, radgraph-xl @xcite , focus on structuring chest x-ray and ct/mr reports, respectively. unlike x-rays, ct scans provide 3d imaging, which enables radiologists to observe detailed characteristics such as the shape and condition of findings. however, radgraph-xl lacks specific labels for characteristics and temporal changes, instead labels them as findings ("observations" in their schema). our approach extracts relevant attributes, such as characteristics, as distinct labels ## finding-centric structuring following discussions with three board-certified radiologists, we developed a set of entities and relations to capture critical information. ## jradfcs we constructed jradfcs, a dataset of japanese ct reports annotated by our schema. two annotators, each with over 10 years of experience in annotation for medical nlp tasks, were employed to annotate nes and their relations. each report was annotated by a single annotator. we collected all ct reports written during a specific period from each facility. the training set consists only of the ouh reports to evaluate the performance for other-facility reports ( ## experiments in this section, we evaluate the performance of the structuring model trained on ouh reports when applied to those from other facilities. specifically, we compare the performance of different bertbased models, including uth-bert @xcite , tohoku-bert (2024) and our bert trained on radiology reports. additionally, we analyze the performance gaps among the facilities and explore potential reasons for these gaps. ## conclusion we addressed two key challenges in structuring radiology reports: the lack of a practical schema and datasets to evaluate model generalizability. to address these challenges, we proposed a fcs that structures radiology reports by each finding and constructed jradfcs, a large-scale dataset containing 8,428 japanese ct reports from seven facilities. we evaluated the performance of a model trained on single-facility reports applied to reports from other facilities, revealing performance gaps. we identified factors causing performance gaps and confirmed improvements of f1 scores on ner and fgs through augmentation based on these factors. moreover, we observed that the improvement is larger for facilities with lower initial performance. our future work is to extend the jradfcs dataset to include reports from other imaging modalities such as magnetic resonance and ultrasound. additionally, we plan to demonstrate whether the fcs schema actually improves any downstream tasks.