R ad G raph- XL : A Large-Scale Expert-Annotated Dataset for Entity and Relation Extraction from Radiology Reports (2024)

Abstract: In order to enable extraction of structured clinical data from unstructured radiology reports, we introduce RadGraph-XL, a large-scale, expert-annotated dataset for clinical entity and relation extraction. RadGraph-XL consists of 2,300 radiology reports, which are annotated with over 410,000 entities and relations by board-certified radiologists. Whereas previous approaches focus solely on chest X-rays, RadGraph-XL includes data from four anatomy-modality pairs - chest CT, abdomen/pelvis CT, brain MR, and chest X-rays. Then, in order to automate structured information extraction, we use RadGraph-XL to train transformer-based models for clinical entity and relation extraction. Our evaluations include comprehensive ablation studies as well as an expert reader study that evaluates trained models on out-of-domain data. Results demonstrate that our model surpasses the performance of previous methods by up to 52% and notably outperforms GPT-4 in this domain. We release RadGraph-XL as well as our trained model to foster further innovation and research in structured clinical information extraction.

## introduction radiology reports, which are critical for patient care, present a challenge for clinical research and applications due to their unstructured format and complex language. to address this, various methods have been developed to automatically extract important information from these reports (langlotz and @xcite @xcite . this is essential for tasks like training medical imaging models and monitoring diseases @xcite @xcite . however, the effectiveness of these methods is often limited by the specific types of information they are designed to extract and the scarcity of densely annotated datasets, which are costly to produce due to the need for expert input. recent initiatives have been directed towards addressing the complexities of deriving structured clinical data from the unstructured text of radiology reports. one such approach is radgraph foot_0 @xcite , which comprises a dataset and schema that aim to capture a wide array of clinically relevant information, such as observation and anatomical details. this schema is designed to streamline and standardize the annotation process, thereby facilitating the extraction of meaningful insights from radiology narratives. however, its application has been primarily confined to chest x-ray reports, which limits its utility across the diverse spectrum of radiology documentation (as shown in additionally, large language models (llms) have also been explored to extract information with various prompting strategies @xcite , beginning with a single example and expanding up to 200-shot examples to maximize the gpt-4 model's context window. in addition to the impracticalities of the approach, including issues with access, costs, and inference time. moreover, there is no certainty that llms will perform equally well across different modalities and anatomical studies, and this hypothesis remains untestable due to the lack of annotated data in these areas. motivated by these limitations, we introduce radgraph-xl, a large-scale dataset featuring 2,300 radiology reports with approximately 410,000 expert annotations by radiologists (as shown in the structure of the paper is organized as follows. first, we introduce the radgraph-xl dataset in section 3.1 and discuss its differences from radgraph-1.0. next, we outline the process of annotating the dataset and the challenges encountered in section 3.2, and present some key statistics in 3.3. our focus then shifts to experiments in section 4, where we elaborate on our model's training process (section 4.1), our methodology for selecting the best-performing model (section 4.2), and its comparison with a solution that employs a large language model (llm) as the transformer backbone (section 4.3). importantly, we highlight our model's performance on entities defined as measurements (section 4.4)-a novel aspect of our annotations-and compare our model's performance with that of models from previous studies (section 4.5). the experiments section concludes with a brief evaluation of gpt-4 against our reference test set (section 4.6). following this, section 5 presents the outcomes of our reader study, and the paper concludes with section 6. ## reader study we conduct a reader study on out-of-domain data, namely deep vein thrombosis (dvt) ultrasound reports, in order to evaluate the ability of our model to generalize to new radiological text. we chose 20 reports with semantic diversity, extracted the impressions section, and ran our top-performing model biomedvlp-cxr-bert(b) to predict entities and relations. our model generated 265 entities (13.25 per report) and 207 relations (10.35 per report). a board-certified radiologist was tasked to detect critical errors, imprecise or ambiguous classifications and unclear labels, and provide a subjective overview summary. ## conclusion we introduced radgraph-xl, an expansive dataset comprising 2,300 radiology reports enriched with over 410,000 expert annotations from radiologists (section 3). this dataset spans a variety of entities, relations, and measurements across multiple modality-anatomy pairs, enriching the data extracted from radiological texts with unprecedented precision and depth. we have conducted experiments (section 4) using transformer-based models trained for automatic annotation of radiology reports, employing state-of-the-art frameworks for entity and relation extraction. through comprehensive ablation studies (section 4.2, 4.3, and 4.4) and a reader study (section 5) that extends to outof-domain data, we meticulously evaluated our model's performance. the results reveal that our model not only sets a new benchmark, but also outperforms previous methods by as much as 52% (section 4.5) and notably outperforms gpt-4 (section 4.6) in this specific field. to encourage further innovation and research, we release the reports, the annotations, and our trained model.