Extracting and Encoding: Leveraging Large Language Models and Medical Knowledge to Enhance Radiological Text Representation (2024)

Abstract: Advancing representation learning in specialized fields like medicine remains challenging due to the scarcity of expert annotations for text and images. To tackle this issue, we present a novel two-stage framework designed to extract high-quality factual statements from free-text radiology reports in order to improve the representations of text encoders and, consequently, their performance on various downstream tasks.In the first stage, we propose a Fact Extractor that leverages large language models (LLMs) to identify factual statements from well-curated domain-specific datasets. In the second stage, we introduce a Fact Encoder (CXRFE) based on a BERT model fine-tuned with objective functions designed to improve its representations using the extracted factual data. Our framework also includes a new embedding-based metric (CXRFEScore) for evaluating chest X-ray text generation systems, leveraging both stages of our approach. Extensive evaluations show that our fact extractor and encoder outperform current state-of-the-art methods in tasks such as sentence ranking, natural language inference, and label extraction from radiology reports. Additionally, our metric proves to be more robust and effective than existing metrics commonly used in the radiology report generation literature. The code of this project is available at https://github.com/PabloMessina/CXR-Fact-Encoder .

## introduction in the context of medical image analysis, radiology reports serve as a rich source of information. radiologists routinely generate these free-text reports, which typically include sections such as comparison, indication, findings, and impression, as illustrated in figure 1 . radiology reports have been employed for various purposes, including label extraction for structured supervision in medical image tasks @xcite @xcite , comparison: chest radiographs xxxx. indication: xxxx-year-old male, chest pain. findings: the cardiomediastinal silhouette is within normal limits for size and contour. the lungs are normally inflated without evidence of focal airspace disease, pleural effusion, or pneumothorax. stable calcified granuloma within the right upper lung. no acute bone abnormality. impression: no acute cardiopulmonary process. figure 1 : example image and report from the iu x-ray dataset @xcite training data for models in report generation @xcite @xcite , summarization tasks @xcite , and the development of multimodal models capable of jointly understanding medical images and text @xcite @xcite . a crucial aspect in addressing such tasks is the accurate comprehension of factual information within the report. specifically, the findings and impression sections can be considered as repositories of factual information regarding the imaging examination. these factual statements encompass various elements, including observations (such as abnormalities, diseases, or devices), interpretations derived from observations, references to anatomical locations, discussions on severity or confidence levels, comparisons to previous studies, and more. for instance, in figure 1 , one factual statement indicates no acute bone abnormality (a normal observation), while another describes a stable calcified granuloma within the right upper lung (an abnormality found in a specific anatomical site). despite the various aforementioned applications that recent research has explored in the use of radiology reports, the persistent absence of effective methods for precise fact extraction and encoding from medical reports remains a critical challenge. as demonstrated in our experimental evaluation (section 4), existing encoders and label extraction techniques frequently struggle to capture the nuanced details within free-text radiology reports, re-sulting in incomplete or inaccurate depictions of clinical information. this is evident in various aspects. for instance, existing text encoders developed for the medical domain may struggle to generate consistent representations of paraphrased statements ( in this work, we propose a novel approach that leverages the capabilities exhibited by large language models (llms) such as chatgpt, which have showcased outstanding performance in medical contexts @xcite @xcite , to improve factual statement representation. our methodology also takes advantage of the existing knowledge in expert-annotated datasets. these datasets offer indispensable training data and also serve as a benchmark to enhance our model's clinical terminology and context comprehension. concretely, our contributions are three-fold: we release the weights of all our models, as well as the data and code necessary to replicate the results. we also release cxrfescore as a python library for ease of use by the research community. ## related work in this section, we discuss prior work on bertbased approaches to radiology text representation and label extraction from radiology reports, and leave discussion of prior work on evaluation of factual correctness in radiology text generation, applications of llms to medical text, and knowledge distillation from llms to appendix a.1. ## method we introduce a two-stage method for encoding the information within a cxr report. in the first stage, called fact extraction (section 3.1), we utilize llms to extract facts from the original sentences of the report. in the second stage, called fact encoding (section 3.2), we employ a bert-based text encoder to generate sentence embeddings for each extracted fact. when used in tandem, these two stages form a cohesive system capable of producing vectorial representations of the factual statements found within a cxr report. ## experimental results our experiments evaluate different versions of cxrfe, each defined by a subset of the six tasks (t, c, nli, ec, er, sd) outlined in section 3.2. this results in a total of 64 potential combinations. however, for the sake of simplicity, we heuristically assess only 11 combinations. further details on our rationale are provided in appendix a.3. triplet ranking. we evaluate cxrfe and mul-tiple baselines from the literature on triplet ranking accuracy, using a separate test set of 1000 triplets per rule sampled according to the sampling rules detailed in section 3.4.2. the left side of ## conclusions & future work in this work, we present a novel two-stage framework for extracting and encoding factual information from radiology reports. the first stage, fact extraction, uses chatgpt and t5 to extract factual statements. the second stage, fact encoding, introduces cxrfe, a specialized variant of cxr-bert, fine-tuned through multitask learning by incorporating tasks that support representation improvement. our system's effectiveness is validated through comprehensive evaluations. additionally, we introduce cxrfescore, a novel metric for evaluating radiology text generation, leveraging our two-stage system. we anticipate that our work will stimulate further research in enhanced fact extraction and representation, llm utilization, advanced training methodologies, and improved evaluation metrics. for future work, we aim to expand our framework to integrate visual modality, focusing on image-based fact detection and visual grounding.