CID at RRG 24: Attempting in a Conditionally Initiated Decoding of Radiology Report Generation with Clinical Entities (2024)

Abstract: Radiology Report Generation (RRG) seeks to leverage deep learning techniques to automate the reporting process of radiologists. Current methods are typically modelling RRG as an image-to-text generation task that takes X-ray images as input and generates textual reports describing the corresponding clinical observations. However, the wording of the same clinical observation could have been influenced by the expression preference of radiologists. Nevertheless, such variability can be mitigated by normalizing textual reports into structured representations such as a graph structure. In this study, we attempt a novel paradigm for incorporating graph structural data into the RRG model. Our approach involves predicting graph labels based on visual features and subsequently initiating the decoding process through a template injection conditioned on the predicted labels. We trained and evaluated our model on the BioNLP 2024 Shared Task on Large-Scale Radiology Report Generation and submitted our results to the ViLMedic RRG leaderboard. Although our model showed a moderate ranking on the leaderboard, the results provide preliminary evidence for the feasibility of this new paradigm, warranting further exploration and refinement.

## introduction radiology report generation (rrg) seeks to liberate radiologists from the repetitive reporting process, allowing them to focus on revising the reports and thereby enhancing the accuracy and efficiency of clinical communication. as a multi-modality task, rrg models usually employ the encoderdecoder architecture, where the encoder is a vision model that is responsible for extracting visual features from radiology images while the decoder is a language model that is responsible for converting visual features into narrative reports. compared with the general image captioning task, the clinical observations in radiology images are more subtle. moreover, the wording of the same clinical observation could have been influenced by the expression preference of radiologists. this raises a challenge to the model's learning ability in terms of extracting fine-grained visual features and generate accurate clinical narratives. our recent review of this field proposed that structured reports can alleviate the inherent diversity of natural language, thus contributing to more accurate results in the model training and evaluation @xcite . benefiting from the advent of radgraph @xcite , a graph-based representation of clinically significant fine-grained information extracted from reports, recent research has commenced utilising such structured representation of reports to enhance the rrg models. relevant studies can be broadly classified into two paradigms. one paradigm fuses the graph features with visual features, letting the decoder learn how to generate the next word from a given input and the fused features @xcite @xcite . another paradigm focuses on graph generation based on the visual features and decouples the visual features from the decoding stage, allowing the language model to learn solely how to generate text based on the predicted graph @xcite . this has sparked our interest, as it raises a research question of whether there exists a new paradigm that can explicitly leverage graph structures to improve the quality of generative language models, while also enabling visual features to supplement the predicted graph with missing information. based on this idea, we attempt a novel approach, whereby the predicted graphs are fed into a template prompt, replacing the traditional special token as the initial input to the decoder, aiming to enable a clearer query to the associated image features during the generation process. ## related work in early research on rrg, many studies introduced disease labels to enhance their models @xcite @xcite @xcite . as research progressed, some studies began to explore the use of graph to replace disease labels as it can represent more fine-grained information @xcite . the graph is considered as a normalized representation of a report in terms of the the key information entities and their relationships @xcite . to utilise graph data, nooralahzadeh et al. (2021) and @xcite proposed modelling rrg as a pipeline of image-to-graph and graph-to-text tasks. @xcite followed the same paradigm although their study contained only the first part. in contrast, @xcite interpreted the rrg as an image-to-text task where a graph prediction module was appended to the visual encoder. additionally, the graph features were combined with visual features and passed to the text decoder, allowing the text decoder to learn to attend to different features. @xcite and @xcite employed a similar feature fusion approach, yet their graph was not directly predicted from visual features, but rather retrieved from the paired report of a similar image identified by comparing their visual features. ## conclusion in this study, we propose a novel approach for utilizing graph structural data to support rrg. this approach involves predicting graph labels based on visual features and leveraging the predicted labels to initialize the decoder input through a template injection. we evaluated our model following the bionlp 2024 shared task 1: radiology report generation, where the results have been submitted to the vilmedic rrg leaderboard. we discuss the limitations of our preliminary rrg model and the initial experiments and outline several directions for improving our model. our model and codes are available on github @xcite .