Multimodal Generation of Radiology Reports using Knowledge-Grounded Extraction of Entities and Relations (2022)

Abstract: Automated reporting has the potential to assist radiologists with the time-consuming procedure of generating text radiology reports. Most existing approaches generate the report directly from the radiology image, however we observe that the resulting reports exhibit realistic style but lack clinical accuracy. Therefore, we propose a two-step pipeline that subdivides the problem into factual triple extraction followed by free-text report generation. The first step comprises supervised extraction of clinically relevant structured information from the image, expressed as triples of the form (entity1, relation, entity2). In the second step, these triples are input to condition the generation of the radiology report. In particular, we focus our work on Chest X-Ray (CXR) radiology report generation. The proposed framework shows state-of-the-art results on the MIMIC-CXR dataset according to most of the standard text generation metrics that we employ (BLEU, METEOR, ROUGE) and to clinical accuracy metrics (recall, precision and F1 assessed using the CheXpert labeler), also giving a 23% reduction in the total number of errors and a 29% reduction in critical clinical errors as assessed by expert human evaluation. In future, this solution can easily integrate more advanced model architectures - to both improve the triple extraction and the report generation - and can be applied to other complex image captioning tasks, such as those found in the medical domain.

## introduction chest x-ray (cxr) studies are among the most frequent radiology studies undertaken in healthcare @xcite . each cxr is accompanied by a text report written by a radiologist or trained radiographer which describes the findings within the study. unfortunately, cxr reports are subject to delays, often due to institutional factors, which can result in adverse patient outcomes @xcite . a possible solution to improve the radiology workflow, and to facilitate timely delivery of accurate reports, is to automate the generation of text reports. however, generating clinically accurate radiology reports is a challenging task. the task of generating a textual description for an image is referred to as image captioning, and recent methods have often adopted encoder-decoder architectures, in which the image embeddings are computed using convolutional neural networks (cnns) (e.g., @xcite and the text is generated using recurrent neural networks (rnns) (e.g., @xcite @xcite , or, more recently, using transformerbased architectures @xcite . such architectures have been proposed to perform automated report generation in the medical domain, with some custom modules introduced for this specific task. for instance, some recent works in cxr report generation have introduced relational memory modules @xcite to allow the model to memorise information from previous generation, and cross-modal memory modules @xcite to encourage alignment between visual and textual information. another line of work has explored ways to inject external knowledge into the model @xcite , based on pre-constructed knowledge graphs or by retrieving other similar reports within the dataset. the above methods all attempt to generate the radiology report directly from the image, using only supervision with a standard cross-entropy loss of the generated text compared to the target text, which will reward verbatim replication of the target text (style), whilst not emphasising accurate reporting of the clinically important findings (content). this concern was partially treated by intro- step 1 triples extractor triples low modify lung volume modify lung opacity located_at right opacity located_at base pneumothorax status absent effusion status absent effusion located_at pleural ## methods in this section we describe how the ground truth triples were extracted from the finding section of each original report. further, we introduce the two-step pipeline, describing in detail the model architectures. in figure 1 , we show a high level design of the two-step pipeline. ## results here we evaluate our proposed method on the mimic-cxr dataset at each step: triples extraction and report generation. every experiment is repeated 3 times using different random seeds to initialise the model weights and randomise batch shuffling; we report the average scores between the 3 different runs. we also conduct some human evaluation on the generated reports, to further assess their clinical accuracy. ## conclusion in this work, we present a two-step framework for cxr automated radiology reporting, which splits the task into triples extraction and report generation. we propose a semi-automated annotation schema, which extracts structured information from a radiology report in the form of triples, and serves to supervise the first step of our approach. further, our method shows state-of-the-art performances on the mimic-cxr dataset for most of the nlg metrics and all the ce metrics. moreover, we conduct human evaluation to assess errors in the generated text, showing how our proposed twostep approach generates 23% fewer errors and 29% fewer critical errors compared to the baseline. nevertheless, end-to-end supervised report generation from images requires further research on improving clinical accuracy in order to have utility as a diagnostic tool. in future, this solution can easily integrate more advanced model architectures -to both improve the triple extraction and the report generation -and can be applied to other complex image captioning tasks, such as those found in the medical domain.