Memory-aligned Knowledge Graph for Clinically Accurate Radiology Image Report Generation (2022)

Abstract: Automatic generating the clinically accurate radiology report from X-ray images is important but challenging. The identification of multi-grained abnormal regions in image and corresponding abnormalities is difficult for data-driven neural models. In this work, we introduce a Memory-aligned Knowledge Graph (MaKG) of clinical abnormalities to better learn the visual patterns of abnormalities and their relationships by integrating it into a deep model architecture for the report generation. We carry out extensive experiments and show that the proposed MaKG deep model can improve the clinical accuracy of the generated reports.

## introduction medical images are complex and hard to understand without specialized expertise. given that the volume of radiology images is large, automatically generating the reports by the computer-aided system can alleviate the radiologists from the timeconsuming reporting task. recently, many deep learning models are studied in the automated radiology report generation @xcite @xcite . the deep encoder-decoder architecture has been commonly adopted in the report generation, where visual features were extracted from the input medical images using a convolutional neural network and fed to a recurrent neural network to generate the report. different from image captioning which inputs one image and output one sentence, the report has much longer length while the correctness of medical entities generated in the report is the core requirement. more than the requirement of detecting abnormalities accurately like classification, the report is expected to provide the support details of present abnormalities. thus, generating accurate report with readable and logical descriptions by natural language generation model is the key challenge in the report generation task. generating correct reports is impossible if the pathology of abnormal regions and corresponding abnormalities cannot be identified at first. most existing studies @xcite @xcite proposed the attention and memory mechanism to enhance the identification of abnormal regions. however, different status of the same abnormality may have their specifics and the correlations of these visual patterns are ignored. in addition, identifying the actual abnormalities from abnormal regions is also challenging since the complex and rare abnormalities are hard to determined without professional knowledge. to incorporate the prior medical knowledge, several research @xcite @xcite applied medical knowledge graph of certain abnormalities in the report generation aiming to learn the abnormality relationships. the corresponding representations, i.e., graph embedding, are computed by graph neural network given the input images. however, such representations are affected by the inner-connections of abnormalities for each input where the general characteristic of abnormalities are missing. for example, the representations of "effusion", computed as graph embedding, are different when "effusion" appears with or without "atelectasis". but the general characteristic of "effusion" over all relevant observations, e.g., density or shapes, are only determined by itself independently. this general but independent characteristic is still missed to model by existing approaches which limits the effectiveness the knowledge graph. to alleviate the above challenges, in this work, we propose to learn the memory-aligned graph model, aiming to enhances the pathology identification and prior medical knowledge incorporation. the memory features of possible abnormal regions are first aligned by the input visual feature in an alternative manner, and concatenated with a universal memory embedding before feeding to the graph attention network to compute the graph embedding. the graph embedding are later learned by the classification and fine-tuned in the report generation. we evaluate the proposed approach using two publicly accessible datasets. the evaluation results show the effectiveness of utilizing memory-aligned knowledge graph in generating the clinically accurate radiology report. ## conclusions in this work, we propose a memory-aligned knowledge graph (makg) to enhance the clinically accurate report generation by modeling the relationship between abnormal regions and particular abnormalities. the experiments prove the effectiveness of integrating makg with the generation model is able to generate descriptive report with both correct abnormalities and associated attributes. in ad-dition, the proposed makg is not limited to the specific knowledge graph structure which give the opportunities on incorporating different professional knowledge for specific medical applications.