Cross-modal Memory Networks for Radiology Report Generation (2021)

Abstract: Medical imaging plays a significant role in clinical practice of medical diagnosis, where the text reports of the images are essential in understanding them and facilitating later treatments. By generating the reports automatically, it is beneficial to help lighten the burden of radiologists and significantly promote clinical automation, which already attracts much attention in applying artificial intelligence to medical domain. Previous studies mainly follow the encoder-decoder paradigm and focus on the aspect of text generation, with few studies considering the importance of cross-modal mappings and explicitly exploit such mappings to facilitate radiology report generation. In this paper, we propose a cross-modal memory networks (CMN) to enhance the encoder-decoder framework for radiology report generation, where a shared memory is designed to record the alignment between images and texts so as to facilitate the interaction and generation across modalities. Experimental results illustrate the effectiveness of our proposed model, where state-of-the-art performance is achieved on two widely used benchmark datasets, i.e., IU X-Ray and MIMIC-CXR. Further analyses also prove that our model is able to better align information from radiology images and texts so as to help generating more accurate reports in terms of clinical indicators.

## introduction interpreting radiology images (e.g., chest x-ray) and writing diagnostic reports are essential operations in clinical practice and normally requires considerable manual workload. therefore, radiology report generation, which aims to automatically generate a free-text description based on a radiograph, is highly desired to ease the burden of radiologists while maintaining the quality of health care. recently, substantial progress has been made towards research on automated radiology report generation models @xcite @xcite @xcite . most existing studies adopt a conventional encoder-decoder architecture, with convolutional neural networks (cnns) as the encoder and recurrent (e.g., lstm/gru) or non-recurrent networks (e.g., transformer) as the decoder following the image captioning paradigm @xcite . although these methods have achieved remarkable performance, they are still restrained in fully employing the information across radiology images and reports, such as the mappings demonstrated in figure 1 that aligned visual and textual features point to the same content. the reason for the restraint comes from both the limitation of annotated correspondences between image and text for supervised learning as well as the lack of good model design to learn the correspondences. unfortunately, few studies foot_0 are dedicated to solving the restraint. therefore, it is expected to have a better solution to model the alignments across modalities and further improve the generation ability, although promising results are continuously acquired by other approaches @xcite @xcite . in this paper, we propose an effective yet simple approach to radiology report generation enhanced by cross-modal memory networks (cmn), which is designed to facilitate the interactions across modalities (i.e., images and texts). in detail, we use a memory matrix to store the cross-modal information and use it to perform memory querying and memory responding for the visual and textual features, where for memory querying, we extract the most related memory vectors from the matrix and compute their weights according to the input visual and textual features, and then generate responses by weighting the queried memory vectors. afterwards, the responses corresponding to the input visual and textual features are fed into the encoder and decoder, so as to generate reports enhanced by such explicitly learned cross-modal information. experimental results on two benchmark datasets, iu x-ray and mimic-cxr, confirm the validity and effectiveness of our proposed approach, where state-of-the-art performance is achieved on both datasets. several analyses are also performed to analyze the effects of different factors affecting our model, showing that our model is able to generate reports with meaningful image-text mapping while requiring few extra parameters in doing so. ## the proposed approach we regard radiology report generation as an imageto-text generation task, for which there exist sev-eral solutions @xcite @xcite . although images are organized as 2-d format, we follow the standard sequence-to-sequence paradigm for this task as that performed in @xcite . in detail, the source sequence is @xmath0, x 2 , ..., x s , ..., x s }, where x @xmath1, y 2 , ..., y t , ..., y t }, where y @xmath2, t the length of the report and v the vocabulary of all possible tokens. the entire generation process is thus formalized as a recursive application of the chain rule the model is then trained to maximize p(y|i) through the negative conditional log-likelihood of y given the i: ## related work in general, the most popular related task to ours is image captioning, a cross-modal task involving natural language processing and computer vision, which aims to describe images in sentences @xcite @xcite @xcite . among these studies, the most related study from @xcite also proposed to leverage memory matrices to learn a priori knowledge for visual features using memory networks @xcite @xcite @xcite @xcite @xcite , but such operation is only performed during the encoding process. different from this work, the memory in our model is designed to align the visual and textual features, and the memory operations (i.e., querying and responding) are performed in both the encoding and decoding process. recently, many advanced nlp techniques (e.g., pre-trained language models) have been applied to tasks in the medical domain @xcite @xcite @xcite @xcite @xcite . being one of the applications and extensions of image captioning to the medical domain, radiology report generation aims to depicting radiology images with professional reports. existing methods were designed and proposed to better align images and texts or to exploit highly-patternized features of texts. for the former studies, @xcite proposed a co-attention mechanism to simultaneously explore visual and semantic information with a multi-task learning framework. for the latter studies, @xcite introduced a template database to incorporate patternized information and chen et al. (2020) improved the performance of radi-ology report generation by applying a memorydriven transformer to model patternized information. compared to these studies, our model offers an effective yet simple alternative to generating radiology reports, where a soft intermediate layer is provided to facilitate the mappings between visual and textual features, so that more accurate descriptions are produced for generation. ## conclusion in this paper, we propose to generate radiology reports with cross-modal memory networks, where a memory matrix is employed to record the alignment and interaction between images and texts, with memory querying and responding performed to obtain the shared information across modalities. experimental results on two benchmark datasets demonstrate the effectiveness of our model, which achieves the state-of-the-art performance. further analyses investigate the effects of hyper-parameters in our model and show that our model is able to better align information from images and texts, so as to generate more accurate reports, especially with the fact that enlarging the memory matrix does not significantly affect the entire model size.