Reinforced Cross-modal Alignment for Radiology Report Generation (2022)

Abstract: Medical images are widely used in clinical decision-making, where writing radiology reports is a potential application that can be enhanced by automatic solutions to alleviate physicians’ workload. In general, radiology report generation is an image-text task, where cross-modal mappings between images and texts play an important role in generating high-quality reports. Although previous studies attempt to facilitate the alignment via the co-attention mechanism under supervised settings, they suffer from lacking valid and accurate correspondences due to no annotation of such alignment. In this paper, we propose an approach with reinforcement learning (RL) over a cross-modal memory (CMM) to better align visual and textual features for radiology report generation. In detail, a shared memory is used to record the mappings between visual and textual information, and the proposed reinforced algorithm is performed to learn the signal from the reports to guide the cross-modal alignment even though such reports are not directly related to how images and texts are mapped. Experimental results on two English radiology report datasets, i.e., IU X-Ray and MIMIC-CXR, show the effectiveness of our approach, where the state-of-the-art results are achieved. We further conduct human evaluation and case study which confirm the validity of the reinforced algorithm in our approach.

## introduction radiology report generation aims to automatically generate a free-text description from a specific clinical radiograph (e.g., chest x-ray), which can significantly alleviate the burden of radiologists and thus improve the quality and standardization of health care. with the advantages of its applications, radiology report generation has become an † corresponding author. 1 our code for this paper is released at https:// github.com/cuhksz-nlp/r2genrl . interesting research topic attracted in both artificial intelligence and clinical medicine. recently, to generate more accurate reports, approaches based on deep learning techniques are adapted to this task and have achieved great success @xcite @xcite . to effectively perform radiology report generation, most existing studies adopted conventional encoder-decoder architectures with convolutional neural networks (cnn) as the encoder and recurrent neural networks (e.g., lstm @xcite , gru @xcite ) or non-recurrent neural networks (e.g., transformer @xcite ) as the decoder. considering that there is alignment between radiographs and their corresponding doctor-written reports (such as the mappings demonstrated in figure 1 where visual and textual features representing the same content are highlighted in the same color), the ability of a model to learn such alignment is the key to achieve outstanding performance. to model the alignment information, @xcite proposed a co-attention mechanism to explicitly learn the linking between visual features in the radiographs and the semantic information in the corresponding doctor-written text reports, where the model is trained to generate text sequences via maximum likelihood estimation (mle). however, one challenge to learn the alignment is that there is no annotated alignment for such research to perform supervised learning to accurately map cross-modal information, so that normal learning procedure may not fit this scenario. to address this challenge, reinforcement learning (rl) is a potential solution, because it is able to guide the learning process of the cross-modal alignment with appropriate supervision from carefully designed rewards. although there are studies following this paradigm by using rl to perform report generation, they focused on other aspects of this task rather than facilitating the mappings for cross-modal information. for example, @xcite designed sentence-and word-level rewards to guide the model to choose to either retrieve a template sentence or generate a new sentence, and @xcite ultilized multiagent rl to capture the imbalanced distribution between abnormality and normality. therefore, rl on cross-modal alignment is expected to be studied and has the potential for further improvements. in this paper, we propose to enhance radiology report generation via reinforced cross-modal alignment to alleviate the requirement of annotated supervision while facilitate the interactions across modalities (i.e., images and texts). in detail, our approach is based on @xcite , where a cross-modal memory (cmm) module is used to stores the cross-modal information that bridges the visual and textual features. based on cmm, the proposed rl algorithm is applied to leverage the signals from natural language generation (nlg) metrics, i.e., bleu @xcite , me-teor @xcite ) and rouge @xcite , to guide the cross-modal mappings so as to better matching features from images and texts as well as have a direct target of learning outcome for report generation. experimental results confirm the validity of our approach, which outperforms strong baselines and achieves the state-ofthe-art performance on two widely used benchmark datasets, i.e., iu x-ray (demner-fushman et al., 2016) and mimic-cxr @xcite . moreover, we perform human evaluation and case study to further illustrate the validity of the proposed rl in our approach. ## the proposed approach following previous studies @xcite @xcite , we treat radiology report generation as a sequenceto-sequence task, where the source sequence are patch features @xmath0, x 2 , ..., x s , ..., x s } from an input image, where x @xmath1, and the target sequence @xmath2, y 2 , ..., y t , ..., y t } is the corresponding report to the image, with y @xmath3, t the length of the report and v the vocabulary of all possible tokens. on the top of the general sequence-to-sequence paradigm, we add cmm which allows the proposed rl to take the signal from y and use it to guide cross-modal mappings for x and y. an overview of our proposed approach is presented in figure 2 , where the details for different parts are illustrated as follows. ## related work the task that is the most relevant to ours is image captioning, which aims to generation text captions that describe the content of the given images @xcite @xcite . being one of its applications and extensions to the medical domain, radiology report generation aims to depicting radiology images with professional texts @xcite @xcite @xcite @xcite . in general, existing approaches for radiology report generation were mainly designed and proposed to better align images and texts or to ## conclusion in this paper, we propose an rl approach based on cmm to better align visual and textual features for radiology report generation. in detail, a shared memory is used to store the cross-modal informa-tion and rl is applied to leverage the signals from nlg metrics to guide cross-modal mappings so as to better link features from images and texts. the experimental results on two benchmark datasets (i.e., iu x-ray and mimic-xcr) demonstrate the effectiveness of our model, which achieves the state-of-the-art performance on both datasets. human evaluation and the case study further confirm that our approach is able to generate high-quality reports with meaningful image-text alignment.