Cross-modal Contrastive Attention Model for Medical Report Generation (2022)

Abstract: Medical report automatic generation has gained increasing interest recently as a way to help radiologists write reports more efficiently. However, this image-to-text task is rather challenging due to the typical data biases: 1) Normal physiological structures dominate the images, with only tiny abnormalities; 2) Normal descriptions accordingly dominate the reports. Existing methods have attempted to solve these problems, but they neglect to exploit useful information from similar historical cases. In this paper, we propose a novel Cross-modal Contrastive Attention (CMCA) model to capture both visual and semantic information from similar cases, with mainly two modules: a Visual Contrastive Attention Module for refining the unique abnormal regions compared to the retrieved case images; a Cross-modal Attention Module for matching the positive semantic information from the case reports. Extensive experiments on two widely-used benchmarks, IU X-Ray and MIMIC-CXR, demonstrate that the proposed model outperforms the state-of-the-art methods on almost all metrics. Further analyses also validate that our proposed model is able to improve the reports with more accurate abnormal findings and richer descriptions.

## introduction medical report generation task in practice demands radiologists carefully examine details of images and write corresponding reports, which is timeconsuming and technically rigorous. in addition, with the explosion of medical images, generating medical reports has increasingly become a tough burden for radiologists in clinical diagnosis and treatment. thus, it is extremely desired for automatically generating medical reports from medical images, which has also attracted increasing attention especially in the chest x-ray report generation. recently, the widely used encoder-decoder framework in image captioning task (karpathy and figure 1 : one example of chest x-ray image with the corresponding ground truth, baseline @xcite and our model generated reports. the abnormal regions and their corresponding descriptions are marked in same colors, showing serious data biases of this task. @xcite @xcite has been successfully inherited by medical report generation and has made great improvements @xcite @xcite @xcite . nevertheless, as shown in figure 1 , different from image captioning, medical report generation faces typical data biases which cause the failing of generating accurate descriptions: 1) the abnormal regions are usually tiny, rare and hard to be recognized in medical images with monotonous and homogeneous features @xcite @xcite ; 2) the abnormal text descriptions are correspondingly rare in reports and the normal descriptions dominate the whole datasets @xcite @xcite . to tackle these issues, jing et al. @xcite employed an auxiliary detector to identify abnormality terms. liu et al. @xcite compared the input image with normal samples to distill the visual abnormal information. however, these approaches mainly probed abnormalities from images themselves or comparing with manual selected normal samples, without considering the importance of exploiting abnormal information from historical similar cases and making use of their visual and semantic information. based on the observation that similar images are more likely to have similar reports @xcite , we presume that taking the most similar historical case as a contrastive reference will make models able to relieve the data biases and capture more critical visual and semantic information. unfortunately, exploring positive semantic information from cases faces another challenge on cross-modal matching @xcite : the retrieved report which contains useful but noisy semantic information is hard to be aligned with the input image solely across the unmatched visualsemantic modalities. in this paper, we propose a novel cross-modal contrastive attention (cmca) model to tackle the aforementioned problems. cmca first retrieves the most similar case for the input image from a historical database, then generates a contrastive feature by enlarging the differences and maintaining the commons between the input image and the retrieved image. subsequently, the contrastive feature is used to extract visually abnormal and semantically matched information through two modules: a visual contrastive attention module (vcam) and a cross-modal attention module (cam). specifically, vcam extracts discriminative abnormal visual information from the contrastive feature, where the unique abnormal regions of input image are enhanced and similar regions are retained. cam matches the positive semantic information from the retrieved report by aligning it with the contrastive feature, which builds interactions across the unmatched visual and semantic modalities. finally, we propose a parallel attention module (pam) to further enhance the feature representation for generating accurate report. extensive experiments on the widely-used benchmark iu x-ray @xcite dataset and the largest public mimic-cxr @xcite dataset demonstrate that our model outperforms the state-of-the-art methods. overall, our contributions are as follows: ## method in this section, we introduce the background and the details of the proposed cmca model in order.