Weakly Supervised Contrastive Learning for Chest X -Ray Report Generation (2021)

Abstract: Radiology report generation aims at generating descriptive text from radiology images automatically, which may present an opportunity to improve radiology reporting and interpretation. A typical setting consists of training encoder-decoder models on image-report pairs with a cross entropy loss, which struggles to generate informative sentences for clinical diagnoses since normal findings dominate the datasets. To tackle this challenge and encourage more clinically-accurate text outputs, we propose a novel weakly supervised contrastive loss for medical report generation. Experimental results demonstrate that our method benefits from contrasting target reports with incorrect but semantically-close ones. It outperforms previous work on both clinical correctness and text generation metrics for two public benchmarks.

## introduction automated radiology report generation aims at generating informative text from radiologic image studies. it could potentially improve radiology reporting and alleviate the workload of radiologists. recently, following the success of deep learning in conditional text generation tasks such as image captioning @xcite , many methods have been proposed for this task @xcite @xcite @xcite . unlike conventional image captioning benchmarks (e.g. ms-coco @xcite ) where referenced captions are usually short, radiology reports are much longer with multiple sentences, which pose higher requirements for information selection, relation extraction, and content ordering. to generate informative text from a radiology image study, a caption model is required to understand the content, identify abnormal positions in an image and organize the wording to describe findings in images. however, the standard approach of training an encoder-decoder model with teacher forcing and cross-entropy loss often leads to text generation outputs with high frequency tokens or sentences appearing too often @xcite . this problem could be worse for chest x-ray report generation, since the task has a relatively narrow text distribution with domainspecific terminology and descriptions for medical images, and models often struggle to generate long and diverse reports @xcite . to tackle these challenging issues, we propose to introduce contrastive learning into chest x-ray report generation. however, simply using random non-target sequences as negative examples in a contrastive framework is suboptimal @xcite , as random samples are usually easy to distinguish from the correct ones. hence, we further introduce a weakly supervised contrastive loss that assigns more weights to reports that are semantically close to the target. by exposing the model to these "hard" negative examples during training, it could learn robust representations which capture the essence of a medical image and generate high-quality reports with improved performance on clinical correctness for unseen images. overall, our contributions are three-fold: medical report generation medical report generation, which aims to automatically generate descriptions for clinical radiographs (e.g. chest xrays), has drawn attention in both the machine learning and medical communities. many methods have been proposed to solve this task. for example, jing et al. (2017) proposed a co-attention hierarchical cnn-rnn model. li et al. (2018, 2019) used hybrid retrieval-generation models to complement generation. most recently, ni et al. (2020) proposed to learn visual semantic embeddings for cross-modal retrieval in a contrastive setting, and showed improvements in identifying abnormal findings. however, this is hard to scale or generalize since we need to build template abnormal sentences for new datasets. chen et al. (2020b) leveraged a memory-augmented transformer model to improve the ability to generate long and coherent text, but didn't address the issue of generating dominant normal findings specifically. our work proposes to incorporate contrastive learning into training a generation-based model, which benefits from the contrastive loss to encourage diversity and is also easy to scale compared to retrieval-based methods. ## conclusion in this paper, we present a weakly supervised contrastive learning framework for generating chest x-ray reports. our contrastive loss could lead to better results on both clinical correctness and text generation metrics than previous methods. we also show that exposing the model to semantically-close negative samples improves generation performance. in the future, we will extend our method to other medical image datasets other than chest x-rays.