Fine-grained Medical Vision-Language Representation Learning for Radiology Report Generation (2023)

Abstract: Given the input radiology images, the objective of radiology report generation is to produce accurate and comprehensive medical reports, which typically include multiple descriptive clinical sentences associated with different phenotypes. Most existing works have relied on a pre-trained vision encoder to extract the visual representations of the images. In this study, we propose a phenotype-driven medical vision-language representation learning framework to efficiently bridge the gap between visual and textual modalities for improved text-oriented generation. In contrast to conventional methods which learn medical vision-language representations by contrasting images with entire reports, our approach learns more fine-grained representations by contrasting images with each sentence within the reports. The learned fine-grained representations can be used to improve radiology report generation. The experiments on two widely-used datasets MIMIC-CXR and IU X-ray demonstrate that our method can achieve promising performances and substantially outperform the conventional vision-language representation learning methods.

## introduction medical images, such as those from radiology and pathology, and medical reports, which consist of multiple clinical sentences describing both the normalities and abnormalities in the medical images, are frequently used for disease diagnosis and treatment @xcite . therefore, medical report generation has the potential to reduce the heavy workload of experienced radiologists in report writing and remind inexperienced radiologists of abnormalities @xcite . similar to the task of image captioning @xcite , which aims to describe the visual content in the images, lots of encoder-decoder-based medical report generation models are proposed @xcite @xcite @xcite . in the encoding * both authors contributed equally to this paper. in implementation, existing contrastive learning models first extract the visual representations of images and textual representations of entire reports, and then pre-train models by contrasting the visual representations with the textual representations. in this study, we (i) first construct a set of phenotypes; (ii) introduce an attention mechanism to transform each sentence within the reports into phenotype-based textual representations; (iii) adopt the phenotype-based textual representations to extract phenotype-based visual representations; (iv) and per-train the model by contrasting the two types of phenotype-based representations. in this way, by splitting the entire report into multiple sentences, phenotypeclip not only learns fine-grained representations, but also scales up contrasting learning, resulting in boosting the downstream medical report generation task. we perform the experiments on two benchmark datasets, i.e., mimic-cxr @xcite and iu x-ray @xcite . the results validate the effectiveness of phenotypeclip, which substantially outperforms the conventional contrastive learning methods on all widely used evaluation metrics. overall, the main contributions of this paper are: ## related work we will introduce our related work from medical report generation and contrastive learning. ## approach in this section, we introduce the proposal for medical report generation in detail. ## experiments we first describe two benchmark datasets, the metrics, and the settings used for evaluation. then, we present the main results and analysis of our proposal on the two datasets. @xcite 0.088 0.118 0.266 0.084 0.068 -0.308 0.295 topdown @xcite 0.074 -0.250 0.073 ----transformer @xcite 0.090 0.125 0.265 -0.135 0.164 0.342 -r2gen @xcite ## conclusion in this work, we propose a phenotype-based (finegrained) contrastive learning framework, pheno-typeclip, for medical imaging report generation. the proposed phenotypeclip can efficiently bridge the gap between visual and textual modalities to provide a solid basis for generating accurate medical reports. in the implementation, pheno-typeclip learns fine-grained representations, i.e., phenotype-based representations, by performing contrastive learning on the image and each sentence within the report. experiments on two widelyused datasets mimic-cxr and iu x-ray datasets prove our arguments and show that phenotypeclip achieves competitive results with previous state-ofthe-art methods, especially exhibiting a remarkable improvement over conventional contrastive learning techniques across all evaluation metrics. in future research, it is interesting to explore extracting the topics or keywords, e.g., abnormalities @xcite , from the reports to build the set of phenotypes.