A Self-training Framework for Automated Medical Report Generation (2023)

Abstract: Medical report generation, focusing on automatically generating accurate clinical findings from medical images, is an important medical artificial intelligence task. It reduces the workload of physicians in writing reports. Many of the current methods depend heavily on labeled datasets that include a large amount of image-report pairs, but such datasets labeled by physicians are hard to acquire in clinical practice. To this end, in this paper, we introduce a self-training framework named REMOTE (i.e., Revisiting sElf-training for Medical repOrT gEneration) to exploit the unlabeled medical images and a reference-free evaluation metric MedCLIPScore to augment a small-scale medical report generation dataset for training accurate medical report generation model. Experiments and analysis conducted on the MIMIC-CXR and IU-Xray benchmark datasets demonstrate that, our REMOTE framework, using 1% labeled training data, achieves competitive performance with previous fully-supervised models that are trained on entire training data.

## introduction generating medical reports automatically involves producing clinical descriptions based on the input visual medical images @xcite @xcite . this is similar to the task of image captioning @xcite , which aims to generate visual descriptions to describe the input images. therefore, based on the benchmark dataset mimic-cxr @xcite , inspired by the success of image captioning, various state-of-the-art data-driven models, especially those based on the encoder-decoder structure @xcite @xcite , have achieved significant advancements. however, medical data labeling requires specialized expertise from physicians and also involves privacy concerns. therefore, acquiring medical report generation datasets is time-consuming and costly @xcite . as a result, when compared to datasets used for general image captioning datasets such as conceptual captions @xcite , the size of the medical dataset mimic-cxr is relatively small. this size limitation becomes a challenge when dealing with novel diseases like covid-19, where collecting and labeling adequate training data promptly is difficult. it hinders the application of existing medical report generation models in addressing novel diseases to alleviate the workload of physicians efficiently. considering that there are a lot of public imageonly datasets, e.g., chexpert @xcite , rsna pneumonia @xcite , covid images @xcite , in the literature. to this end, we propose a self-training framework re-mote, which enhances the performance of the medical report generation model by simultaneously utilizing high-quality paired image-report datasets and image-only datasets. in implementation, we adopt the noisy student self-training framework @xcite as the basis to build our remote for medical report generation, which consists of a "teacher" model and a "student" model. it begins by training a "teacher" model on the high-quality annotated image-report pairs, e.g., mimic-cxr dataset @xcite . subsequently, the teacher model is used to generate pseudo-reports for medical images in the imageonly dataset without annotated reports. we then employ medclipscore to score each generated pseudo image-report pair and filter out low-scoring pseudo image-report pairs. finally, we train a "student" model on both the annotated high-quality image-report pairs and the generated pseudo imagereport pairs. in the next training step, we consider the "student" model as the new "teacher" model, and by repeating the above steps, we can generate new pseudo image-report pairs and train new "student" models. through iterating these steps, we ultimately obtain an accurate and robust medical report generation model. it is worth noting that, while the self-training framework has been explored in uni-modal tasks such as image classification @xcite and machine translation @xcite , self-training in medical report generation has not been well explored. this is because medical report generation is a multi-modal medical task, incorporating disparities between the visual and the textual modalities. thus, inspired by the great success of clip @xcite , which is trained to align image and text modalities, we follow the clipscore @xcite to construct the medclipscore to obtain a high-quality pseudo image-report pairs. in detail, we train the medclip @xcite on the mimic-cxr dataset, and use it as med-clipscore to boost the performance and robustness of the medical report generation model. overall, the main contributions of this paper are as follows: ## approach in this section, we will introduce the core three components of our approach, i.e., the "teacher" model, medclipscore, and the "student" model. ## experiments we first describe a benchmark dataset, our built dataset, the metrics, and the settings used for evaluation. then, we present the results of our approach. ## analysis we will provide quantitative and qualitative analyses to understand our method. ## conclusion in this work, we presented a novel self-training framework, remote, aimed at boosting the performance of medical report generation from unlabeled visual medical images, especially in scenarios with limited labeled training data. by harnessing the power of both high-quality paired imagereport datasets and unlabeled image-only datasets, our approach effectively reduces the reliance on extensive labeled training data, which are both timeconsuming and costly to obtain. our remote introduces a "teacher" model and a "student" model to generate pseudo image-report pairs and refine the generation, respectively. specifically, we introduced medclipscore to ensure the quality of the generated pseudo image-report pairs. the experiments on the widely-used benchmark dataset mimic-cxr and a covid-19 dataset validated the robustness and effectiveness of remote. in particular, using only 1% of labeled training data, our approach could achieve competitive performances comparable to fully-supervised state-ofthe-art methods.