Competence-based Multimodal Curriculum Learning for Medical Report Generation (2021)

Abstract: Medical report generation task, which targets to produce long and coherent descriptions of medical images, has attracted growing research interests recently. Different from the general image captioning tasks, medical report generation is more challenging for data-driven neural models. This is mainly due to 1) the serious data bias and 2) the limited medical data. To alleviate the data bias and make best use of available data, we propose a Competence-based Multimodal Curriculum Learning framework (CMCL). Specifically, CMCL simulates the learning process of radiologists and optimizes the model in a step by step manner. Firstly, CMCL estimates the difficulty of each training instance and evaluates the competence of current model; Secondly, CMCL selects the most suitable batch of training instances considering current model competence. By iterating above two steps, CMCL can gradually improve the model’s performance. The experiments on the public IU-Xray and MIMIC-CXR datasets show that CMCL can be incorporated into existing models to improve their performance.

## introduction medical images, e.g., radiology and pathology images, and their corresponding reports, which describe the observations in details of both normal and abnormal regions, are widely-used for diagnosis and treatment @xcite . in clinical practice, writing a medical report can be time-consuming and tedious for experienced radiologists, and error-prone for inexperienced radiologists. therefore, automatically generating medical reports can assist radiologists in clinical decision-making and emerge as a prominent attractive research direction in both artificial ground truth: co-attention: ours: the heart is enlarged. there is no pneumothorax. no acute bony abnormality. there is a moderate right pleural effusion with associated atelectasis. the left lung is clear. no pneumothorax is seen. the heart and mediastinum are normal. the lungs are clear. 1 there is mild blunting of the right costophrenic xxxx. there is no infiltrate, mass or pneumothorax. the right internal jugular catheter has been removed. ## framework in this section, we briefly describe typical medical report generation approaches and introduce the proposed competence-based multimodal curriculum learning (cmcl). as shown in the top of figure 2 , many medical report generation models adopt the encoderdecoder manner. firstly, the visual features are extracted from the input medical image via a cnn model. then the visual features are fed into a sequence generation model, like lstm to produce the medical report. in the training phase, all training instances are randomly shuffled and grouped into batches for training. in other words, all training instances are treated equally. different from typical medical report generation models, cmcl builds the training batch in a selective manner. the middle part of figure 2 displays the framework of cmcl equipped with one single difficulty metric. cmcl first ranks all training instances according to this difficulty metric and then gradually enlarges the range of training instances that the batch is selected. in this manner, cmcl can train the models from easy to difficult instances. since medical report generation involves multimodal data, like visual medical images and textual reports, it is hard to design one single metric to estimate the overall difficulty. therefore, we also propose a cmcl with multiple difficulty metrics. as shown in the bottom of figure 2 , the training instances are ranked by multiple metrics independently. at each step, cmcl generates one batch for each difficulty metric and then calculates the perplexity of each batch based on current model. the batch with highest perplexity is selected to train the model. it can be understood that cmcl sets multiple syllabus in parallel, and the model is optimized towards the one with lowest competence. ## difficulty metrics in this section, we define the difficulty metrics used by cmcl. as stated in section 2, the key challenge of medical report generation is to accurately capture and describe the abnormalities @xcite @xcite . there- random shuffle batch sampled uniformly dataset ranking by single difficulty metrics batch sampled by model competence dataset ranking by difficulty metrics 1 ranking by difficulty metrics 2 ranking by difficulty metrics 3 ranking by difficulty metrics n batch sampled by model perplexity and competence fore, we assess the difficulty of instances based on the difficulty of accurately capturing and describing the abnormalities. ## approach in this section, we first briefly introduce the conventional single difficulty-based curriculum @xcite . then we propose the multiple difficulty-based curriculum learning for medical report generation. ## experiment we firstly describe two public datasets as well as the widely-used metrics, baselines and settings. then we present the evaluation of our cmcl. ## conclusion in this paper, we propose the novel competencebased multimodal curriculum learning framework (cmcl) to alleviate the data bias by efficiently utilizing the limited medical data for medical report generation. to this end, considering the difficulty of accurately capturing and describing the abnormalities, we first assess four sample difficulties of training data from the visual complexity and the textual complexity, resulting in four different curricula. next, cmcl enables the model to be trained with the appropriate curricula and gradually proceed from easy samples to more complex ones in training. experimental results demonstrate the effectiveness and the generalization capabilities of cmcl, which consistently boosts the performance of the baselines under most metrics.