F eature D ifference M akes S ense: A medical image captioning model exploiting feature difference and tag information (2020)

Abstract: Medical image captioning can reduce the workload of physicians and save time and expense by automatically generating reports. However, current datasets are small and limited, creating additional challenges for researchers. In this study, we propose a feature difference and tag information combined long short-term memory (LSTM) model for chest x-ray report generation. A feature vector extracted from the image conveys visual information, but its ability to describe the image is limited. Other image captioning studies exhibited improved performance by exploiting feature differences, so the proposed model also utilizes them. First, we propose a difference and tag (DiTag) model containing the difference between the patient and normal images. Then, we propose a multi-difference and tag (mDiTag) model that also contains information about low-level differences, such as contrast, texture, and localized area. Evaluation of the proposed models demonstrates that the mDiTag model provides more information to generate captions and outperforms all other models.

## introduction image captioning is a research area that generates text describing natural images, representing a convergence of computer vision and natural language processing. there are several existing methods for image captioning. one way involves filling up templates with detected objects or properties @xcite , but this has limitations about diversity. especially, sentences describing abnormal findings in medical images are relatively diverse and rare. another involves retrieving the captions of images that are similar to the query image and selecting relevant phrases from those captions to generate new captions @xcite . however, this method does not generalize well when applied to unfamiliar images. to overcome the weaknesses of current methods, we adopted the encoder-decoder architecture with an attention mechanism. the encoder encodes an image into a feature vector, and the decoder decodes the feature vector into text. the encoder-decoder is one of the neural networks successfully used in other recent image captioning studies @xcite @xcite @xcite . medical image captioning is the field of generating medical reports that describe medical images, as shown in figure 1 . the first challenge in medical image captioning is the lack of quality in training sets. researchers have difficulty accessing chest x-ray datasets, which slows the development of related technologies. there are publicly available datasets that have images and reports: iu x-ray, peir gross, and iclef-caption @xcite . using only these datasets, state-of-the-art caption generation models do not generate medical reports correctly. recently, mimic-cxr @xcite , the largest dataset with images, reports, and labels, is released. the second challenge is that there are too many normal descriptions in the dataset, which creates a skewed dataset that poses problems for supervised learning. besides, some types of significant abnormal findings appear too rarely in the dataset to appropriately train the model. in this study, we propose a model that can identify and focus on abnormal findings more specifically and precisely, similar to the way that physicians would typically read, interpret, and write chest x-ray reports. since physicians look for the differences between the normal group and the disease group, we also focus on image feature differences. therefore, the proposed model sets the criteria based on a normal x-ray image and creates a feature difference vector that explains the difference between a normal x-ray image and a patient's x-ray image. this feature difference vector is a subtraction of visual feature vectors extracted from the two images. to improve the model, we also exploit tag information obtained from the medical report. tags provide important information about the images and also convey meaningful semantics to the decoder. several previous studies @xcite @xcite show methods that leverage feature vectors of images to account for differences between two images. next, since physicians obtain information not only from the overall image but also from the localized lesion areas, we consider that each convolutional level would also convey meaningful details such as contrast, texture, and localized area. therefore, another proposed model fully exploits information contained in each layer. previous studies @xcite @xcite analyze and interpret convolutional neural networks (cnns) utilizing feature vectors extracted from lower convolutional layers. the following section describes the organization of the dataset, and section 3 introduces the baseline and our proposed models. section 4 provides the experimental settings and results with analysis, and draws some conclusions in section 5. ## dataset this study uses iu x-ray, which consists of a series of image-text-tag triplets. this dataset is anonymous and is from the open access biomedical image search engine (openi) foot_0 @xcite . the 7,470 chest x-ray images have two views: posteroanterior (pa) and lateral. the baseline model uses all images, but the proposed model uses only 3,821 images, which are pa views. the report corresponding to each image has four sections: comparison, indication, findings, and impression. the output of the model is a concatenation of the findings and the impression section @xcite . the findings section describes observations in each area of the body, and the most crucial impression section explains the problem and then provides a diagnosis. the output excludes the comparison and indication sections, which contain patient information and symptoms. one or more tags are automatically extracted from each report using the medical text indexer (mti) foot_1 program @xcite . mti produces index recommendations based on medical subject heading (mesh) foot_2 terms. there are a total of 210 unique tags, with an average of 2 tags per image. without the normal tag, there is an average of 25 images per tag. class imbalance arises because 1,502 images contain normal tag, so we randomly sample 75 images for a better balance between tags. the tags still have a class imbalance because the scope is too broad, making the term rare. the prepared datasets are 3,821 image-text-tag triplets, all pa view images. after adjusting the number of images with the normal tag, we use random selection to get 1,911, 238, and 245 triplets for the training, validation, and test sets. ## conclusion we propose models that exploit feature differences and tag information. as expected, the model that uses low-level convolutional features from the cnn model can convey low-level details, such as contrast, texture, and localized area. some of our models outperform the conventional image captioning models in terms of bleu score, rouge-l, and cider. the mditag(-) model performs best according to every metric. based on these experiments, we can conclude that the feature differences between images and semantic tags are crucial elements necessary for training. in the future, we will strengthen tags that contain semantic information to extract keywords for more accurate information, such as disease information, location, and size. furthermore, improving the accuracy of multiple tag prediction is crucial to deliver semantic facts accurately. we are also considering obtaining more images from hospitals to reduce the proportion of abnormal images in the datasets.