M ed IC a T : A Dataset of Medical Images, Captions, and Textual References (2020)

Abstract: Understanding the relationship between figures and text is key to scientific document understanding. Medical figures in particular are quite complex, often consisting of several subfigures (75% of figures in our dataset), with detailed text describing their content. Previous work studying figures in scientific papers focused on classifying figure content rather than understanding how images relate to the text. To address challenges in figure retrieval and figure-to-text alignment, we introduce MedICaT, a dataset of medical images in context. MedICaT consists of 217K images from 131K open access biomedical papers, and includes captions, inline references for 74% of figures, and manually annotated subfigures and subcaptions for a subset of figures. Using MedICaT, we introduce the task of subfigure to subcaption alignment in compound figures and demonstrate the utility of inline references in image-text matching. Our data and code can be accessed at https://github.com/allenai/medicat .

## introduction scientific document understanding necessitates the analysis of various components of scientific papers, including recognizing relationships between the text, figures, and references of papers. in the medical domain, connections between the text and figures in a paper are useful to enable the retrieval of figures via textual queries and to produce systems that are capable of analyzing and understanding medical images. modern academic search engines are able to extract figures and associated captions from papers, so that queries can be matched against captions to retrieve relevant images. however, scientific figures often contain subfigures (40% of figures in pubmed @xcite ), and for a given query, only some of these may be relevant. consider a user searching "lung cyst ct"; ideally, the user would be shown just parts (b) and (d) from figure 1 . but selecting relevant subfigures requires finding and aligning subcaptions with subfigures, and current systems lack this ability. existing largescale datasets @xcite explicitly exclude compound figures, while datasets with subfigure annotations @xcite do not provide annotations for aligning subfigures with subcaptions, and therefore cannot support subfigure retrieval. images and textual descriptions in medical papers are also useful for developing systems for automated medical image analysis. previous work @xcite has used such data for tasks such as captioning and concept tagging, but the textual descriptions used in their work are limited to captions. important details about figures are also often available in the main body of a paper, as in figure 1 , where an inline reference provides additional context that the ct is of the thorax and that the cysts seen are thin-walled. these inline references, if available, are useful for training medical image analysis systems. we introduce medicat, a dataset of medical figures, captions, subfigures/subcaptions, and inline references that enables the study of these figures in context.our dataset includes compound figures (75% of figures) with manual annotations of subfigures and subcaptions (7507 subcaptions for 2069 compound figures). using medicat, we introduce the task of subfigure-subcaption alignment, a challenging task due to high variation in how subfigures are referenced. we provide a strong baseline model based on a pre-trained transformer that obtains an f1 score of 0.674, relative to an estimated inter-annotator agreement of 0.89. med-icat is also richer than previous datasets in that it includes inline references for 74% of figures in the dataset. we show that training on references in addition to captions improves performance on the task of matching images with their associated descriptions. by providing this rich set of relationships between figures and text in papers, our dataset enables the study of scientific figures in context. ## the medicat dataset we extract figures and captions from open access papers in pubmed central using the results of @xcite . to add inline references, we match extracted figures to corresponding figures in the publicly-available s2orc corpus @xcite , then extract inline references for these figures from the s2orc full text (see appendix a for details). we exclude figures found in roco @xcite to create a disjoint dataset, though we identify and release inline references for roco figures. ## image-text matching to demonstrate the utility of inline references in medicat, we conduct experiments in image-text matching @xcite , a task that has been studied extensively in the domain of natural images. given a piece of text, the system's goal is to return the matching image from a database. like image captioning, this task assesses the model's ability to align the image and text modalities, but the matching task avoids the issue of evaluating generated text @xcite . in the context of medical figures and captions from papers, we attempt to retrieve a corresponding figure given its caption. since our dataset provides more than one textual description of each image via inline references, we analyze the benefit of using references as additional training data, and demonstrate improvements over models trained on captions only. at test time, only captions are used to enable a fair comparison of the models. we use a model similar to @xcite with a different token type embedding for inline references. linking with roco roco is a dataset of non-compound radiology figures and captions extracted from literature @xcite . 9 the roco dataset consists of around 82k figures (train/validation/test splits: 65k, 8175, 8177 respectively). we identify papers associated with figures in the roco dataset and extract the associated inline references, and release these as part of med-icat. we find inline references for approximately 25k figures in the roco dataset (approximately 21k in train and 4k each in validation and test splits). model our model follows @xcite with a few modifications. we tokenize the input text and pass the token embeddings through a bert encoder @xcite . the bert encoder is initialized with scibert weights, and we use scibert uncased tokenization @xcite . pre-trained weights improve performance on the image-text matching task considerably. we insert the visual representations, projected into the hidden state dimensionality, as extra hidden states in the middle of the encoder (layer 6). in contrast to @xcite , we do not use an object detector to find regions of interest in the image, since objects tend to be sparse in medical images. instead, the visual representation is obtained by an affine transformation of the feature vector produced for the entire image by a resnet-50 network pretrained on imagenet. 10 the other noteworthy difference in our model is that we use different token type embeddings for inline references and captions, since inline references are different in style and content from captions. during training, for each piece of text, the model is given the correct image as well as two other images (negative images) sampled uniformly at random from all images. the training objective is choosing the correct one of these three images. the validation accuracy is measured on the same task, except 20 negative images are sampled for each piece of text. details on hyperparameter tuning and training are provided in appendix d. ## discussion medicat allows medical figures to be studied in the context of their source papers by providing links between subfigures and subcaptions and between figures and inline references. we propose the task of subfigure-subcaption alignment and provide a strong baseline model, and we demonstrate the utility of inline references for image-text matching. medicat can also benefit other medical visionlanguage tasks (e.g. captioning, vqa). pretraining techniques that have worked well for these problems in the general domain by using large numbers of aligned text and images @xcite can leverage the aligned data in medicat. the techniques we use to construct medicat can also be extended beyond "medical images" to study the relationships between figures and text in scientific documents from other domains. daekeun you, sameer antani, dina demner-fushman, venu govindaraju, and george r thoma. 2011. detecting figure-panel labels in medical journal articles using mrf. in 2011 international conference on document analysis and recognition, pages 967-971. ieee. luowei zhou, hamid palangi, lei zhang, houdong hu, jason j corso, and jianfeng gao. 2020. unified vision-language pre-training for image captioning and vqa. aaai.