MISTI : Metadata-Informed Scientific Text and Image Representation through Contrastive Learning (2024)

Abstract: In scientific publications, automatic representations of figures and their captions can be used in NLP, computer vision, and information retrieval tasks. Contrastive learning has proven effective for creating such joint representations for natural scenes, but its application to scientific imagery and descriptions remains under-explored. Recent open-access publication datasets provide an opportunity to understand the effectiveness of this technique as well as evaluate the usefulness of additional metadata, which are available only in the scientific context. Here, we introduce MISTI, a novel model that uses contrastive learning to simultaneously learn the representation of figures, captions, and metadata, such as a paper’s title, sections, and curated concepts from the PubMed Open Access Subset. We evaluate our model on multiple information retrieval tasks, showing substantial improvements over baseline models. Notably, incorporating metadata doubled retrieval performance, achieving a Recall@1 of 30% on a 70K-item caption retrieval task. We qualitatively explore how metadata can be used to strategically retrieve distinctive representations of the same concept but for different sections, such as introduction and results. Additionally, we show that our model seamlessly handles out-of-domain tasks related to image segmentation. We share our dataset and methods (https://github.com/Khempawin/scientific-image-caption-pair/tree/section-attr) and outline future research directions.

## introduction the famous saying "a picture is worth a thousand words" takes on a new meaning in the context of scientific publications. scientific articles are filled with textual descriptions paired with images, diagrams, and figures that are essential for understanding the results presented. previous work on natural images, such as clip @xcite , has shown that contrastive learning can be used to jointly represent text and images. relative to traditionally available natural image datasets, scientific images are unique in two ways. first, they have precise textual descriptions, known as captions, which provide additional context and information. second, they have metadata, such as a paper's title, sections, and curated concepts, that enable us to contextualize an image. new open access repositories (e.g., @xcite ; @xcite ) allow us to take advantage of these unique characteristics. in this article, we propose and evaluate a novel model that uses contrastive learning to jointly represent text and images in scientific publications enriched with such metadata information. scientific images are often accompanied by captions that provide additional context and information @xcite . for example, a caption might describe the experimental setup, the results of an experiment, or the conclusions drawn from the results. captions are essential for understanding the research presented in scientific articles, but they are sometimes overlooked in image retrieval tasks. recent work has shown the potential of multimodal models @xcite , and their application to science is yet to be explored. in addition to captions, scientific images are often associated with metadata such as the paper's title, sections, and curated concepts. this metadata provides valuable information about the context in which the image appears and can be used to enhance the representation learned by the model. for example, the section of an article potentially provides information about the type of image (e.g., experimental setup, results, discussion) while the title can provide information about the general topic of the research. although previous work has explored this line of research @xcite , there is still much more to be done to fully leverage the range of metadata available in science. understanding the content of scientific diagrams and images and producing descriptions for them are challenging tasks. for instance, a diagram might contain multiple components with a specific function or meaning. captions provide additional context and information about the diagram but are often brief and may not fully explain its content. conversely, writing captions for scientific diagrams is challenging, requiring concisely conveying complex information. these challenges become apparent during information retrieval, where we want to retrieve figures based on a textual description or vice versa. building on the concepts of the clip model @xcite , we aim to build a vision text dual encoder model to facilitate scientific article generation. similar to what happened to clip, the model can be used for other downstream tasks, such as image segmentation or zero-shot classification. there are many image-caption datasets such as flickr8k, flickr30k @xcite , mscoco @xcite , rsicd @xcite , medicat (sanjay subramanian and hajishirzi, 2020) and other adaptations of image classification datasets namely cifar-10, and cifar-100 @xcite . these datasets are exceptional for their specific purposes, but the models do not generalize well into the scientific context. for instance, the captions and images in mscoco are mostly descriptions of everyday objects in everyday scenes. rsicd is a specialized dataset with captions of satellite images. medicat is a specialized dataset with captions in the medical domain. on the other hand, captions in scientific articles are domainspecific for a particular field, with complex images representing diagrams, charts, or graphs. apart from these datasets, there are image-caption scientific datasets with a diverse fields such as scicap @xcite , scimmir @xcite and sci-3000 @xcite . although these datasets are diverse but they lack the metadata associated with each image-caption pair which limits the context they belong to. another aspect that is limited is the size of these datasets which will affect the representations learned. to harness the richness found in images, we propose creating a new vision text dual encoder model to improve the performance of image retrieval tasks in scientific publications. we develop a dataset of scientific image captions based on open-access articles from pubmed open access subset @xcite . this dataset contains images and captions extracted from scientific arti-cles and metadata such as a paper's title, sections, and curated concepts. we evaluate the usefulness of our model and dataset on various tasks. in sum, the contributions of our work are as follows: ## dataset and methodology this work focuses on enhancing the representation learning of scientific texts and images through metadata-informed embeddings. our approach leverages scientific publications' unique vocabulary and metadata, such as concepts, keywords, and field-specific terminologies. to this end, we in-troduce a novel method that directly incorporates metadata into the training process of the contrastive learning models, and a specialized tokenizer to handle scientific metadata. this approach aims to improve the performance of embedding models on tasks such as zero-shot classification, information retrieval, and recommendation systems in the scientific domain. this section will describe the model architecture of contrastive learning with a dual vision-text encoder, the information retrieval system, the process of building the dataset, the training parameters, and the evaluation metrics for information retrieval. the process includes extracting imagecaption pairs from articles and metadata from ope-nalex. ## conclusion the goal of this project was to create representations of images and captions in scientific images based on the simultaneous learning of their relationship using contrastive learning. this representation can be applied to multiple tasks, such as improving the understanding of figures and textual descriptions of scientific concepts and helping generate better images and captions. our zero-shot segmentation example demonstrates the numerous downstream tasks where our model can be applied. the substantial performance differences between the generic image-text models and our metadatainformed model show that more work is needed in the scientific domain. for future work, additional metadata such as authors, publication venues, and citations might be explored further to augment the data.