S ci MMIR : Benchmarking Scientific Multi-modal Information Retrieval (2024)

Abstract: Multi-modal information retrieval (MMIR) is a rapidly evolving field where significant progress has been made through advanced representation learning and cross-modality alignment research, particularly in image-text pairing.However, current benchmarks for evaluating MMIR performance on image-text pairings overlook the scientific domain, which has a notable gap with the generic data since the caption of scientific charts and tables usually describes the analysis of experimental results or scientific principles in contrast to human activity or scenery depicted in generic images.To bridge this gap, we develop a sci entific domain-specific MMIR benchmark ( SciMMIR ) by leveraging open-access research paper corpora to extract data relevant to the scientific domain. This benchmark comprises 530K meticulously curated image-text pairs, extracted from figures and tables with detailed captions from scientific documents.We further annotate the image-text pairs with a two-level subset-subcategory hierarchy to facilitate a more comprehensive evaluation of the baselines. We conduct zero-shot and fine-tuned evaluations on prominent multi-modal image-captioning and visual language models, such as CLIP, BLIP, and BLIP-2.Our findings offer critical insights for MMIR in the scientific domain, including the impact of pre-training and fine-tuning settings and the effects of different visual and textual encoders.

## introduction information retrieval (ir) systems are expected to provide a relevant piece of information from a vast, yet organised, collection of data, according to given user queries. with the advancement of representation learning @xcite , the methodological paradigm of ir systems has evolved from using lexical matching to retrieve textual data @xcite @xcite to a mixture of similarity matching approaches in a learned representation space, consequently supporting additional modalities such as images and audio, alongside text @xcite @xcite . in scientific domains, offering users a finegrained multi-modal retrieval service presents considerable practical significance. although previous studies have evaluated the image-text retrieval task across a range of general topics on large-scale datasets such as wikipedia @xcite @xcite @xcite , there is a notable research gap in comprehensively assessing mmir models within scientific domains, specifically. integrating both in-domain and out-of-domain data in the pre-training phase significantly boosts the performance of visual language models (vlms) on downstream tasks. however, the training of most vlms has focused exclusively on common generic topics concerning the mundane events of daily life @xcite , such as images depicting scenery and human activities. as a result, this pre-training overlooks data pertinent to scientific domains such as elements related to model architectures, illustrations of scientific principles, and the results of experiments. due to the substantial differences in the data distribution characteristics between generic data and scientific data, many vlms may not have an adequate ability to perform mmir for scientific domains. additionally, existing table-related works, such as table generation tasks, have mainly focused on textual representations of tables, while overlooking image-based representations of tabular data. this presents problems for human-computer interaction, as users may desire to input information in the form of screenshots and expect an interactive system to present results in a graphical format. to address the aforementioned research gap, we introduce scimmir, a scientific multi-modal information retrieval benchmark. scimmir (outlined in figure 1 ) is the first benchmark to comprehensively evaluate a model's mmir ability in the scientific domain. to build our data collection, we retrieve images of figures and tables, and their associated captions, from scholarly documents available on arxiv, an open-access archival corpus, to construct image-text pairs. in order to comprehensively evaluate the cross-modality aligned representations learned by models, our scimmir benchmark defines the retrieval task as bi-directional, involving searching for the correct textual caption in a candidate pool from a given image (@xmath0. given the disparity among various data types, we contend that achieving uniform model performance across diverse data formats is challenging. for example, a model may excel at retrieving data related to experimental results but demonstrate average performance regarding data related to model architectures. if an overall improvement is sought for the performance of vlms, this improvement may not be observed in specific sub-domains of information. consequently, such improvements do not necessarily translate into observable boosts to a vlm's performance for a specific use case. as a result, we annotate and categorise the imagetext pairs into three figure-caption and two tablecaption subcategories based on the type of content they describe (such as experimental results, model architectures, and scientific principles). we then conduct fine-grained evaluation on each subset. by analysing performance across subcategories, we are better able to carry out targeted improvements to a model for a specific subcategory of interest. to explore the mmir capabilities of existing image captioning models and vlms in scientific domains, as well as different subcategories, we conduct extensive experiments in both zero-shot and fine-tuned settings across various subcategories. furthermore, we extract ocr-text data from the images and investigate its influence on the performance of vlms. we present our key insights as follows: these findings underscore the importance of tailored approaches for different data types within the scientific mmir framework. a more in-depth exploration of these findings is given in §5. ## dataset construction data collection. we collect pdf files from a 6-month period (i.e. papers submitted between may and october 2023) from arxiv using the official api foot_0 . we use an open-source tool @xcite to locate non-textual elements (i.e., figures and tables) in the papers and extract their corresponding caption text. all tables and figures are stored in the form of images, and we remove the figure/table entries that have empty captions. the aforementioned collection process results in cs 52.5% astro-ph 10.4% physics 6.9% cond-mat 5.8% math 5.4% eess 4.7% stat 3.0% quant-ph 2.9% hep-ph 2.0% gr-qc 1.2% others 5.2% figure 2: the ratio of different subject image-caption data in scimmir. 'hep-ph' denotes high energy physics -theory, 'quant-ph' denotes quantum physics, and 'astro-ph' denotes astrophysics. 'gr-qc' denotes general relativity and quantum cosmology, 'eess' denotes electrical engineering and systems science, 'cond-mat' denotes condensed matter, and 'cs' denotes computer science. the scimmir dataset that comprises 530k imagecaption samples, with an average caption length of 43.19 words as shown in ## experiment 4.1 retrieval baseline to investigate the capabilities of current vlms on the scimmir task and to assess whether data from different categories influences their performance, we evaluate a wide range of baseline models. furthermore, we collect information regarding the pretraining strategy for each baseline model in ## conclusion in summary, we introduce a novel benchmark and a corresponding dataset designed to address the gap in evaluating multi-modal information retrieval (mmir) models in the scientific domain. additionally, we categorise the images into fine-grained subcategories based on the characteristics of the figures and tables to facilitate a more comprehensive evaluation and analysis. our evaluation of zeroshot and fine-tuned approaches, which we conduct on extensive baselines within various subsets and subcategories, offers valuable insights for future research.