Harnessing CLIP for Evidence Identification in Scientific Literature: A Multimodal Approach to Context24 Shared Task (2024)

Abstract: Knowing whether scientific claims are supported by evidence is fundamental to scholarly communication and evidence-based decision-making. We present our approach to Task 1 of the Context24 Shared Task—Contextualizing Scientific Figures and Tables (SDP@ACL2024), which focuses on identifying multimodal evidence from scientific publications that support claims. We finetune CLIP, a state-of-the-art model for image-text similarity tasks, to identify and rank figures and tables in papers that substantiate specific claims. Our methods focus on text and image preprocessing techniques and augmenting the organizer-provided training data with labeled examples from the SciMMIR and MedICaT datasets. Our best-performing model achieved NDCG@5 and NDCG@10 values of 0.26 and 0.30, respectively, on the Context24 test split. Our findings underscore the effectiveness of data augmentation and preprocessing in improving the model’s ability in evidence matching.

## introduction scientific claims, essential for communicating findings in literature reviews, research problem formulation, and interpreting conflicting data, require careful contextualization and verification with supporting empirical evidence and methodological details. figures and tables in papers presenting key results, measures, and sample characteristics are critical for understanding and validating claims. however, retrieving such contextual information from scientific papers is challenging and timeconsuming, particularly when researchers and practitioners encounter claims without direct access to the source materials. recent advancements in multimodal ai models, which excel at image-text similarity tasks, present an opportunity to automate the retrieval and contextualization of evidence. however, their application for evidence identification in scientific literature remains underexplored. existing research primarily focuses on text summarization @xcite @xcite , entity recognition @xcite , and information retrieval @xcite , with limited attention to integrating visual evidence from figures and tables. this paper presents our approach to task 1 of the context24 shared task, 1 focusing on evidence identification for scientific claims. the task involves predicting a ranked list of figures or tables from a relevant research paper that provides supporting evidence for a given scientific claim. we leverage the contrastive language-image pretraining (clip) model's image-text similarity capabilities, augment training data with diverse datasets and augmentation techniques, and employ textimage pre-processing methods to retrieve and rank relevant figures and tables from scientific papers. the contributions of our work are as follows: • we introduce a comprehensive dataset that combines augmented training data from the shared task with additional data from scimmir @xcite and medicat @xcite , significantly expanding the scope and diversity of the training samples ( §3). neural models and machine learning further transformed capabilities in this domain. techniques such as bi-lstm @xcite and bilstm-crf @xcite were adapted to parse and analyze argumentative structures in scientific texts for sentencelevel classification and multi-task learning. these works enhanced the ability to distinguish between different types of evidence based on linguistic cues and discourse patterns. further, @xcite introduced learning-to-rank techniques combined with metadata features, such as journal impact factor and citation metrics, to optimize the retrieval process. @xcite introduced the task of scientific claim verification, to select abstracts from research literature that either support or refute a given scientific claim; the authors introduced the scifact dataset and baseline models, demonstrating that domain adaptation techniques significantly improve performance compared to models trained on wikipedia or political news. the authors later extended the scifact dataset into the open domain by integrating retrieval in the data curation process @xcite . image-text matching recent advancements in vision-language pretraining (vlp) have significantly reshaped the landscape of evidence identification. vlp aims to learn universal and transferable features from images and text, applicable across image-text retrieval tasks @xcite @xcite @xcite @xcite @xcite . multimodal models like clip @xcite , blip-2 @xcite , and stable diffusion @xcite that have undergone significant multimodal pretraining, have the potential to improve the alignment of text-based claims with image-based evidence. our work builds on these advancements by fine-tuning a model based on the clip architecture @xcite on large-scale scientific datasets with an image-text matching objective. extending beyond the typical focus on biomedical or computer science domains in prior works @xcite @xcite , our approach covers a broader range of scientific disciplines. we leverage these image-text embeddings to retrieve and match scientific claims with their supporting evidence. ## datasets this section describes the context24 dataset ( §3.1) and outlines our strategies for data enrichment ( §3.2) and external data augmentation ( §3.3). ## experiments we detail the model selection and training process, and experiments to assess the impact of training data variation and preprocessing techniques. ## results following the context24 shared task, we report normalized discounted cumulative gain (ndcg) at 5 and 10 to assess the ranking of the retrieved figures and tables. additionally, we report precision and recall for the top 5 and 10 retrieved images (i.e., p@5, r@5, p@10, and r@10). ## future work our current approach leverages the entire medi-cat subcaption subset and scimmir datasets for training the clip model. however, we can potentially improve performance by selecting subsets of these datasets that more closely align with the scientific domains or types of claims in the con-text24 dataset. this approach might enhance the currently, we use references and captions from figures and tables as additional image-text pairs to train the clip model. another potential enhancement involves augmenting the claims with this contextual information before computing similarity scores with the images. this method could provide a more holistic understanding of the claim and its context by incorporating essential details directly alongside the claim text in claim verification. ## conclusion in this paper, we presented our approach to task 1 of the context24 shared task, which focuses on identifying multimodal evidence for scientific claims. we enhanced our training data by integrating sub-captions and inline references as well as additional data from scimmir and medicat. we finetuned the clip model on this enriched dataset, applying image-text preprocessing and augmentation techniques to effectively align scientific claims with their corresponding figures and tables. we secured third place on the leaderboard, with ndcg@5 and ndcg@10 scores of 0.26 and 0.30, respectively on the test dataset. our findings underscore the significance of expanding training data and employing preprocessing techniques on the model's performance.