OSX at Context24: How Well Can GPT Tackle Contexualizing Scientific Figures and Tables (2024)

Abstract: Identifying the alignment between different parts of a scientific paper is fundamental to scholarly document processing.In the Context24 shared task, participants are given a scientific claim and asked to identify (1) key figures or tables that support the claim and (2) methodological details.While employing a supervised approach to train models on task-specific data is a prevailing strategy for both subtasks, such an approach is not feasible for low-resource domains.Therefore, this paper introduces data-free systems supported by Large Language Models.We propose systems based on GPT-4o and GPT-4-turbo for each task.The experimental results reveal the zero-shot capabilities of GPT-4* in both tasks.

## introduction in scientific writing, the alignment between different sections of a paper and the consistency between textual claims and visual elements are paramount for effective communication @xcite . a well-aligned paper presents a coherent narrative from introduction to conclusion, with each section building logically upon the last. this structural integrity is complemented by the harmonious integration of figures and tables with the text, where visual data reinforce and clarify written assertions @xcite . such alignment serves multiple crucial functions: it enhances reader comprehension, strengthens the paper's argumentative force, and facilitates critical evaluation by peers. when research questions, methodologies, results, and interpretations are presented consistently across both prose and visual formats, it becomes easier for readers to grasp the study's significance and situate its findings within the broader scientific context. this holistic approach to alignment not only elevates the quality of individual papers but also contributes to the overall efficiency of knowledge dissemination in the scientific community. building upon the importance of alignment in scientific writing, the context24 shared task at the 4th workshop on scholarly document processing (sdp 2024) addresses a critical challenge in scientific communication: the efficient interpretation and contextualization of scientific claims. this task aligns with the broader goal of enhancing the coherence between claims and supporting evidence in scientific papers. the two tracks of the shared task -evidence identification and grounding context identification -aim to automate the process of linking claims with their supporting visual elements and methodological details, respectively. by facilitating the rapid identification of supporting evidence and grounding context, the context24 task has the potential to significantly enhance the efficiency of scientific communication, enabling researchers to evaluate and build upon existing work more quickly. evidence identification this track requires participants to identify key figures or tables from a given research paper that provide supporting evidence for a specific scientific claim. participants must not only locate the relevant visual data but also ensure that it directly supports the stated claim. ## related work our zero-shot approach to multimodal scientific claim verification is motivated by two key factors in the current research landscape. first, there is a notable scarcity of large-scale annotated datasets specifically designed for multimodal scientific claim verification. as highlighted by @xcite , creating such datasets requires significant domain expertise and is highly resource-intensive. this lack of training data presents a substantial challenge for supervised learning approaches in this domain. secondly, recent advancements in large language models (llms) have demonstrated remarkable zero-shot and few-shot capabilities across various tasks. @xcite showed that gpt-3 can perform complex tasks without task-specific fine-tuning, while wei et al. (2022) further demonstrated the effectiveness of instruction-tuning in enhancing zero-shot performance. these findings suggest that llms could potentially address complex tasks like scientific claim verification without extensive labeled data. in the realm of multimodal models, several architectures have been developed to bridge the gap between vision and language. clip (contrastive language-image pre-training) @xcite demonstrated strong zero-shot performance on various vision tasks through natural language supervision. more recent models like llava @xcite have extended this approach to incorporate more general vision-language capabilities. in the scientific domain, models like scienceqa @xcite have been developed to handle scientific figures and text, showing promising results in tasks like answering questions about scientific figures and diagrams. our choice of clip for this task was motivated by its strong zero-shot performance and its ability to align visual and textual representations in a shared embedding space. this alignment is particularly crucial for our task, where we need to match textual claims with visual evidence from scientific figures and tables. recent work on effective instruction writing for llms informed the development of our prompting strategy. @xcite demonstrated that carefully crafted prompts can induce step-by-step reasoning in llms, improving their performance on complex tasks. similarly, @xcite showed that prompts encouraging llms to explain their reasoning often lead to more accurate outputs. these insights guided our approach to designing prompts that encourage the model to articulate its reasoning process when evaluating the relevance of visual evidence to textual claims. ## evidence identification in the evidence identification task, we identified key figures and tables using a pipeline system supported by gpt-4o. given a claim x and k candidate images (figures and tables) @xmath0, z 2 , • • • , z k }, we first computed the supportiveness score s i between the claim and the i-th image. afterward, we sorted the candidate images based on their scores to determine the final ranking. ## grounding context identification in the grounding context identification, we identified the methodological details in a zero-shot manner using gpt-4-turbo. most part of the prompt p is cited from the official task definition foot_2 . we only appended a prompt to format the output. given a claim x and the full text of the corresponding article d, we first asked gpt-4-turbo to generate the output @xmath1, d; p ), where p is the prompt shown in ## conclusion in this paper, we introduced gpt-4-based systems designed to identify key images and methodological details for a given claim. our systems achieved considerable high performance in both subtasks within the context24 shared task.