Overview of the Context24 Shared Task on Contextualizing Scientific Claims (2024)

Abstract: To appropriately interpret and use scientific claims for sensemaking and decision-making, it is critical to contextualize them, not just with textual evidence that the claim was in fact asserted, but also with key supporting empirical evidence, such as a figure that describes a key result, and methodological details, such as the methods of data collection. Retrieving this contextual information when encountering claims in isolation, away from their source papers, is difficult and time-consuming for humans. Scholarly document processing models could help to contextualize scientific claims, but there is a lack of datasets designed for this task. Thus, we contribute a dataset of 585 scientific claims with gold annotations for supporting figures and tables, and gold text snippets of methodological details, that ground the key results behind each claim and run the Context24 shared task to encourage model development for this task. This report describes details of our dataset construction process, summarizes results from the shared task conducted at the 4th Workshop on Scholarly Document Processing (SDP), and discusses future research directions in this space. To support further research, we also publicly release the dataset on HuggingFace.

## introduction people read and use scientific claims both within the scientific process (e.g., in literature reviews, problem formulation, making sense of conflicting data) and outside of science (e.g., evidenceinformed deliberation). to appropriately interpret, appraise, and ultimately use claims for sensemaking and decision-making, it is critical to contextualize claims with key supporting empirical evidence (e.g., figures presenting key results) and methodological details (e.g., measures, sample). ## related work the task of contextualizing scientific claims can be formulated similarly to tasks like claim verification and fact checking. additionally, modeling techniques used for this task have parallels in work on figure captioning and data extraction from scientific literature. ## dataset collection and preprocessing our dataset curation process consists of three stages: (i) claim construction, (ii) collecting annotations for evidence and grounding context identification, and (iii) sourcing supplementary data (e.g., extracted figures/tables, paper full-texts, etc.) to ease model development for participants. ## official results the training data for this shared task was released on github, alongside evaluation scripts, on april 11, 2024. the test phase was released as an open competition on eval.ai foot_5 on may 31, 2024. participants were allowed to upload a maximum of five submissions per day during the seven-day testing phase. we received official submissions from a total of 6 teams for track 1, and 2 teams for track 2. the official results from the leaderboard, as well as the performance of some baseline systems, are shown in tables 4 and 5 for tracks 1 and 2, respectively. ## discussion and future directions overall, our baseline experiments and participant submissions for both shared task tracks demonstrate that the problem of contextualizing scientific claims is still challenging but starting to become tractable. for instance, baseline performance was quite low, but two teams were able to substantially outperform these baselines while still leaving some room for improvement. our experiences constructing the dataset, and reflections on participants' technical reports, also suggest fruitful future directions. better image-text similarity models. contrastively trained image-text similarity models like clip could be beneficial for our task since they can adapt to other scientific domains with little training data (necessary for real-world use given high costs of collecting annotated data). however, a common theme across participants' reports was that using image-text similarity models did not perform well compared to relying entirely on text or prompting multimodal llms. exploring why models like clip do not work well on our task and how to improve them could lead to stronger models for our task, as well as advancements in image-text representation learning. one potential direction may be to explore augmentation of figures with textual mentions @xcite . pdf pre-processing. an ideal system for contextualizing scientific claims would be able to operate end-to-end from research paper pdfs. however, state-of-the-art pdf preprocessing toolkits (e.g., papermage @xcite ) were not able to extract figures/tables with sufficient accuracy, particularly for some domains like biology papers; thus, to make this shared task tractable, we provided manually extracted figures/tables/captions to participants. this suggests that further work is needed to improve pdf preprocessing tasks such as structured content extraction @xcite , or converting pdfs to more accessible html @xcite , which could enable development of better end-to-end contextualization methods, especially across scientific domains. ## conclusion in this paper, we described a new dataset and results from a shared task for contextualizing scientific claims with empirical evidence and methodological details. our dataset consists of 585 scientific claims with gold annotations for figures and tables, and gold text snippets of methodological details, that ground the key results behind each claim. experiments with text embedding and (multimodal) llm-based baselines demonstrate the challenging nature of the task. performance gains over these baselines from official submissions to the shared task for this dataset at sdp 2024 show the tractability of the task, and reveal promising directions for future work, such as improving image-text similarity models, pdf-processing, and automated evaluation metrics for context identification.