CSIRO at Context24: Contextualising Scientific Figures and Tables in Scientific Literature (2024)

Abstract: Finding evidence for claims from content presented in experimental results of scientific articles is difficult. The evidence is often presented in the form of tables and figures, and correctly matching it to scientific claims presents automation challenges. The Context24 shared task is launched to support the development of systems able to verify claims by extracting supporting evidence from articles. We explore different facets of this shared task modelled as a search problem and as an information extraction task. We experiment with a range of methods in each of these categories for the two sub-tasks of evidence identification and grounding context identification in the Context24 shared task.

## introduction finding evidence relating to scientific claims in research articles is a time-consuming task. these claims can be supported by the article's text, figures or tables. the context24 international shared task foot_0 , organised as part of the scholarly document processing workshop at the acl 2024 conference, set a challenge for developing methods for two tasks: our team participated in both tasks with methods drawing from information retrieval and natural language processing (nlp) research, including statistical ranking models, dense retrieval and the use of large language models (llms), including multimodal llms. fact checking and fake news detection news is the most common way to disseminate information rapidly. however, fake news-characterised by low-quality, intentionally misleading information-poses significant threats by distorting public perception and causing negative impacts on both individuals and society at large @xcite . addressing the problem has become imperative, prompting researchers to focus on the automatic verification of facts and detection of fake news. numerous studies and public datasets have been developed for the task, covering various domains and applications. averitec @xcite and multifc @xcite are examples of such datasets. to increase the popularity of the task and the development of effective systems, shared tasks such as fact extraction and verification (fever) @xcite have been organised. fever focuses on verifying factoid claims using evidence from wikipedia and also introduced new methods, including machine learning algorithms for fact verification @xcite @xcite , the application of llms for detecting fake news @xcite , and factchecking using multi-modality sources such as images @xcite and videos @xcite . ## dataset the claims and papers used in the context24 shared task come from four separate datasets, each of which comes from a different set of research domains. ## experimental setup we use the default b and k1 parameter values in bm25 implementation, utilising rank_bm25 library for the element-guided approach in task 1 and the retrieval-based approach in task 2, and elasticsearch for the chunk-guided approach in task 1. in task 2, we use two different tools to identify sentences in the full-text articles: nltk @xcite and spacy @xcite , with the best results reported. we use the pos tagger of the nltk library for the retrieval with the rule-based filtering approach proposed for task 2. we retrieved the original pdf files from the publishers using dois in the original articles. this led to a cleaner version of the document collection. it also provided us with information missing from the articles, such as the page numbers. in task 1, we use the following instructions (prompts) to create e5 query embeddings for element-, and chunk-guided approaches, respectively: in task 2, we use the following prompt for zeroshot experiments of llms: ## conclusions we participated in the context24 shared task for multimodal evidence and grounding context identification for scientific claims. there were two subtasks for evidence identification and grounding evidence identification. for the former, we treated it as an information retrieval task where both statistical and dense retrieval methods were investigated. we obtained promising results with the mistral e5 dense retrieval model and with our ranking fusion experiments. for the second subtask, our best method used a filtering mechanism where frequent expressions were identified from the training data. this method yielded higher results than those we obtained using generative models.