SCITAB : A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables (2023)

Abstract: Current scientific fact-checking benchmarks exhibit several shortcomings, such as biases arising from crowd-sourced claims and an over-reliance on text-based evidence. We present SCITAB, a challenging evaluation dataset consisting of 1.2K expert-verified scientific claims that 1) originate from authentic scientific publications and 2) require compositional reasoning for verification. The claims are paired with evidence-containing scientific tables annotated with labels. Through extensive evaluations, we demonstrate that SCITAB poses a significant challenge to state-of-the-art models, including table-based pretraining models and large language models. All models except GPT-4 achieved performance barely above random guessing. Popular prompting techniques, such as Chain-of-Thought, do not achieve much performance gains on SCITAB. Our analysis uncovers several unique challenges posed by SCITAB, including table grounding, claim ambiguity, and compositional reasoning. Our codes and data are publicly available at https://github.com/XinyuanLu00/SciTab.

## introduction scientific fact-checking is a crucial process that involves validating the accuracy of scientific claims by cross-referencing them with established scientific literature, research, or data @xcite . this process is crucial for preserving the integrity of scientific information, preventing the spread of misinformation, and fostering public trust in research findings. however, the sheer volume of scientific data and claims can be overwhelming for manual fact-checking, making automated scientific fact-checking an imperative research area of nlp. scientific fact-checking has advanced significantly with benchmarks including @xcite , sci-fact open @xcite , and covid-fact @xcite . ## the scitab dataset we adopt a human-model collaboration strategy to construct scitab, as shown in figure 2 . we describe the steps involved in data preparation (section 2.1), automatic claim generation (section 2.2), and manual claim verification (section 2.3). ## data analysis ## experiment we formally define the task of scientific table-based fact-checking as follows. a scientific table t consists of a table caption p and the table content ({t i,j |@xmath0, @xmath1, where t i,j is the content in the (i, j)th cell. given a claim c describing a fact to be verified against the table t , a table fact-checking model f predicts a label y to verify whether c is supported, refuted, or can not be verified by the ## conclusion and future work we present scitab, a novel dataset for scientific fact-checking that addresses the limitations of existing benchmarks. by incorporating realworld scientific claims and their corresponding evidence in the form of tables, scitab offers a more comprehensive and fine-grained representation of scientific reasoning. the challenging nature of scitab is evident from the performance of the state-of-the-art, highlighting the need for further research. for example, we believe that addressing the challenges posed by ambiguous claims represents a crucial direction for research in scientific fact-checking @xcite . one potential approach is to enhance the disambiguation of ambiguous claims by leveraging contextual information or external knowledge sources. additionally, studying the compositionality in table-based reasoning is an interesting direction. consider the work of self-ask @xcite , which proposed the "compositionality gap" metric to measure the capability of llms in compositional reasoning. such evaluations can be enriched by annotating scitab with ground-truth reasoning depths and structured reasoning graphs. beyond this, another direction worth exploring is equipping the llms with external tools to further improve the model. for example, the use of gpt-4 plugins, program-guided fact-checking @xcite or adopting approaches from other toolaugmented llms like toolformer @xcite and chameleon @xcite .