Overview and Insights from the SCIVER shared task on Scientific Claim Verification (2021)

Abstract: We present an overview of the SCIVER shared task, presented at the 2nd Scholarly Document Processing (SDP) workshop at NAACL 2021. In this shared task, systems were provided a scientific claim and a corpus of research abstracts, and asked to identify which articles Support or Refute the claim as well as provide evidentiary sentences justifying those labels. 11 teams made a total of 14 submissions to the shared task leaderboard, leading to an improvement of more than +23 F1 on the primary task evaluation metric. In addition to surveying the participating systems, we provide several insights into modeling approaches to support continued progress and future research on the important and challenging task of scientific claim verification.

## introduction due to both rapid growth in scientific publications and the proliferation of mis-and dis-information online, there is a growing need for automated systems that can assist humans in assessing the veracity of scientific claims with evidence found in the research literature. for the sciver shared task, systems are given a claim about a scientific entity and a corpus of abstracts from peer-reviewed research articles, and are asked to identify the articles in the corpus that support or refute the claim; each prediction is required to be accompanied with evidentiary sentences, or rationales, from the abstract that justify the labeling decision. this task poses various challenges to systems. for example, entailment relationships between claims and evidences found in scientific writing are often complex, and understanding them to arrive at a correct support or refute labeling decision may require reasoning about numerical quantities, increases and decreases in measurements, or causal chains. furthermore, since annotations require scientific expertise, training data for this task are scarce. as a result, systems must employ techniques to overcome the lack of training data, such as domain adaptation or transfer learning. here, we report the results on the sciver shared task. a total of 11 teams made 14 submissions to the shared task leaderboard, leading to a collective improvement of +23 f1 on the primary evaluation metric, compared to previous baselines. the remainder of this report is organized as follows: in §2, we describe the shared task setup, including the choice of dataset, task definition, and evaluation metrics. in §3, we provide an overview of the systems submitted for the task, and highlight unique features of individual systems. in §4, we present the results from the shared task evaluation. finally, in §5, we identify several insights into modeling approaches demonstrated by participating systems. ## shared task description we briefly describe the dataset, task, and evaluation. additional details on the data collection process and evaluation metrics can be found in @xcite . ## overview of systems ## results ## insights a number of participating teams reported performance on individual pipeline components in publications associated with their systems (see ## conclusion the sciver shared task on scientific claim verification received 14 submissions from 11 teams. the submitted systems achieved substantial improvements on all three stages of the scientific claim verification pipeline -abstract retrieval, evidentiary sentence identification, and label prediction -improving on the previous state-of-the-art by 60% and 44% on the two primary evaluation metrics. we've surveyed the various approaches taken by participants and discussed several findings that have emerged from this shared task. looking forward, the strong performance of systems on sciver suggests it may be time to explore other more ambitious challenges on the path toward building real-world scientific claim verification systems. for instance, future work could study retrieval from a much larger scientific corpus, approaches to consolidate evidence from multiple documents while taking into account the strength and credibility of different evidence sources, or techniques for building efficient systems that can approximate performance of the top submissions at a fraction of the cost.