Scientific Claim Verification with V er T 5erini (2021)

Abstract: This work describes the adaptation of a pretrained sequence-to-sequence model to the task of scientific claim verification in the biomedical domain. We propose a system called VerT5erini that exploits T5 for abstract retrieval, sentence selection, and label prediction, which are three critical sub-tasks of claim verification. We evaluate our pipeline on SciFACT, a newly curated dataset that requires models to not just predict the veracity of claims but also provide relevant sentences from a corpus of scientific literature that support the prediction. Empirically, our system outperforms a strong baseline in each of the three sub-tasks. We further show VerT5erini’s ability to generalize to two new datasets of COVID-19 claims using evidence from the CORD-19 corpus.

## introduction the popularity of social media and other means of disseminating content, combined with automated algorithms that create "echo chamber" effects, has increased the proliferation of misinformation online. this has led to increased attention in the community on building better fact verification systems. until recently, most fact verification datasets were constrained to domains such as wikipedia, discussion blogs, and social media @xcite . in the current environment, amidst the covid-19 pandemic and the unease that comes with insufficient insight about the virus, there has been a sharp increase in curiosity among the general public toward scientific knowledge. while such curiosity is always appreciated, this has inadvertently resulted in a large spike of scientific facts being misrepresented, often to push a personal or political agenda, inducing ineffective and frequently even harmful policies and behaviours. to mitigate this issue, @xcite introduced the task of scientific claim verification, where systems need to evaluate the veracity of a claim against a scientific corpus. to facilitate this, they introduced the scifact dataset that consists of scientific claims accompanied with abstracts that either support or refute the claim. the dataset also provides a set of rationale sentences for each claim that is necessary and sufficient to conclude its veracity. in addition, the authors provide verisci, a baseline for this task that takes inspiration from previous state-of-the-art systems @xcite for the fever claim verification dataset @xcite . this pipeline retrieves relevant abstracts by tf-idf similarity, uses a bertbased model @xcite to select rationale sentences, and finally labels each abstract as either supports, noinfo, or refutes with respect to the claim. despite the success of bert for tasks like passage-level @xcite , documentlevel @xcite @xcite and sentence-level @xcite retrieval, there is evidence that ranking with sequence-tosequence models can achieve even better effectiveness, particularly in zero-shot scenarios or with limited training data @xcite . this was further demonstrated in the trec-covid challenge @xcite where one of the best performing systems used sequence-to-sequence models for retrieval @xcite . similar trends are noted in covidqa @xcite , a question answering dataset for covid-19, where zeroshot sequence-to-sequence models outperformed other baselines. hence, we propose vert5erini, where all three steps-abstract retrieval, sentence selection, and label prediction exploit t5 @xcite , a powerful sequence-to-sequence language model. vert5erini outperforms the verisci baseline on the scifact tasks by a large margin and advances the state of the art for the task of scientific claim verification. we also demonstrate the effectiveness of our system in verifying two different sets of covid-19 claims with no additional training or hyperparameter tuning. ## task in the scifact task @xcite , systems are provided with a scientific claim q and a corpus of abstracts c and tasked to return: given the ground truth label y(q, a), the set of gold abstracts e(q), and the set of gold rationales r(q, a) (each gold rationale is a set of sentences), the predictions are evaluated in two ways: specifically, scifact uses a corpus of 5,183 abstracts. abstracts that support or refute each claim are annotated with rationale sentences (see ## methods our proposed system, vert5erini (see figure 1 ), has three major components: ## conclusions in this paper, we introduced vert5erini, a novel system for scientific claim verification that exploits a generation-based approach to abstract ranking, sentence selection, and claim verification. such systems are of significance in this age of misinformation, amplified by the covid-19 pandemic. experiments show that our system outperforms the state of the art in the end-to-end task on the sci-fact dataset. we note improvements in each of the three sub-tasks, demonstrating the importance of this sequence-to-sequence approach as well as zero-shot and few-shot transfer capabilities. finally, we find that vert5erini generalizes to two new covid-19 related datasets with no tuning of parameters while maintaining high effectiveness. yet, there is still a large gap between our system and an oracle. ideally, a system that performs scientific claim verification should possess additional attributes such as: future work that incorporates such attributes might be critical towards building higher-quality scientific fact verification systems. we report progress, but there is much more work to be done.