M ulti V er S : Improving scientific claim verification with weak supervision and full-document context (2022)

Abstract: The scientific claim verification task requires an NLP system to label scientific documents which Support or Refute an input claim, and to select evidentiary sentences (or rationales) justifying each predicted label. In this work, we present MultiVerS, which predicts a fact-checking label and identifies rationales in a multitask fashion based on a shared encoding of the claim and full document context. This approach accomplishes two key modeling goals. First, it ensures that all relevant contextual information is incorporated into each labeling decision. Second, it enables the model to learn from instances annotated with a document-level fact-checking label, but lacking sentence-level rationales. This allows MultiVerS to perform weakly-supervised domain adaptation by training on scientific documents labeled using high-precision heuristics. Our approach outperforms two competitive baselines on three scientific claim verification datasets, with particularly strong performance in zero / few-shot domain adaptation experiments. Our code and data are available at https://github.com/dwadden/multivers .

## introduction the proliferation of scientific mis-and disinformation on the web has motivated the release of a number of new datasets @xcite @xcite and the development of modeling approaches @xcite @xcite for the task of scientific claim verification. the goal of the task is to verify a given scientific claim by labeling scientific research abstracts which support or refute the claim, and to select evidentiary sentences (or rationales) reporting the findings which justify each label. a common approach to this task is to first extract rationales from the larger document context, and then make label predictions conditioned on the figure 1 : a claim from the healthver dataset, refuted by a research abstract. the sentence in red is a rationale reporting a finding that refutes the claim. however, this finding cannot be interpreted properly without the context in blue, which specifies that the finding applies to ibuprofen as a treatment for covid. multivers incorporates the full context of the evidence-containing abstract when predicting fact-checking labels. motivated by these challenges, we introduce multivers (multitask verification for science): given a claim and evidence-containing scientific abstract, multivers creates a shared encoding of the entire claim / abstract context, using the longformer encoder @xcite to accommo-date long sequences. then, it predicts an abstractlevel fact-checking label and sentence-level rationales in a multitask fashion, enforcing consistency between the outputs of the two tasks during decoding. this modeling approach ensures that label predictions are made based on all available context, and enables training on instances derived via weak supervision for which abstract-level labels are available, but sentence-level rationales are not. in experiments on three scientific claim verification datasets, we find that multivers outperforms two state-of-the-art baselines, one of which has more than 10x the parameters of our system. in addition, we show that training multivers on weakly-labeled in-domain data substantially improves performance in the zero / few-shot domain adaptation settings. the ability to achieve reasonable performance given limited labeled data is especially valuable in specialized domains, due to the high cost of collecting expert annotations. in summary, our contributions are as follows: 1. we introduce multivers, a multitask system for full-context scientific claim verification. multivers improves fully-supervised factverification performance by an average of 11% on three datasets over two state-of-the-art baselines, with improvements of 14% and 26% in the few-shot and zero-shot settings. 2. we present weak supervision heuristics to assign fact-checking labels to two large scientific datasets, and show that training on these annotations more than doubles zero-shot domain adaptation performance. 3. through ablations and analysis, we demonstrate that our multitask modeling approach achieves our goals of incorporating full-document context into label predictions, and facilitating zero / few-shot domain adaptation. ## the multivers model we propose the multivers model for fullcontext claim verification. in §3.1, we describe our modeling approach. rather than predicting rationales r(c, a) followed by the overall fact-checking label y(c, f r ( r(c, a))), we predict y(c, a) directly based on an encoding of the entire claim and abstract, and enforce consistency of r(c, a) with y(c, a) during decoding. a similar idea has been shown to be effective on sentiment analysis and propaganda detection with token-level rationales @xcite . in §3.2, we explain how our approach facilitates few-shot domain adaptation using weakly-labeled scientific documents. ## experimental setup we describe our model training procedure, the systems against we compare multivers, and our ablation experiments. ## experimental results we compare multivers performance relative to our baseline systems, and present ablation results. ## related work background on scientific claim verification is covered in §2; we discuss other relevant work here. @xcite have previously observed that document-level context is often required to properly interpret scientific findings. deyoung et al. (2020) use an "extract-then-label" pipeline for the original evidenceinference task. multitask label prediction and rationale selection was proposed by @xcite and applied to sentiment analysis and propaganda detection. as in this work, the authors condition on the predicted label when predicting rationales. another alternative to supervised rationale selection is to treat rationales as latent variables @xcite . long-document encodings for fact verification have been explored by @xcite , who use big bird @xcite for full-document evidence extraction from fever. domain adaptation for scientific text has been studied in a number of works, including @xcite ; @xcite ; @xcite ; @xcite . in those works, the primary focus is on language model pretraining. here, we focus on training on the target task using out-of-domain and weaklylabeled data. ## conclusion this work points to a number of promising future directions for scientific claim verification. these include applying the approach presented here to develop scientific claim verification models for new scientific sub-domains or other specialized fields given a handful of labeled examples, and extending the task definition to verify claims against longer contexts (e.g. full scientific papers) or larger corpora. our task formulation also offers an opportunity to study the effects of rationale decontextualization @xcite , especially in cases where models may be making predictions based on insufficient evidence. in presenting the multivers system, we addressed two challenges associated with scientific claim verification: incorporating relevant information beyond rationale boundaries by modeling fulldocument context, and facilitating zero / few-shot domain adaptation through weak supervision enabled by a multitask modeling approach. our experiments show that multivers outperforms existing systems across several scientific claim verification datasets. we hope that the task, data, and modeling resources presented in this paper will encourage further work and progress towards the broader goals of identifying and addressing scientific mis-and disinformation.