Towards Few-shot Fact-Checking via Perplexity (2021)

Abstract: Few-shot learning has drawn researchers’ attention to overcome the problem of data scarcity. Recently, large pre-trained language models have shown great performance in few-shot learning for various downstream tasks, such as question answering and machine translation. Nevertheless, little exploration has been made to achieve few-shot learning for the fact-checking task. However, fact-checking is an important problem, especially when the amount of information online is growing exponentially every day. In this paper, we propose a new way of utilizing the powerful transfer learning ability of a language model via a perplexity score. The most notable strength of our methodology lies in its capability in few-shot learning. With only two training samples, our methodology can already outperform the Major Class baseline by more than an absolute 10% on the F1-Macro metric across multiple datasets. Through experiments, we empirically verify the plausibility of the rather surprising usage of the perplexity score in the context of fact-checking and highlight the strength of our few-shot methodology by comparing it to strong fine-tuning-based baseline models. Moreover, we construct and publicly release two new fact-checking datasets related to COVID-19.

## introduction few-shot learning is being actively explored to overcome the heavy dependence on large-scale labeled data that serves as a crucial bottleneck to machine learning models. recently, researchers have explored few-shot learning that leverages the powerful transfer learning ability of pre-trained large language models (lms) in various nlp tasks. @xcite demonstrated that an lm serves as a good zero-shot learner on the question-answering task due to its encoded commonsense knowledge. going further, brown et al. illustrated the impressive potential of lms as strong zero-shot and fewshot learners across translation, commonsense reasoning and natural language inference (nli). however, little or no exploration has been made on fewshot learning in the fact-checking domain, which is a timely and important task in which data-scarcity is particularly problematic. previous works have proposed different ways of leveraging lms to conduct zero-or few-shot learning. one common approach is to query the lm for the missing token (i.e., "answer") for the zeroshot question-answering task @xcite by transforming questions into a form of statement. another approach is to adopt an in-context learning approach where the input context of the lm is carefully crafted to control the output. for example, a natural language task instruction (e.g., "translate english to french:") or training sample (e.g., "sea @xmath0. in this work, we explore a new way of leveraging lms for few-shot learning in the fact-checking task. this is done by leveraging a perplexity score from unsupported claims perplexity supported claims perplexity 5g network can spread diseases. 826.70 beyonce is one of the most famous singers in the world. 23.03 all dogs speak english fluently. ## preliminary exploration of hypothesis in this section, we conduct a preliminary investigation to validate the potential of our hypothesis that the perplexity score from an evidence-conditioned lm can provide a signal for claims unsupported by evidence. for our exploration, we first collect a small set of supported and unsupported claims that can be verified based on the training corpus of the target lm (namely, wikipedia which is used in the training of many pre-trained lms). then, we compare the perplexity scores between them. to recap, perplexity is a commonly used metric for measuring the performance of lms. it is defined as the inverse of the probability of the test set normalized by the number of words: another way of interpreting perplexity is as a measure of the likelihood of a given test sentence with reference to the training corpus. from ## analysis and discussion in this section, we conduct multiple analysis to further evaluate and understand aspects of our perplexity-based approach. ## future research directions in this work, we conduct the fever experiments in a binary set-up to keep all the experimental settings consistent across all three datasets. however, the original fever task has three classes -support, refute, and not enough info (nei). since the distinction between nei and refute cases is also an important problem, it would be important future work to extend our binary-class setting to the three-class setting. moreover, we believe our method can easily be augmented into other existing approaches, for instance, leveraging the perplexity score in the final step of the fever fact-checkers as additional input. it would be a useful future direction to explore and discover the most effective way of incorporating the perplexity-based approach into other existing fact-checking systems. ## conclusion in this paper, we propose a novel way of leveraging the perplexity score from lms for the few-shot fact-checking task. through experimental analysis from an ablation study to the discussion of potential applications, we further explore and evaluate the capability of the perplexity score to act as an indicator of unsupported claims. we hope our proposed approach encourages future research to continue developing lm-based methodologies as well as the few-shot approach for fact-checking. by doing so, our community can move towards a data-efficient approach that is not constrained by the requirement of a large labeled dataset.