Prompt to be Consistent is Better than Self-Consistent? Few-Shot and Zero-Shot Fact Verification with Pre-trained Language Models (2023)

Abstract: Few-shot or zero-shot fact verification only relies on a few or no labeled training examples. In this paper, we propose a novel method called ProToCo, to Prompt pre-trained language models (PLMs) To be Consistent, for improving the factuality assessment capability of PLMs in the few-shot and zero-shot settings. Given a claim-evidence pair, ProToCo generates multiple variants of the claim with different relations and frames a simple consistency mechanism as constraints for making compatible predictions across these variants. We update PLMs by using parameter-efficient fine-tuning (PEFT), leading to more accurate predictions in few-shot and zero-shot fact verification tasks. Our experiments on three public verification datasets show that ProToCo significantly outperforms state-of-the-art few-shot fact verification baselines. With a small number of unlabeled instances, ProToCo also outperforms the strong zero-shot learner T0 on zero-shot verification. Compared to large PLMs using in-context learning (ICL) method, ProToCo outperforms OPT-30B and the Self-Consistency-enabled OPT-6.7B model in both few- and zero-shot settings.

## introduction the problem of misinformation has sparked significant attention on the task of fact verification within the natural language processing (nlp) community. such task, typically represented by fact extraction and verification (fever) benchmark @xcite , requires models to verify if pieces of evidence support, refute, or contain not enough information (nei) to validate a given claim. fully supervised fact verification has been widely studied and achieved good performance on the data of different domains @xcite @xcite . however, collecting a large set of training data is laborintensive, time-consuming and costly especially evidence: coronavirus disease 2019 is a zoonotic infectious disease caused by severe acute respiratory syndrome coronavirus 2. ## related work existing methods tried to address few-shot fact verification by utilizing the implicit knowledge of plms encoded in their parameters without gradients update. @xcite hypothesized that the perplexity of concatenated claim-evidence text sequence evaluated by a language model could benefit claim verification, and used a few training instances to find the threshold of perplexity scores for determining the label of test claim. @xcite utilizes plms to create a set of representative vectors for each class based on the semantic difference between claim and evidence of a few training instances, which are used to label test claims based on euclidean distance during inference. however, these models do not update model parameters solely relying on the preencoded knowledge of plms, which cannot improve the language model itself and may not generalize well in new domains. and they also cannot perform zero-shot task as a few labeled instances are required as the anchors for labeling new instances. our method aims to update plms efficiently to utilize new knowledge in a few examples and enforce model's consistency for improving both few-shot and zero-shot verification. recently, several studies worked to generalize plms to the target domain by fine-tuning the model with the full training dataset of fact verification from a different domain @xcite @xcite . meanwhile, some works targeted to instruct plms to generate task-specific training data used to fully train a classifier for fact verification @xcite . such works need a carefully crafted generation policy based on real corpus of the task, and the performance heavily depends on the quality of generated data. these approaches are considered distantly supervised, and significantly differ from ours as they do not aim to build any few-shot or zero-shot model. unlike these studies, we assume that the language model is minimally aware of fact verification task with only a few task-specific examples, which may be even unlabeled. in general, plms have shown strong few-shot learning ability in various nlp tasks @xcite . in-context learning (icl) uses natural language prompts or instructions to elicit desired output from plms without gradient updates @xcite . however, icl is hard to deal with many prompted instances @xcite , sensitive to the prompt design @xcite and performs worse than fine-tuning @xcite ). an alternative approach is parameter-efficient fine-tuning (peft) by updating only a small number of parameters to bridge the gap with standard fine-tuning @xcite @xcite @xcite @xcite . our method utilizes t-few @xcite , a state-of-the-art peft-enabled model, as our backbone to perform the factuality-grounded consistency training. previous works evaluate the self-consistency of plms by modifying the context of input sentences @xcite @xcite and empirically show that plms are insufficient to transfer self-consistency to downstream tasks. some works in question answering (qa) prompt large plms (e.g., @xcite ) to improve qa accuracy by strengthening the consistency of predicted answers. @xcite prompts plm to generate multiple explanations and candidate answers and choose the answer that consistently occurs as the prediction. the maieutic prompting @xcite designed for true-or-false commonsense qa, and concord @xcite designed specifically based on selfconsistency benchmarks, both of which elicit plms to generate distributions for possible candidate an-swers, followed by a maxsat solver @xcite to infer the most probable answer by eliminating contradictory candidates. both methods are based on different consistency definitions from ours and may not be suitable for the fact verification task. ## problem definition let @xmath0, y i )} be a fact verification dataset, containing training set c train and test set c test , where each instance consists of the input x i and ground-truth label y @xmath1, nei, refute}. let x @xmath2, e i ), and the task aims to predict if the given pieces of evidence e i supports, refutes or has not enough information to validate the claim c i . in the few-shot setting, we randomly sample k instances per class from c train for training as the class distribution is unknown. as a result, the total number of instances is 3k and the few-shot training set is denoted as c f s @xmath3, y i )} 3k . the zero-shot setting is similar but only uses x i for each instance and the unlabeled training set is given as c zs @xmath4. note that the absence of ground-truth label makes the setting zero-shot @xcite . similar to previous works @xcite , we do not assume the availability of development set as it is more realistic in a limit data scenario. our goal is to generalise a plm m θ to the unseen test set c test , fine-tuned only using c f s train or c zs train , where θ denotes language model parameters. ## conclusion and future work we propose a model called protoco to improve few-and zero-shot fact verification based on consistency training of plms. experiments on three public datasets show that protoco achieves promising fact verification performance by outperforming the existing few-and zero-shot baselines, the in-context learning on large plms, and the self-consistency chain-of-thought method. our method also outperforms fully-supervised model on fever dataset. in the future, we will explore few-and zero-shot solutions for other stages of fact-checking, e.g., evidence retrieval and justification generation, and combine them with protoco. we also plan to conduct experiments to evaluate the performance and level of consistency of larger language models (e.g., @xcite , instructgpt @xcite and llama @xcite ) on the fact verification task, when the computing resources are available.