Active PET s: Active Data Annotation Prioritisation for Few-Shot Claim Verification with Pattern Exploiting Training (2023)

Abstract: To mitigate the impact of the scarcity of labelled data on fact-checking systems, we focus on few-shot claim verification. Despite recent work on few-shot classification by proposing advanced language models, there is a dearth of research in data annotation prioritisation that improves the selection of the few shots to be labelled for optimal model performance. We propose Active PETs, a novel weighted approach that utilises an ensemble of Pattern Exploiting Training (PET) models based on various language models, to actively select unlabelled data as candidates for annotation. Using Active PETs for few-shot data selection shows consistent improvement over the baseline methods, on two technical fact-checking datasets and using six different pretrained language models. We show further improvement with Active PETs-o, which further integrates an oversampling strategy. Our approach enables effective selection of instances to be labelled where unlabelled data is abundant but resources for labelling are limited, leading to consistently improved few-shot claim verification performance. Our code is available.

## introduction as a means to mitigate online misinformation, research in automated fact-checking has experienced a recent surge of interest. research efforts have resulted in survey papers covering different perspectives @xcite @xcite @xcite and novel datasets with enriched features @xcite @xcite @xcite @xcite . recent work has addressed various challenges, e.g. generating and utilising synthetic data @xcite @xcite , joint verification over text and tables @xcite , investigating domain adaptation @xcite , achieving better evidence representations and selections @xcite @xcite , and performing subtasks jointly @xcite @xcite . as a core component of a fact-checking system, a claim validation pipeline consists of document retrieval, rationale selection and claim verification @xcite . our main focus here is claim verification, the task of assessing claim veracity with retrieved evidence. it is typically treated as a natural language inference (nli) task: given a claim and an evidence, the aim is to predict the correct veracity label out of "support", "neutral" and "contradict". substantial improvements have been achieved in the performance of claim validation models when a considerable amount of training data is available @xcite @xcite @xcite . however, where new domains needing fact-checking emerge, collecting and annotating new relevant datasets can carry an impractical delay. availability of unlabelled data can often be abundant, but given the cost and effort of labelling this data, one needs to be selective in labelling a small subset. in these circumstances, rather than randomly sampling this subset, we propose to optimise the selection of candidate instances to be labelled through active learning, such that it leads to overall improved few-shot performance. to the best of our knowledge, our work represents the first such effort in proposing an approach leveraging an active learning strategy for the claim verification problem, as well as the first in furthering pattern exploiting training (pet) with an active learning strategy. to achieve this, we propose active pets, a novel methodology that enables the ability to leverage an active learning strategy through a committee of pets. figure 1 illustrates the application of the active learning strategy on data annotation priotisation. by exploring effective prioritisation of unlabelled data for annotation and making better use of a small amount of labelled data, we make the following novel contributions: ## methodology in this section, we first describe pet, then introduce our model active pets, and finally describe the oversampling mechanism we use. ## results we next discuss results for our experiments. ## ablation study with scifact we designed a slightly different pipeline where we conduct both evidence retrieval and claim verification -a setting that wasn't provided with cfever. to assess the impact of the addition of the evidence retrieval component on scifact, we further perform ablation experiments on scifact with oracle evidence. with oracle evidence, the number of "neutral" claim-evidence pairs are significantly reduced, resulting in a more balanced overall label distribution. after reserving 100 instances from each class for the test set, the unlabelled pool has 765 instances in total, where "support" takes 46.54%, "neutral" takes 38.43% and "contradict" takes 15.03%. as shown in figure 5 , overall few-shot performance is much better and active learning demonstrates lesser performance gains. sampling with baseline active learning strategies in general leads to similar results as random sampling. surprisingly, coupling active pets with oversampling when the labelled pool is reasonably balanced, still maintains performance advantages across models. under this setting, bertbase peaks at 0.645, roberta-base peak at 0.655; deberta-base peaks at 0.766; bert-large peaks at 0.68; roberta-large peaks at 0.657; debertalarge peaks at 0.86. as demonstrated above, active learning is much more helpful for scifact in a real-world setting than in an oracle setting. we could expect that if this finding generalises to cfever, active learning in a real-world setting involving evidence retrieval could possibly lead to larger performance gains. ## analysis to better understand the impact of data prioritisation, we delve into the labelled data. in the interest of focus, we compare active pets with the sota cold-start method alps by analysing the best-performing plm deberta-large where 300 instances are selected. we first look at the distribution of labels for the selected data. figure 6 shows remarkable difference on label distribution for different active learning strategies. alps samples over 80% data from "neutral", less than 10% from "support" and very few from "contradict" for scifact; over 60% data from "neutral", over 20% from "support" and less than 20% from "contradict" for cfever. they correlate well with original label distribution of each unlabelled pool, as presented in ## conclusions we present the first study on data annotation prioritisation for claim verification in automated factchecking. with our novel method active pets, we demonstrate the potential of utilising a committee of pets to collaboratively select unlabelled data for annotation, furthering in turn the extensibility of pet to active learning for the first time. experiments on the scifact and cfever datasets demonstrate the effectiveness of our proposed method, particularly in dealing with imbalanced data. our proposed model consistently outperforms the random, badge, cal and alps baselines by a margin. further integration with an oversampling strategy that does not impact labelling effort leads to consistent performance improvements in all tested settings. data that is more balanced shows to have higher lexical richness and semantic similarity, leading to better training results. while we have shown its effectiveness for claim verification here, in the future we aim to investigate active pets in other downstream tasks.