MAPLE : Micro Analysis of Pairwise Language Evolution for Few-Shot Claim Verification (2024)

Abstract: Claim verification is an essential step in the automated fact-checking pipeline which assesses the veracity of a claim against a piece of evidence. In this work, we explore the potential of few-shot claim verification, where only very limited data is available for supervision. We propose MAPLE (Micro Analysis of Pairwise Language Evolution), a pioneering approach that explores the alignment between a claim and its evidence with a small seq2seq model and a novel semantic measure. Its innovative utilization of micro language evolution path leverages unlabelled pairwise data to facilitate claim verification while imposing low demand on data annotations and computing resources. MAPLE demonstrates significant performance improvements over SOTA baselines SEED, PET and LLaMA 2 across three fact-checking datasets: FEVER, Climate FEVER, and SciFact. Data and code are available.

## introduction the proliferation of misinformation and fake news has become a significant concern in today's information landscape. fact-checking has emerged as a crucial task to combat the spread of false information (thorne and vlachos, 2018; kotonya and toni, 2020a; nakov et al., 2021; zeng et al., 2021; guo et al., 2022). a body of natural language processing (nlp) research has investigated the task of claim verification: determining the veracity of a claim based on retrieved evidence. it is often addressed in a natural language inference (nli) fashion, namely making predictions on the claim with reference to evidence out of three candidate labels: 'supports', 'refutes', and 'not_enough_info'. while the majority of previous work tackles the problem with fully supervised methods (li et al., 2021; zeng and zubiaga, 2021; zhang et al., 2021; wadden et al., 2022; rana et al., 2022b,a), deploying these methods face practicality issues. emerging domains of misinformation often involve novel claims, limiting the availability of relevant labeled data. fact-checkers often need to evaluate claims with time constraints, limiting the time allowed for conducting extensive fine-tuning of pretrained language models (plms). hence, performing claim verification in few-shot scenarios is of particular importance in the realworld combat of misinformation. the current state-of-the-art (sota) methods for few-shot claim verification are semantic embedding element-wise difference (seed) (zeng and zubiaga, 2022) and pattern exploiting training (pet) (schick and schütze, 2021a,b). however, their few-shot performance relies on the use of nlitrained plms, limiting their applicability to only cases where nli data and nli-trained plms are available, excluding scenarios such as low-resource languages. moreover, these methods excel when the data is similar to nli data but struggle when dealing with dissimilar data. in contrast, we propose to embrace the potential of leveraging unlabeled data, which is more readily available in a fact-checking pipeline, to enhance few-shot claim verification. an alternative strand of research in the realm of general few-shot classification advocates for generative large language models (llms) endowed with billions of parameters, exemplified by models like gpt-4 (openai, 2023) and llama 2 (touvron et al., 2023). these models demonstrate impressive few-shot performance, though introducing a reliance on advanced computational resources and prolonged inference times. in contrast, our work challenges this paradigm by demonstrating that smaller models, such as t5-small (raffel et al., 2020), possess the inherent capability to excel in few-shot learning scenarios. leveraging unlabeled data and advanced semantic measures, our approach underscores the efficacy of compact models in achieving effective and robust few-shot performance without the need for extensive computational resources. we present maple (micro analysis of pairwise language evolution), a novel approach designed for few-shot claim verification. maple innovatively builds upon the concept of language transition 1 , scrutinizing the semantic shift that occurs as a sequence-to-sequence model learns to generate a target sequence from a given input sequence. in this paper, such language transition from the input sequence to the output sequence over the training epochs is referred to as pairwise language evolution. by intricately capturing and harnessing this pairwise language evolution, maple aims to facilitate accurate predictions even in scenarios with minimal labeled data. our key novel contributions include: ## methodology traditionally, generative models are often used in classification tasks by generating corresponding labels given input sentences (pradeep et al., 2021). however, such an approach does not fully exploit the potential of generative models on tasks such as claim verification. in this section, we present the maple method and its application to few-shot claim verification. the intuition of maple is that sentence pairs of various relationships bring diverse learning challenges to the seq2seq generation task. as the data difficulty is reflected in the seq2seq training process, such learning difficulty associated with each sample could be further transformed into various signs to indicate the relationship within a sentence pair. we explore such potential to be leveraged for effective claim verification, where the goal is to determine the veracity of a claim based on its relationship with the provided evidence. maple consists of three steps, as illustrated in figure 1 . ## experiments in this section, experiments comparing maple with previous sota methods are presented. ## results in this section, we present the results of our experiments with a focus on few-shot settings. figure 2 illustrates the f1 performance within the 5-shot setting. @xcite across the four dataset configurations, maple shows noticeable performance advantages within the 5-shot setting, validating its effectiveness in few-shot scenarios and robustness across datasets. it achieves this primarily by starting from a high performance point and steadily improving within 5 shots. although seed underperforms maple, it showcases strong learning capabilities, and its relatively lower performance is primarily due to a low starting point. surprisingly, pet and llama 2 perform poorly within the 5shot range, generally starting low and exhibiting limited learning capabilities. on the fever dataset, maple demonstrates significant improvements over the baselines. specifically, maple achieves a very high f1 score over 0.6 at 1 shot, outperforming seed, pet, and llama 2, which commence at approximately 0.25, 0.37, and 0.38, respectively. within 5 shots, maple exhibits a steady performance improvement, surpassing an f1 score of 0.7. while seed and pet also experience notable performance boosts, with seed approaching just below 0.6 and pet reaching below 0.5, llama 2 encounters a slight performance drop, settling around 0.36. on the cfever dataset, the performance of all methods exhibits a considerable decrease compared to fever, highlighting the challenging nature of the dataset. while maple maintains its leading position overall, the performance margin is narrower. it initiates above 0.3 and achieves scores surpassing 0.4. seed begins even lower, below 0.3, but manages to surpass 0.4, albeit slightly trailing behind maple. pet encounters greater challenges overall, commencing below seed and only slightly exceeding 0.3. llama 2 excels initially with a score of 0.38 but experiences a drop to 0.37. on the scifact_oracle dataset configuration, despite the overall performance being better than @xcite please see detailed classwise performance in appendix b.2 cfever but worse than fever across all methods, maple maintains superiority within 5 shots. it initiates around 0.4 and concludes around 0.45. seed begins around 0.3 and lags behind maple, while pet starts higher than seed but lower than maple, failing to surpass them within 5 shots. llama 2 performs comparably to pet, starting at 0.37 and finishing at 0.40. on the scifact_retrieved dataset configuration, maple demonstrates a slightly better performance compared to scifact_oracle, while all baseline methods exhibit a substantial decline in performance compared to scifact_oracle. consequently, maple achieves a larger performance margin. it commences above 0.4 and concludes around 0.5. seed starts at a very low point, below 0.3, and approaches 0.4 at 5 shots. pet initiates around 0.35 but struggles to learn effectively within 5 shots, resulting in an even lower score. llama 2 starts at 0.32 and 0.29 and experiences a notable drop to 0.18 and 0.17 immediately afterwards. @xcite in general, llama 2 displays reasonable oneshot performance but shows limited learning capabilities within 5 shots. despite pet's use of gradient descent to update the parameters of a large language model, this strategy does not yield satisfactory results within the 5-shot range. on the other hand, maple and seed showcase relatively rapid convergence due to their limited number of trainable parameters. maple stands out with a significantly higher level of performance compared to all baselines overall, demonstrating its capacity to leverage limited data for notable results and effectiveness as a few-shot claim verification model. it's crucial to highlight that while most experiments are conducted in oracle settings, real-world claim verification often introduces the challenge of imperfect evidences. therefore, achieving optimal performance in the scifact_retrieved dataset, where evidence is noisy and lengthy, is particularly significant. this accomplishment highlights maple's robustness to noisy and challenging data in realistic fact-checking scenarios. ## ablation studies training algorithms with the growing interest in reinforcement learning (rl) and parameterefficient training, this ablation study investigates the effects of utilizing different training algorithms. specifically, we comprare lora, supervised fine-tuning (sft) and natural language policy optimization (nlpo), an innovative rl method that offers enhanced stability and performance compared to previous policy gradient methods (ramamurthy et al., 2023). as presented in figure 4 , the overall differences in performance among the algorithms are relatively marginal. sft demonstrates best results on the fever and cfever datasets, while nlpo outperforms on the scifact_oracle and scifact_retrieved datasets. notably, despite the largely reduced computational burden by utilizing lora, foot_4 the observed performance drops are modest. therefore, maple conducts in-domain seq2seq training with lora. metrics maple uses our proposed 'semsim' metric to measure and analyze the pairwise language evolution. this ablation section presents the comparison with a number of established nlg metrics, including 'bleu', 'rouge', 'meteor', 'sacrebleu', 'bleurt', and 'bartscore'. figure 4 illustrates the performance variations of maple when employing different metrics. across all datasets, the 'semsim' metric demonstrates superior performance compared to other metrics, showcasing a significant improvement gap. this highlights the advantages of 'semsim', establish-ing it as the optimal choice for maple. by focusing on measuring semantic similarity as a primary component, we can effectively analyze the micro pairwise evolution of language in a seq2seq learning process, which is captured by generated mutations across training epochs. in contrast, metrics based solely on lexical overlap, or utilizing an llm that is not trained on substantial sentence pair data, may be less indicative in capturing the nuances of language evolution. the emphasis on fine-grained semantic similarity provides highly informative insights, particularly in assessing the learning difficulty of instances for seq2seq generation. as 'semsim' surpasses many established nlg metrics in this task, it shows its potential for broader applications as a general nlg evaluation metric. ## analysis and discussion despite recent research on generating rationales and explanations (atanasova et al., 2020; kotonya and toni, 2020b; schuster et al., 2021), existing approaches heavily depend on directly finetuning plms, hindering the understanding of their decision-making process. maple stands out by providing tangible and traceable solutions, guided by the principle that sentence pairs with different relations present distinct challenges for seq2seq generation. figure 5 further supports this principle and elucidates the effectiveness of maple. overall, the 'semsim' scores for 'not_enough_info' are significantly lower than those for 'supports' and 'refutes', enabling easy differentiation between 'not_enough_info' and other classes foot_5 . furthermore, generating a piece of evidence from a claim proves to be more challenging than generating a claim from a piece of evidence. generating claims primarily needs the removal of redundant or unnecessary content, while generating evidence requires the model to expand the existing content. furthermore, figure 5 shows that generating a claim is easier for 'supports' than for 'refutes', while generating evidence is easier for 'refutes' than for 'supports'. this pattern allows for a distinction between the two categories. with its enhanced interpretability and traceability, maple aims to bolster the reliability and trustworthiness of the claim verification process. moreover, by comparing the difficulty among datasets based on the above information, we can gain insights into the varying challenges posed by different domains. for example, if a dataset such as fever consistently exhibits high 'sem-sim' scores and low standard deviation during in-domain seq2seq training, it suggests that the claims and evidences within that dataset are easier to match and converge upon. on the other hand, datasets such as cfever with lower 'semsim' scores, higher standard deviation, and longer convergence time indicate greater difficulty in aligning claims and evidences. this comparative analysis allows us to understand the relative complexities of fact-checking in different settings and further enhances the interpretability of maple's performance across datasets. moreover, maple's low demand on annotations and computing facilities enhances its efficiency and accessibility. both step (1) in-domain seq2seq training and step (2) semsim transformation only require unlabeled claim-evidence pairs and limited annotations are only required for step (3) logistic classifier training with few-shot labelled data. while performing steps (1) and ( 2 ) over the entire unlabeled pool may seem burdensome, such practice only takes from minutes to few hours. @xcite due to maple's efficiency and accessibility by design, training and deploying can be easily accomplished on google colab with a free account or even on a personal laptop. in real-world scenarios where the claim verification team has accumulated a substantial collection of claim-evidence pairs, which can be claims with annotated oracle evidences or claims with retrieved noisy evidences, they can initiate steps (1) and ( 2 ) and this process can be completed while the team actively acquires a small number of labeled samples. subsequently, step (3) training a logistic classifier with the newly acquired data only takes seconds and maple is ready for deployment. by designing such an efficient workflow, the application of maple in realworld scenarios can bring in a decent claim verifi-cation model with minimal cost in annotation and computational resources. overall, maple holds practical value for fact-checking in real-world contexts, particularly as a tool to assist fact-checkers in combating emerging domains of misinformation. ## future directions with the development of maple, several promising directions for future research emerge: self-supervised extensions currently, maple combines language transition signals with a traditional logistic classifier for classification. a further research avenue could include its development into a fully self-supervised system by integrating clustering methods. ## conclusions in this paper, we introduce maple, a novel approach for few-shot claim verification. by leveraging language transition signals during seq2seq training convergence, maple achieves sota performance in precisely predicting claim veracity labels with reference to associated evidences in few-shot learning scenarios. through extensive experiments and analysis on multiple datasets, we validate its effectiveness, robustness, interpretability, efficiency and accesibility.