Rethinking Loss Functions for Fact Verification (2024)

Abstract: We explore loss functions for fact verification in the FEVER shared task. While the cross-entropy loss is a standard objective for training verdict predictors, it fails to capture the heterogeneity among the FEVER verdict classes. In this paper, we develop two task-specific objectives tailored to FEVER. Experimental results confirm that the proposed objective functions outperform the standard cross-entropy. Performance is further improved when these objectives are combined with simple class weighting, which effectively overcomes the imbalance in the training data. The source code is available (https://github.com/yuta-mukobara/RLF-KGAT).

## introduction the fact extraction and verification (fever) shared task @xcite challenges systems to verify a given claim by referencing wikipedia articles. a system for fever typically begins by extracting sentences from wikipedia that potentially support or refute the claim. subsequently, the verdict predictor in the system classifies the claim, in conjunction with the retrieved sentences, into one of three verdict classes: as this verification step is a multiclass classification task, verdict predictors are usually trained using the cross-entropy loss function. however, cross-entropy treats all misclassification types uniformly, which is problematic given the heterogeneity among the verdict classes in fever; labels sup and ref both assume evidence is present in the retrieved sentences, whereas a claim is deemed nei only when such evidence is missing. consequently, it is debatable, for example, whether misclassifying a sup claim as ref or as nei should be considered equally severe errors, especially when the retrieved sentences indeed contain supporting evidence, such as when a verdict predictor is trained with oracle sentences. in this paper, we explore objective functions designed to capture the heterogeneity among verdict classes. notation for a 𝐾-class classification problem, let @xmath0, . . . , 𝑦 𝐾 ) ∈ {0, 1} 𝐾 denote a one-hot class representation vector where each index represents a class. depending on the context, we also use y to denote the corresponding class itself. let @xmath1, . . . , 𝑝 𝐾 ) ∈ [0, 1] 𝐾 denote a predicted class distribution (i.e., 𝐾 𝑖=1 𝑝 𝑖 = 1). for fever verdict prediction, 𝐾 = 3, and let the indexes 1, 2, 3 correspond to sup, ref, nei, respectively. ## experiments due to limited space, only the main experimental results are presented below. additional results and analysis can be found in appendix b. ## related work the fever shared tasks @xcite @xcite have been the subject of extensive research. most proposed approaches utilize transformer-based models to embed claims and evidence @xcite @xcite , whereas some researchers @xcite use graph-based methods to aggregate information from multiple pieces of evidence. none of these studies focus on the objective function to optimize, and most employ the standard cross-entropy objective. recently, dehaven and scott (2023) have used class weighting to mitigate class imbalance in the fever dataset, although the detailed weighting scheme is not reported. in machine learning, @xcite analyzes various loss functions used for multiclass classification, including a general form of one-versus-rest (or oneversus-all) loss functions, which also have terms accounting for the complement set of the ground-truth class. @xcite study complementarylabel learning scenarios @xcite @xcite ) extending zhang's losses. ## conclusion we introduced loss functions that take into account the heterogeneity of verdict classes in the fever task. in empirical evaluation, they consistently outperformed the standard cross-entropy loss. in future work, we will evaluate the proposed loss functions in other fact verification tasks. we also plan to apply them to sota models for fever. as these models use the cross-entropy loss, our auxiliary loss terms are readily applicable.