Core: Robust Factual Precision with Informative Sub-Claim Identification (2025)

Abstract: Hallucinations pose a challenge to the application of large language models (LLMs) thereby motivating the development of metrics to evaluate factual precision. We observe that popular metrics using the Decompose-Then-Verify framework, such as FActScore, can be manipulated by adding obvious or repetitive subclaims to artificially inflate scores. This observation motivates our new customizable plug-and-play subclaim selection component called Core, which filters down individual subclaims according to their uniqueness and informativeness. We show that many popular factual precision metrics augmented by Core are substantially more robust on a wide range of knowledge domains. We release an evaluation framework supporting easy and modular use of Core and various decomposition strategies, which we recommend adoption by the community. We also release an expansion of the FActScore biography dataset to facilitate further studies of decomposition-based factual precision evaluation.

## introduction automatically generating long-form text is prevalent since the rise in large language models (llms) @xcite . these models are trained on vast amounts of textual data that provide abundant information, enabling them to serve as a significant source of knowledge @xcite @xcite . a running concern is ensuring llm-generated content is faithful to real-world facts or user inputs, devoid of hallucination @xcite . to this end, various automatic factuality evaluation pipelines have been proposed @xcite @xcite @xcite . mainstream methods typically involve two key steps: first, a decomposition step, where the generated text is broken down into natural language prompt: tell me a bio of adil rami. ## the proposed core method core is a unique subclaim selection and filtering process that works with any subclaim identifier φ from any of the popular decompose-then-verify metrics discussed in subsection 2.1. 3 given subclaims identified by φ, the goal of core is to filter a subset of subclaims that are unique and informative. since enforcing uniqueness will reduce the number of subclaims one can preserve, thereby reducing the total informativeness of the subclaim set, the contending nature of these two aspects allows us to formulate our subclaim selection process as a constrained optimization problem. this section describes the formulation of core in detail. an overview of our method can be found in figure 2 . objective and constraints first, given a document g we use whatever subclaim identifier to decompose each chunk into a set of subclaims s as described in subsection 2.1. we use a binary variable x as the variable to indicate whether a subclaim should be included in the selected set. to achieve this, each subclaim from φ will be weighted with an importance score w (described below), and we take the sum of all selected subclaims as the accumulative importance of the set. this leads to the following integer programming problem to select the most important set of subclaims maximize the objective of the integer programming is to find the set with maximum accumulative importance under the following constraints: 1 ⃝ at least @xmath0, 1] of the subclaims are correctly identified; 2 ⃝ there does not exist s i , s j from the selected set ŝ such that verifying s i immediately verifies s j or vice versa. constraint 1 ⃝ is necessary as decomposed subclaims are not always faithful @xcite . we characterize both constraints 1 ⃝ and 2 ⃝ using textual entailment relationships. a subclaim s i is correctly identified if the subclaim is entailed by the chunk it comes from. foot_3 two subclaims s i , s j are considered check-worthy at the same time only if none of them are entailed by the other. the official algorithm of the core selection process is listed out in appendix a. ## experiments and results we aim to answer two important research questions: 1 ⃝ can models artificially boost their reported fp by generating uninformative and repetitive text of low quality? 2 ⃝ how effective is core in mitigating this issue? as previous works have shown that eliciting human ground truth for factual precision is challenging, and often can only be annotated under underspecified instructions @xcite or with the help of biasing llm decompositions @xcite , we instead adopt a behavioral evaluation scheme. subsection 5.3 demonstrates that adding uninformative or repetitive content can superficially increase the factual precision of corrupted bios over clean ones, but not for factual precision calculated with core adjustments. subsection 5.4 illustrates that core shows the desired behavior of a robust fp metrics and effectively guards against adversarial inputs to decompose-then-verify metrics. we compare fp metrics with and without core on the dataset created in subsection 4.2. for pair-wiseentailment and docentailment evaluation, we use deberta-v3-base-mnli-feveranli foot_5 from the hugging face model hub to model entail. to estimate w info (c) for each subclaim c, we fine-tune a strong nli model robertalarge-snli_mnli_fever_anli_r1_r2_r3-nli foot_6 @xcite on unli @xcite , as described in section 3. we also binarize a "capmodel" deberta-v3-base-mnli-fever-anli as "entailment" and "non-entailment" to make sure that subclaims entailed by bleached claims will not get included in the verification step. we give the exact formulation of this capped winfo (c) in appendix a. additionally the weighting function can be further adjusted to cater to relevancy concerns, leading to the following combined scoring function: we further observe that the particualr choice of nli models for entailment evaluation does not have strong empact on core computation, as shown in ## conclusion we demonstrate that popular factual precision evaluation metrics following the decompose-then-verify pipeline often assign superficially high scores to obvious or repetitive generations. we introduce core, a plug-and-play module that addresses this issue efficiently and effectively. we further show that when augmented with core, various decompose-then-verify metrics demonstrate a consistent trend of being more robust and become less prone to repetitive and non-informative adversarial inputs. consequently, we argue that adjustments like core should be adopted for more accurate factual precision evaluation, especially in scenarios where models can optimize against automatic metrics. future research can explore deeper into the interplay between the evaluation of factual precision and the actual factual accuracy of models, and potentially also develop more effective subclaim selection methods within the core framework and explore more comprehensive approaches to factuality evaluation.