F in G r A ct: A Framework for FIN e- GR rained Evaluation of ACT ionability in Explainable Automatic Fact-Checking (2025)

Abstract: The field of explainable Automatic Fact-Checking (AFC) aims to enhance the transparency and trustworthiness of automated fact verification systems by providing clear and comprehensible explanations. However, the effectiveness of these explanations depends ontheir actionability—the extent to which an AFC explanation pinpoints the error, supplies the correct fact, and backs it with sources. Despiteactionability being critical for high-quality explanations, no prior research has proposed a method to evaluate it. This paper introducesFinGrAct, a fine-grained evaluation framework that can access the web and is designed to assess actionability in AFC explanations through well-defined criteria. We also introduce a novel dataset to evaluate actionability in AFC explanations. FinGrAct surpasses state-of-the-art (SOTA) evaluators, achieving the highest Pearson and Kendall correlation with human judgments while demonstrating the lowest egocentricbias, making it a more robust evaluation approach for actionability evaluation in AFC.

## introduction explanation of claim veracity (see figure 1 for examples of explanations) is essential in automated fact-checking as it enhances transparency, fosters trust, and educates users by clarifying why a claim is deemed true or not. in the domain of explainable automated factchecking (afc), the quality of the explanation of a claim's predicted veracity is assessed based on the presence of specific desired properties, referred to as "desiderata," as outlined by @xcite . while some of these desiderata have been explored in the field of summarization like coherence and fluency, others, such as actionability, a critical property in fact-checking, remain unexplored. to date, no automatic evaluator for actionability has been developed. however, some general purpose sota evaluators emerged, but only for summarization tasks. in this work, these evaluators are adapted to afc. actionability remains underexplored due to its complexity. according to @xcite , actionability in afc refers to providing factual corrections for identified errors in a nonfactual claim, supported by sources and references. this suggests that actionability is highly correlated with other key properties, such as relevance to the claim and completeness, where the explanation must provide a comprehensive context for why the claim is considered true or not. providing an automatic evaluator for this desideratum is critical, because it yields a reproducible, fine-grained metric that (i) enables large-scale benchmarking across systems and (ii) offers a reliable reward signal for training or selecting explanation models, thereby accelerating research cycles, enhancing misinformation mitigation while safeguarding user trust. evaluating actionability, however, presents significant challenges as it requires well-defined criteria for judgment, as well as access to external sources in the internet to evaluate the supporting references. figure 1 shows examples of explanations with different degrees of actionability. to tackle the challenge of automatically evaluating the actionability of explanations in afc systems, we introduce fin-gract. fingract is a fine-grained auto-evaluator that systematically measures the degree of actionability in these explanations. the evaluation framework is illustrated in figure 1 . our contributions: we present a dataset, constructed from existing sources, containing explanations for claim labels with varying degrees of actionability. additionally, it includes human-rated actionability scores, which can serve as benchmarks for evaluating the performance of different evaluators of actionability. we present fingract a novel fine-grained evaluator of explanations for actionability within the context of explainable afc, based on llm prompting, that correlates better with human ratings than other label: false explanation 1 (full actionability): the claim has two errors in earth's description. the errors are in the words 'flat' and 'red'. the correct version of the claim is : "earth is round and blue". check nasa at: https://explorer1.jpl.nasa.gov/galleries/earth-from -space/ explanation 2 (partial actionability): earth is a round planet. check nasa at: https://explorer1.jpl.nasa.gov/galleries/earth-fro m-space/ explanation 3 (no actionability): earth is a planet in the solar system. ## related work with the emergence of llms such as chatgpt, recent studies have employed these models as evaluators @xcite to assess llm performance across various benchmarks, including g-eval @xcite , and finesure @xcite . nonetheless, it has been observed that llms functioning as auto-raters tend to show a preference for their own generated responses @xcite . this is called ego-centric bias. training open-source general-purpose llm autoraters has been investigated recently. tigerscore, a llama-2 model trained on gpt-4 generated error analysis data across multiple tasks, such as summarization, translation, and instruction-following, is presented by @xcite . however, it doesn't follow a likert-scale scoring or can it be scaled to it. for instance, it can produce a score of -12 making it hard to be compared to other evaluators. prometheus @xcite , instructscore @xcite , and prometheus-2 @xcite are comparable methods. @xcite developed reward models used for aligning llms to human preferences via reinforcement learning and called their model flame. however, they haven't published their model or their dataset collection. sota evaluators like g-eval and prometheus are widely used in summarization to assess properties such as coherence and faithfulness. unlike property-specific tools like finesure, they are adaptable for evaluating new attributes. finesure gives a specific fine-grained definition for each property it measures in a summary based on the key-facts it found in the transcript. making its adaptation to new tasks and new properties very hard. all the mentioned work is directed towards evaluating certain desired properties in diverse tasks like summarization and question answering. however, little effort has been directed towards developing an autorater for the desiderata of the explanations of afc. for instance, @xcite tried to evaluate properties like (self)-contradiction, hallucination, convincingness and overall quality. @xcite evaluated free-form properties like coherence, deductive properties like nonredundancy, and argumentative properties like selfsupport. @xcite introduced a new evaluation protocol -citation masking and recovery -to assess attribution quality in generated explanations. however, they didn't address some critical properties, mentioned by @xcite and @xcite , and were deemed critical for fact-checking explanation like actionability for instance. ## experimentation and results we applied our fine-grained evaluation methodology for actionability using openai's gpt-4-1106preview model. this choice was made because gpt-4-1106-preview serves as the primary model for g-eval and other baseline evaluators, ensuring a fair and consistent comparison between fingract and existing evaluators such as g-eval. the focus of our study is not on the model itself but rather the evaluation methodology, ensuring that the assessment framework remains the central point of analysis. we design four distinct experiments: comparison with sota evaluation methods: the first experiment aims to compare our evaluation methodology with existing sota models. the primary metric used for comparison is the correlation with human annotations. ## conclusion the paper introduces fingract, a fine-grained evaluation method for assessing actionability in afc, a crucial but underexplored property. the study shows that fingract outperforms sota evaluators, achieving the highest pearson and kendall correlation with human judgments and exhibiting the lowest ego-centric bias, making it more reliable. additionally, incorporating retrieved and summarized content from referenced sources further improved actionability evaluation across all models. these findings establish fingract as a superior framework for assessing actionability in afc.