Improving Explainable Fact-Checking via Sentence-Level Factual Reasoning (2024)

Abstract: Most existing fact-checking systems are unable to explain their decisions by providing relevant rationales (justifications) for their predictions. It highlights a lack of transparency that poses significant risks, such as the prevalence of unexpected biases, which may increase political polarization due to limitations in impartiality. To address this critical gap, we introduce SEntence-Level FActual Reasoning (SELFAR), aimed at improving explainable fact-checking. SELFAR relies on fact extraction and verification by predicting the news source reliability and factuality (veracity) of news articles or claims at the sentence level, generating post-hoc explanations using SHAP/LIME and zero-shot prompts. Our experiments show that unreliable news stories predominantly consist of subjective statements, in contrast to reliable ones. Consequently, predicting unreliable news articles at the sentence level by analyzing impartiality and subjectivity is a promising approach for fact extraction and improving explainable fact-checking. Furthermore, LIME outperforms SHAP in explaining predictions on reliability. Additionally, while zero-shot prompts provide highly readable explanations and achieve an accuracy of 0.71 in predicting factuality, their tendency to hallucinate remains a challenge. Lastly, this paper also presents the first study on explainable fact-checking in the Portuguese language.

## introduction while journalism is tied to ethical standards, including truth and fairness, it often strays from impartial facts @xcite . as a result, low credibility news may be produced and spread on modern media ecosystem. nowadays, fact-checking organizations have manually provided lists of unreliable articles and media sources, however it is a very time-consuming task, needs to be updated faster and relies on specific expertise @xcite . towards addressing this issue, fact-checking systems have classified claims of unknown veracity (factuality), identifying evidences and predicting whether they support or refute the claims @xcite . nevertheless, as low credibility news or claims may comprise multiple sentences containing facts, media bias, and fake information, fact-checking at scale should be able to accurately predict both news source reliability and factuality at a fine-grained level. furthermore, the veracity of claims can be verified using metadata @xcite , wikipedia @xcite , social networks @xcite , scientific assertions @xcite , manually checked-claims from social media provided by fact-checking organizations @xcite , the language used in claims @xcite , llms @xcite , generating justifications for verdicts on claims @xcite . for example, the fever @xcite , scifact @xcite , liar @xcite and check-covid @xcite are widely used datasets for this setting. in recent years, there has been significant progress in the area of fact-checking e.g., new comprehensive datasets @xcite @xcite , high performance of deep learning models @xcite , different domains aside from political @xcite @xcite @xcite . however, while justifying the verification of a claim's veracity is the most important part of the manual process, most existing fact-checking systems are unable to explain their decisions, which could assist human fact-checkers and help mitigate the lack of transparency (baly n. sentence-level news article label s1 president jair bolsonaro touch a sore point of europeans when he pointed out that the increased use of fossil fuels is a serious environmental setback, in his opening speech at the un general assembly, tuesday (20). ## conclusions this paper introduces a new method to improve explainable fact-checking. the selfar predicts reliability and the factuality of news articles or claims at the sentence level, generating post-hoc explanations using lime/shap and zero-shot prompts. our experiments showed that unreliable news stories are comprised mostly of subjective words, in contrast to reliable ones. thus, predicting unreliable news stories by analyzing text impartiality and subjectivity is promising for fact extraction and improving explainable fact-checking. in addition, lime outperforms shap in explaining reliability predictions. lastly, while zero-shot prompts provide highly readable explanations and achieve an accuracy of 0.71 in predicting factuality, their tendency to hallucinate presents a challenge. we also present baselines for explainable fact-checking in portuguese.