U rdu F act C heck: An Agentic Fact-Checking Framework for U rdu with Evidence Boosting and Benchmarking (2025)

Abstract: The rapid adoption of Large Language Models (LLMs) has raised important concerns about the factual reliability of their outputs, particularly in low-resource languages such as Urdu. Existing automated fact-checking systems are predominantly developed for English, leaving a significant gap for the more than 200 million Urdu speakers worldwide. In this work, we present UrduFactBench and UrduFactQA, two novel hand-annotated benchmarks designed to enable fact-checking and factual consistency evaluation in Urdu. While UrduFactBench focuses on claim verification, UrduFactQA targets the factuality of LLMs in question answering. These resources, the first of their kind for Urdu, were developed through a multi-stage annotation process involving native Urdu speakers. To complement these benchmarks, we introduce UrduFactCheck, a modular fact-checking framework that incorporates both monolingual and translation-based evidence retrieval strategies to mitigate the scarcity of high-quality Urdu evidence. Leveraging these resources, we conduct an extensive evaluation of twelve LLMs and demonstrate that translation-augmented pipelines consistently enhance performance compared to monolingual ones. Our findings reveal persistent challenges for open-source LLMs in Urdu and underscore the importance of developing targeted resources. All code and data are publicly available at https://github.com/mbzuai-nlp/UrduFactCheck.

## introduction in recent years, the way we find and share information has changed dramatically. large language models (llms) like gpt- @xcite are now capable of answering questions, generating articles, and even holding conversations that sound convincingly human. ## related work prior efforts in urdu fact-checking have focused mainly on fake news classification. the urdu-fake@fire2021 @xcite shared task framed this as a binary problem, highlighting generalization issues under domain shifts. ax-to-grind @xcite introduced a larger annotated dataset and applied multilingual models like mbert and xlnet. more recently, hook and bait urdu @xcite released the largest urdu fake news corpus, leveraging lora fine-tuning of llama-2 for mono-and multilingual detection. while these systems improved classification and dataset scale, they do not provide end-to-end factuality pipelines. in parallel, several urdu qa datasets have emerged. uqa @xcite , a spanpreserving translation of squad2.0 @xcite , has been used to benchmark multilingual models like mbert and xlm-roberta. other corpora such as uquad1.0 (kazi and khoja, 2021) support extractive qa but do not assess factual correctness. these resources target reading comprehension and general qa, rather than fact-checking or systematic evaluation of generated responses. multilingual benchmarks such as x-fact @xcite have tested llm factuality across several low-resource languages, but urdu remains underrepresented in these evaluations. at the same time, tools like factscore @xcite , factool @xcite , and factcheckgpt @xcite have advanced metrics, retrieval strategies, and modularity in fact-checking, but they are developed mainly for english and lack urduspecific datasets. this absence limits systematic evaluation of factuality in urdu and restricts the transfer of existing methods to this language. in contrast, this work introduces urdufact-bench and urdufactqa, the first datasets for fact-checking and factuality evaluation in urdu. urdufactbench supports claim verification, while urdufactqa evaluates the factual accuracy of llm-generated responses. to support their use, we also provide urdufactcheck, a modular framework with claim processing, evidence retrieval, and verification. together, these resources provide the first foundation for factuality evaluation in urdu, addressing gaps from prior work on classification or general qa. ## datasets to provide a foundation for evaluating automated fact-checkers and measuring the factuality of llms, we draw on three claim verification datasets and two qa datasets, curating them into urdu-factbench and urdufactqa, respectively. ## framework to address the challenge of evaluating factuality in urdu free-form text, we present urdufactcheck, a set of three end-to-end pipelines specifically tailored for the urdu language. the base framework consists of four core agent modules: claim-processor, querygenerator, retriever, and verifier, drawing on well-established automated fact-checking frameworks, as illustrated in figure 1 @xcite @xcite . ## experiments to evaluate urdufactcheck and our benchmarks, we conducted three experiments: (i) analyzing the effect of evidence thresholding on retrieval and verification, (ii) benchmarking automated factcheckers with urdufactbench, and (iii) assessing llm factuality with urdufactqa. we also report api costs for proprietary llms, gpu rental for open-source models, search engine query expenses, and total fact-checking time. open-source experiments ran on an nvidia rtx 6000 gpu (@xmath00.00105. ## conclusion we introduced two new benchmarks for factuality evaluation in urdu: urdufactbench for claim verification and urdufactqa for factual question answering. built using translation and dual annotation, these are the first publicly available datasets enabling systematic study of factuality in urdu, and filling a major resource gap for low-resource languages. in order to demonstrate the utility of these new datasets, we further developed urdufactcheck, a modular framework that integrates claim processing, evidence retrieval, and verification. it serves as a testbed for benchmarking fact-checking strategies and llms in urdu. using these resources, we conducted extensive experiments, including evaluations of twelve llms, revealing clear differences between proprietary and open-source models and underscoring both challenges and opportunities in urdu factuality. our main contribution is the release of two highquality benchmarks, supported by an open framework for experimentation. we hope that these resources will provide a foundation for advancing factuality assessment in low-resource languages, will foster cross-lingual transfer of methods, and will offer practical tools to address misinformation in urdu and beyond.