Multilingual Fact-Checking using LLM s (2024)

Abstract: Due to the recent rise in digital misinformation, there has been great interest shown in using LLMs for fact-checking and claim verification. In this paper, we answer the question: Do LLMs know multilingual facts and can they use this knowledge for effective fact-checking? To this end, we create a benchmark by filtering multilingual claims from the X-fact dataset and evaluating the multilingual fact-checking capabilities of five LLMs across five diverse languages: Spanish, Italian, Portuguese, Turkish, and Tamil on our benchmark. We employ three different prompting techniques: Zero-Shot, English Chain-of-Thought, and Cross-Lingual Prompting, using both greedy and self-consistency decoding. We extensively analyze our results and find that GPT-4o achieves the highest accuracy, but zero-shot prompting with self-consistency was the most effective overall. We also show that techniques like Chain-of-Thought and Cross-Lingual Prompting, which are designed to improve reasoning abilities, do not necessarily improve the fact-checking abilities of LLMs. Interestingly, we find a strong negative correlation between model accuracy and the amount of internet content for a given language. This suggests that LLMs are better at fact-checking from knowledge in low-resource languages. We hope that this study will encourage more work on multilingual fact-checking using LLMs.

## introduction in an era marked by the proliferation of digital misinformation, the need for fact-checking on a global scale has never been more pressing. recent research has shown promising capabilities in large language models (llms) for fact-checking and claim verification @xcite @xcite . however, this research has predominantly focused on english and chinese facts and claims, * equal contribution overlooking billions of people who do not speak these languages @xcite @xcite . in this paper, we evaluate the multilingual fact-checking capabilities of llms across five languages: spanish, italian, portuguese, turkish, and tamil, sourcing claims from the x-fact dataset @xcite . with this selection of languages, we ensure geographic and typological diversity and can probe llms' performance in low-resource as well as high-resource languages. we utilize a variety of prompting techniques, including zero-shot (baseline), english chain-of-thought @xcite , cross-lingual prompting @xcite , and self-consistency @xcite to evaluate the performance of llms. to our knowledge, this is the first work to assess the factual multilingual knowledge and inherent factchecking capabilities of a variety of llms across a spectrum of languages worldwide, using a variety of prompting techniques. the remainder of this paper is organized as follows: in section 2, we review related work. in section 3, we detail the datasets, models, and evaluation method used. in section 4, we discuss the prompting methods we use. in section 5, we present our results. in section 6 we analyze and interpret our findings and explore their implications. finally, we conclude in section 7 and suggest directions for future research. language claim in language english translation label spanish hoy la argentina tiene en el mundo el mejor grado de productividad por hectárea sembrada today argentina has the best degree of productivity per planted hectare in the world true (1) portuguese aqueles que se aposentam mais cedo são aqueles que ganham mais only the female aedes aegypti bites true (1) italian negli anni settanta il venezuela era tra i paesi più ricchi al mondo in the 1970s, venezuela was among the richest countries in the world false (0) turkish i̇skoçya'dan türkiye'ye uzanan 12 bin yıllık gizemli tüneller bulunduğu iddiası it is claimed that there are mysterious 12 thousand year old tunnels extending from scotland to turkey false (0) tamil தமிழ்நாட்டில் 10-ம் நூற்றாண்டிேலேய ெபண்களுக்கு ெசாத்துரிைம வழங்கப்பட்டுள்ளது என்பதற்கான கல்ெவட்டு ஆதாரங்கள் கிைடத்துள்ளன in tamil nadu, inscriptional evidence has been found that women were granted property rights as early as the 10th century true (1) figure 1: examples of claims in the testing datasets for each language, their english translations, and respective ground-truth label preference rankings, leading to improved factual accuracy without human labeling. cheung and lam 2023 incorporate external evidence-retrieval to bolster fact-checking performance for the llama 2 model. in comparison, our work examines llm fact-checking performance in several languages. multilingual fact-checking using llms numerous studies address the linguistic divide caused by focusing solely on llm-based fact-checking for english and chinese. however, the detailed exploration of the multilingual capabilities of llms for fact-checking beyond these two languages is limited. shafayat et al. 2024 examines the factual accuracy of gpt-3.5 and gpt-4 across nine languages and finds that the models exhibit an inherent bias towards factual political information from western continents. huang et al. 2022 augment mbert (a multilingual version of the language model bert) with cross-lingual retrieval to improve the fact-checking performance of llms on the x-fact dataset. cekinel et al. 2024 explores cross-lingual learning and low-resource fine-tuning for fact-checking in turkish. hu et al. 2023 benchmarks the factual knowledge possessed by ten different llms and their multilingual fact-checking capabilities in 27 languages. they also employ several different prompting techniques. however, their study predominantly focuses on smaller models (e.g., under 15b parameters). moreover, their multilingual analysis only distinguishes between en-glish and chinese. all other languages are benchmarked together in a mixed testing set, and interlingual comparisons (besides english and chinese) are not drawn. to the best of our knowledge, our study is the first to benchmark and closely analyze the multilingual fact-checking abilities of several llms across various domains, both political and non-political, using a range of different prompting techniques. ## experiments figure 2 displays the various prompting techniques we explore in this study. zero-shot we use zero-shot prompting to create a baseline for each llm's performance. we add the instruction "answer in english" to our zero-shot prompts to ensure that the llm's response is in english, as in preliminary tests the llm would, in some cases, generate outputs in the language specified in the system prompt. this issue is specific to the zero-shot setting. english chain-of-thought chain-of-thought (cot) prompting performs significantly better than zero-shot prompting on a variety of reasoning tasks @xcite including fact-checking and claim verification @xcite . in cot prompting, models are encouraged through k-shot examples to reason explicitly, in written-out steps. we employ english cot (en-cot) @xcite by adding the instruction "let's reason step-by-step in english" to the original instruction. @xcite builds on en-cot prompting and exhibits substantial performance improvements on multilingual reasoning tasks. a clp prompt includes a cross-lingual alignment prompt and a task-specific solver prompt. the output of the cross-lingual alignment prompt is included as context with the task-specific solver prompt, which generates the final output. ## conclusion and future work in this study, we assessed the performance of five large language models (llms) in verifying claims in five languages (spanish, portuguese, italian, turkish, and tamil) using the x-fact dataset. our findings indicate that both the choice of model and the prompting technique significantly impact fact-checking performance. notably, gpt-4o consistently achieved higher accuracy than the other models, likely due to its advanced architecture and larger size. interestingly, a simple self-consistency and zero-shot prompt combination outperformed all other prompting and decoding strategies, suggesting that not all reasoning strategies are beneficial for claim verification. strategies such as chainof-thought or cross-lingual prompting, which aim to alter the model's reasoning process, often had minimal or negative effects on success rates. in contrast, decoding strategies such as selfconsistency show potential as a future research direction. we also discovered a surprising correlation between higher model accuracy and lower language internet content, indicating that models performed better on low-resource languages. further investigation is needed to understand the causes and extent of this relationship. for future work, we plan to delve deeper into the relationship between model performance and the extent of a language's internet corpora. we will also develop and test new, custom-designed prompting techniques and decoding strategies specifically tailored to enhance claim-verification performance. additionally, we aim to experiment with other leading models such as claude 3 opus, gemini-1.5 pro, and the llama 3.1 model series. we will expand our study to include more high and low-resource languages from the x-fact dataset, such as french, russian, indonesian, and romanian.