V i N um FCR : A Novel V ietnamese Benchmark for Numerical Reasoning Fact Checking on Social Media News (2025)

Abstract: In the digital era, the internet provides rapid and convenient access to vast amounts of information. However, much of this information remains unverified, particularly with the increasing prevalence of falsified numerical data, leading to public confusion and negative societal impacts. To address this issue, we developed ViNumFCR, a first dataset dedicated to fact-checking numerical information in Vietnamese. Comprising over 10,000 samples collected and constructed from online newspaper across 12 different topics. We assessed the performance of various fact-checking models, including Pretrained Language Models and Large Language Models, alongside retrieval techniques for gathering supporting evidence. Experimental results demonstrate that the XLM-R_Large model achieved the highest accuracy of 90.05% on the fact-checking task, while the combined SBERT + BM25 model attained a precision of over 97% on the evidence retrieval task. Additionally, we conducted an in-depth analysis of the linguistic features of the dataset to understand the factors influencing the performance models. The ViNumFCR dataset is publicly available to support further research.

## introduction with the rapid growth of the internet, information is shared widely and instantly, keeping users updated on many topics. however, this also enables the spread of unverified content, which poses significant risks. in vietnam, fake news and misinformation have harmed people's health, finances, families, and reputations. as a result, fact-checking has become essential in fields like journalism and social media. while there has been extensive research for languages such as english and chinese, vietnamese remains underexplored. in particular, previous studies have rarely focused on verifying numerical information -an important yet overlooked aspect. motivated by this, we design and develop a vietnamese fact-checking task centered on numerical verification, aiming to help bridge this gap and enhance the reliability of digital information. the fact checking task @xcite is a process of verifying the accuracy of information, data, or events before they are published to the public. the goal of fact checking @xcite is to determine the correctness of information based on verifiable evidence, such as official documents, reliable sources, or accurately recorded events. in this paper, we describe our task as follows: given a vietnamese sentence containing numerical data a and a vietnamese text passage containing numerical data b, the objective is to develop a system that can verify the accuracy of sentence a by identifying supporting evidence in text b. ## corpus creation we conducted the data construction following a rigorous process to ensure the accuracy, reliability, and quality of the dataset. the process was referenced from the vinli and fever dataset dataset text genre quantity language 3+ labels agree 4+ labels agree numerical focus fever wikipedia ∼185,000 english --no liar newswire ∼12,800 english --no feverous wikipedia 87,026 english --no vitaminc wikipedia ∼400,000 english --no vifactcheck wikipedia ∼7,200 vietnamese --no viwikifc newswire ∼20,000 vietnamese --no vinumfcr (our dataset) newswire ∼10,000 vietnamese 94% 85% yes ## result analysis to better understand the impact of various aspects on performance outcomes, we conducted an analysis on the three models with the highest performance results: phobert large , cafebert, mbert, gemma and xlm-r large . length affects model performance: the impact of text length on model performance on the dev set for cafebert, phobert large , mbert, gemma and xlm-r large is illustrated in figure 7 . when analyzing the combined length of the premise paragraph and the claim, the xlm-r large and cafebert models achieve the best performance when the total length exceeds 150 words. in contrast, the performance of mbert and gemma drops significantly when the total length exceeds 150 words. meanwhile, phobert large maintains relatively stable accuracy across different length ranges. word overlap affects model performance: we analyzed the impact of the jaccard index on model accuracy similarly to that on the vinli dataset @xcite . the performance analysis on the dev set, as shown in figure 8 , indicates that the mbert model achieves the lowest accuracy when the jaccard index falls within the 20% to 30% range. when the jaccard index exceeds 50%, the performance of mbert continues to decline, with an even more pronounced drop observed in the phobert large model. in contrast, within the same jaccard range, the xlm-r large , cafebert, and gemma models achieve the highest accuracy. syntax tree affects model performance: the impact of the syntactic tree level on model performance on the dev set is illustrated in figure 9 . the mbert, cafebert, gemma, and xlm-r large models achieve the highest performance when the syntactic tree level is 3. however, the phobert large model performs best when the tree level is only 1. furthermore, as the tree level increases from 4 onward, the performance of all models tends to decline, with the most noticeable drop observed in the mbert model when the level reaches 4. topic affects model performance: the analysis of topic influence on model performance, as shown in ## conclusion and future work this paper introduces the vinumfcr dataset, a pioneering benchmark for numerical reasoning fact-checking in vietnamese, comprising over 10,000 claims by a rigorous annotation process. we assessed advanced language models including plms, llms, revealing that models such as xlm-r large and cafebert achieved significant accuracy. however, we observed that zero-shot and few-shot techniques with qwen, llama, and gemma models for reasoning and generating predictions proved ineffective, highlighting the unique challenges posed by this dataset. these findings emphasize vinumfcr's role as a robust resource for real-world applications and the advancement of vietnamese nlp research, particularly in managing numerical data on social media. looking ahead, we intend to enrich the dataset by increasing its scale and diversity through expanded data collection and thorough cleaning procedures. we plan to explore fine-tuning techniques on alternative machine learning models, especially large language models, to further enhance perfor-mance, particularly for complex fact-checking samples. moreover, we will tackle current limitations by improving models' ability to process longer and more intricate text passages, thereby boosting the system's reliability for real-world news verification deployments. no. rule example 1 change the sentence structure from active to passive and vice versa. p: một con mèo đang chơi với cuộn len. (a cat is playing with a ball of yarn.) c: cuộn len đang bị một con mèo chơi. (the ball of yarn is being played with by a cat.) 2 replace with synonyms or similar words. p: tôi đi siêu thị mua thực phẩm. (i went to the supermarket to buy groceries.) c: tôi đi siêu thị mua đồ ăn. (i went to the supermarket to buy food.) 3 add or remove modifiers while retaining the original meaning of the sentence. p: cô giáo 23 tuổi xinh đẹp đang dạy tiếng anh. (the beautiful 23-year-old teacher is teaching english.) c: cô giáo 23 tuổi đang dạy tiếng anh. (the 23-year-old teacher is teaching english.) 4 replace representative nouns with relative clauses. p: một cầu thủ đá bóng nam đang sút bóng vào khung thành. (a male soccer player is kicking the ball into the goal.) c: người đàn ông, cái người mà chơi bóng đá đang sút một quả bóng vào khung thành. (the man, who plays soccer, is kicking a ball into the goal.) 5 replace the object with a relative clause. p: người phụ nữ đang dùng một chiếc máy may. (the woman is using a sewing machine.) c: người phụ nữ đang sử dụng một chiếc máy được sản xuất để may. (the woman is using a machine made for sewing.) 6 replace the adjective with a relative clause. p: hai người đàn ông đang nghỉ ngơi sau một chuyến đi trên con đường tuyết. (two men are resting after a trip on a snowy road.) c: hai người đàn ông đang nghỉ ngơi sau một chuyến đi trên con đường mà nó được phủ đầy tuyết. (two men are resting after a trip on a road that is covered with snow.) 7 replace the quantity terms with equivalent ones. p: vài người đang lướt sóng trên một con sóng lớn. (several people are surfing on a big wave.) c: một số người đang lướt sóng trên một con sóng lớn. (some people are surfing on a big wave.) 8 generate a presupposition sentence. p: tôi đã lạc mất con mèo duy nhất của tôi vào sáng nay. (i lost my only cat this morning.) c: tôi có một con mèo. (i have a cat.) 9 others.