Do We Need Language-Specific Fact-Checking Models? The Case of C hinese (2024)

Abstract: This paper investigates the potential benefits of language-specific fact-checking models, focusing on the case of Chinese using CHEF dataset. To better reflect real-world fact-checking, we first develop a novel Chinese document-level evidence retriever, achieving state-of-the-art performance. We then demonstrate the limitations of translation-based methods and multilingual language models, highlighting the need for language-specific systems. To better analyze token-level biases in different systems, we construct an adversarial dataset based on the CHEF dataset, where each instance has a large word overlap with the original one but holds the opposite veracity label. Experimental results on the CHEF dataset and our adversarial dataset show that our proposed method outperforms translation-based methods and multilingual language models and is more robust toward biases, emphasizing the importance of language-specific fact-checking systems.

## introduction since manual fact-checking requires significant time and resources, there has been growing interest in automated fact-checking in recent years @xcite . while misinformation exists in various languages, studies have predominantly focused on claims and evidence in english @xcite . current research in fact-checking in other languages often lacks grounding in real-world claims @xcite or is constrained to a single domain, like @xcite . in this paper, we raise the question: should we develop language-specific fact-checking models, or can we effectively utilize existing english models by translating claims and evidence into english? we present a case study focused on mandarin chinese to investigate it for two reasons. firstly, chi-1 https://github.com/caiqizh/fc_chinese original: 广东两名小学生提干，引发大量讨论。 translated: two primary school students in guangdong raised eyebrows (were promoted), sparking discussion. chatgpt: refuted chef label: supported claim 1: 中国超八成地下水遭受污染，不能饮用。 (over 80% of china's groundwater is polluted and is unfit for drinking.) claim 2: 中国高铁辐射严重引发不孕。(radiation from china's high-speed rail seriously causes infertility.) chatgpt: supported chef label: refuted to study the question proposed, we first develop a novel chinese document-level evidence retriever (dlr), which outperforms state-of-the-art models by 10% in terms of accuracy and macro f1 in chef. paired with either dlr-retrieved or annotated gold evidence, we then demonstrate the limitations of translation-based methods (i.e., first translating chinese claims and evidence into english and then applying english fact-checking models on translated data) or multilingual language models in fact-checking. we further identify the cultural biases in chef and create an adversarial dataset. experiments on our newly proposed dataset show a significant decrease in both accuracy and f1 score. overall, our study highlights the necessity of devising language-specific fact-checking models. ## language-specific models vs translation and multilingual models to construct a chinese fact-checking system, two straightforward approaches are direct translation from chinese to english and the application of multilingual llms. however, as demonstrated in retrievers evidence retrieval is crucial in the fact-checking pipeline @xcite . to align our work with realworld fact-checking, we develop a state-of-theart chinese evidence retrieval component. our novel document-level retriever (dlr) enhances evidence retrieval by considering the context of evidence sentences, unlike prior approaches that isolate evidence selection to pairwise sentence classification @xcite . inspired by stammbach (2021), we train a retriever to assign a score to each chinese token within an evidence document and then aggregate these token scores at the sentence level. in particular, we fine-tune a bigbird @xcite to assign a value of 1 to tokens that belong to annotated evidence for a claim, while assigning a value of 0 to all other tokens. during inference, we compute the average scores for all tokens within each sentence. if the resulting average score exceeds 0.5, we classify the sentence as evidence. we compare our proposed document-level retriever with the annotated gold evidence, and semantic ranker @xcite used in chef. to eliminate the effects of evidence retrieval, we also utilize the gold evidence for the ideal case. more training details of dlr can be found in appendix a. verifiers for the verifiers in results on chef as shown in for the multilingual lms, the cross-lingual approach (gpt-4+mdeberta) show poor performance. this is probably because of the disparity between the languages of the training and testing sets. on the contrary, mdeberta, trained on chinese chef, achieves competitive performance on par with chinese roberta and outperforming several baseline models. overall, the results suggest that training a model specifically on chinese, whether it be a multilingual or monolingual model, is more beneficial than relying on english-centric ones. evidence retrieval the dlr, paired with different verifiers, improves accuracy and macro f1 by about 5% over the semantic ranker. regarding the recall of human-annotated gold evidence, dlr leads to 10% higher recall@5 ( ## biases in chef to explore the reasons behind the deficiency of translation services and multilingual lms, we investigate the biases present in the chef dataset in this section. prior research has demonstrated that fact-checking datasets, such as fever @xcite and multifc @xcite , result in training models that rely on heuristics such as surface-level patterns within claims, potentially impeding their ability to generalize effectively @xcite . in this section, we show that while biases are present as in the english language datasets and models, they are specific to the chinese culture. domain bias first, in chef, claims are categorized into domains such as politics, society, health, and culture and we find a significant skew in the distribution: 64% of social and 66% of health claims are refuted, while 55% in politics and 72% in culture are supported. notably, there is an imbalance in the proportion of social and health claims, which collectively constitute 68% of the total. figure 2 in the appendix details the label distribution across domains. cultural bias we then examine the correlation between phrases within the claims and the corresponding labels. the word distribution within the training set is analyzed for this purpose. initially, all claims in the training set are tokenized by chinese text segmentation tool, jieba. 3 the average length of the words is 2.39 characters. then, two metrics are employed to assess the correlation between phrases and labels. following schuster et al. prior studies in english datasets, such as constraint @xcite one possible reason behind this is that factchecking in china tends to avoid criticism of hardcore public issues, such as politics, economics, and other current affairs (liu and zhou, 2022). on the contrary, it focuses more on providing references for everyday decision-making, such as in health. another political reason could be that the cyberspace administration of china keeps a close watch on online news services @xcite . non-state enterprises are not permitted to criticize politics, economics, and other current affairs. private companies are only authorized to distribute and curate news produced by state-owned media. furthermore, in chef, certain regions such as 台湾 (taiwan), 日本 (japan), and 美国 (united states) are commonly associated with the refuted label. this may also reflect the contentious nature of international relations within the realm of chinese fact-checking. ## adversarial dataset construction our analysis revealed the presence of labels and cultural biases specific to the chinese context ( § 3). these biases can significantly impact the performance and fairness of fact-checking models when applied to chinese-language claims. we therefore introduce an adversarial dataset derived from the chef dataset for a better evaluation of the models. inspired by @xcite , to create it we pair each claim-evidence instance with a synthetic counterpart where claim and evidence have high word overlap with the original ones but the opposite veracity label (figure 3 ). under this setting, determining veracity from the claim alone would be equivalent to a random guess. instead of involving human annotators, we opt for the utilization of gpt-4 to generate the dataset. to control the quality, we invited two chinese native speakers to annotate randomly sampled 25% of claim-evidence pairs with supported, refuted or not enough info. the results demonstrated strong agreement between humans and gpt-4. they agreed with the dataset labels in 89% of cases, with a cohen κ of 0.80 @xcite . our approach overcomes labor-intensive manual annotation and rigid rulebased generation, advocating for automated sample generation using llms. this new test set nullifies the benefit of relying exclusively on cues from claims. details of the dataset construction and the prompt we use can be found in appendix d. ## experiments on adversarial chef results on adversarial chef all models perform worse on adversarial examples compared to the original chef. specifically, we highlight the following findings: (1) chinese deberta drops from 86.69% accuracy on original pairs to 57.84% and 65.01% on adversarial subsets. baselines similarly see over 37% decreases in both accuracy and f1 scores. this underscores the models' reliance on surface features and reveals the aforementioned biases. (2) compare the translation-based gpt-4+deberta (f1 53.04%), the multilingual mdeberta (f1 60.56%), and the chinese deberta (f1 63.74%), we observe a performance increase with more language-specific models, highlighting the benefits of incorporating chinese data during pre-training. (3) gpt-3.5-turbo, which has not been fine-tuned on the chef dataset, demonstrates better robustness but still underperforms compared to the language-specific chinese deberta. overall, we recommend future research to use both the original and our adversarial chef dataset for a comprehensive evaluation. exposing adversarial examples to the models can improve robustness. deberta's performance declines less than that of the baselines including bert, attention, and graph-based models when faced with adversarial examples, about 30% compared to over 37%, suggesting a higher sensitivity to evidence changes. to investigate the reasons behind the decrease in the model's performance and improve model's robustness, we employ the inoculation fine-tuning method @xcite . results show the performance decline observed in the baselines primarily stems from inherent weaknesses within the model family. in contrast, for the deberta model, gradually exposing it to more adversarial samples leads to a gradual reduction in the performance gap. inoculation results by fine-tuning the model with different sizes of adversarial examples are provided in figure 4 in the appendix f. ## conclusion our study reveals the shortcomings of englishcentric fact-checking systems when applied to chinese claims, highlighting the failure of translationbased methods due to linguistic and cultural nuances. we introduce a novel system that achieves best-reported results on chef and provides an adversarial dataset for continued research, underscoring the need for specialized fact-checking models.