CCL - XC o T : An Efficient Cross-Lingual Knowledge Transfer Method for Mitigating Hallucination Generation (2025)

Abstract: Multilingual Large Language Models (MLLMs) demonstrate strong generalization across languages, yet they remain prone to hallucinations, especially in low-resource languages, due to training data imbalances. These hallucinations, which include inaccurate or fabricated outputs, are particularly problematic in domain-specific generation tasks (Chataigner et al., 2024). To address this challenge, we propose CCL-XCoT (Curriculum-based Contrastive Learning-based Cross-lingual Chain-of-Thought), a two-stage fine-tuning framework for mitigating hallucination in MLLMs. Our approach first enhances cross-lingual semantic alignment through curriculum-based contrastive learning combined with next-token prediction during continued pre-training. Building on this foundation, we then introduce a cross-lingual Chain-of-Thought (XCoT) prompting strategy during instruction fine-tuning, which guides the model to reason in a high-resource language before generating answers in the target low-resource language. Experimental results show that CCL-XCoT reduces hallucination rates by up to 62% and substantially improves factual knowledge transfer across language pairs, without relying on external retrieval or multi-model ensembles.

## introduction despite the impressive performance of large language models (llms) across a wide range of natural language processing tasks, hallucination remains a fundamental and unresolved challengeparticularly in multilingual settings. this issue is especially pronounced in low-resource languages, where limited training data increases the likelihood of inaccurate or misleading outputs. hallucinations in llms typically manifest in three forms: a major contributor to context-related hallucinations is the asymmetry in knowledge coverage between high-resource and low-resource languages. recent studies have shown that multilingual models often fail to share factual knowledge across languages @xcite . this limitation is partially attributed to the languageagnostic next-token prediction (ntp) pretraining objective, which does not explicitly enforce crosslingual semantic alignment. as a result, multilingual llms struggle to generalize factual knowledge to underrepresented languages. existing solutions such as retrieval-augmented generation (rag) have demonstrated effectiveness in english, but perform poorly in low-resource settings due to unreliable retrieval and limited language coverage @xcite . other strategies, such as chain-of-thought (cot) prompting @xcite @xcite , improve factual accuracy via step-by-step reasoning. however, cot assumes the model already possesses sufficient internal knowledge in the target language-an assumption that often breaks down in low-resource conditions. to address these gaps, we propose ccl-xcot, a two-stage framework for improving factual generation in low-resource languages without relying on external retrieval. our approach comprises two key components: (i) a curriculum-based contrastive learning strategy combined with next-token prediction during continued pretraining to align semantic spaces between high-and low-resource languages, and (ii) a cross-lingual chain-of-thought strategy during instruction fine-tuning, which enables mod-els to reason in a high-resource language (e.g., english) before generating answers in the target lowresource language. this design facilitates the transfer of non-language-specific knowledge, such as common facts and historical information, across languages. this paper makes the following contributions: (i) we propose ccl-xcot, a two-stage framework that reduces hallucinations in low-resource languages by up to 62%, without requiring retrieval systems or ensemble models. (ii) we introduce curriculum-based contrastive learning to align multilingual semantic spaces, improving factual understanding and yielding up to 20% gains on cross-lingual nlu tasks. (iii) we develop xcot strategy that transfers reasoning patterns from high-resource to low-resource languages, enhancing answer accuracy and completeness. (iv) our layer-wise analysis reveals that mid-level transformer layers are key to effective cross-lingual knowledge transfer, guiding future efficient finetuning. ## related work llms have achieved strong performance in text generation tasks, but remain prone to hallucinations that undermine output reliability. to mitigate this, a range of strategies have been proposed. retrieval-based hallucination mitigation. rag integrates external knowledge to improve llm outputs' factual accuracy @xcite @xcite . for instance, @xcite retrieved relevant task templates using minilm embeddings, while @xcite proposed hallucination aware tuning (hat), which uses hallucination detection and gpt-4-based revision to build a preference dataset for direct preference optimization(dpo). despite promising results in english, rag remains less effective in low-resource languages due to poor retrieval and limited training coverage @xcite . moreover, rag-based systems may generate conflicting outputs if the retrieved context is misinterpreted. ## methodology we propose a framework to improve decoder-only language models for low-resource languages by integrating ntp, two-stage cross-lingual contrastive learning, and cot reasoning. unlike standard causal language modeling, our method explicitly aligns multilingual semantic spaces through contrastive learning and transfers knowledge from high-resource to low-resource languages via cot prompting. an overview of the framework is illustrated in figure 1 . ## logit lens analysis of ccl to further investigate the impact of incorporating contrastive learning during the pre-training stage on the base model, we employ logit lens (nostalgebraist, 2020), a tool that projects hidden states at each layer onto the vocabulary space, allowing us to analyze the evolution of intermediate states during inference. as discussed earlier, cross-lingual knowledge transfer and reasoning typically occur in the middle-to-upper layers of the model. therefore, we focus on layers 18-26 (out of 28 layers in gemma-7b, indexed from 0 to 27). in the fig-ure, each column represents a time step: except for the initial step, the input at each subsequent step is derived from the output of the previous step, with the token at the top of each column denoting the current input. as shown in figure 6a , when the initial token "中国"is given and the model is prompted to continue generation, the intermediate layers of the baseline gemma-7b, when projected through logit lens, often correspond to semantically irrelevant or meaningless tokens. its reasoning process appears to occur in a language-agnostic space. in contrast, after applying ccl, as illustrated by the light green highlight in figure 6b , even when the prompt is in chinese, the model's intermediate reasoning process tends to proceed in english. before generating the next chinese token, it frequently associates with corresponding or related english tokens. this demonstrates that ccl significantly enhances cross-lingual alignment, enabling the model to leverage high-resource languages during inference for low-resource language prompts, thereby facilitating more effective crosslingual knowledge transfer. ## case study to better understand hallucination behavior across languages, we conduct a targeted case study comparing model outputs in medium-resource languages (chinese and malay) and a low-resource language (tamil). after direct instruction finetuning, we observe distinct error profiles. in chinese and malay, hallucinations typically fall into two categories: unnecessary elaborations beyond the scope of the prompt, and misinterpretations of culturally specific or domain-relevant terms, such as cendol or the home ownership scheme. in contrast, hallucinations in tamil are predominantly severe, with over 90% of cases producing responses that are entirely unrelated to the input question, highlighting the fragility of the model's representations in truly low-resource settings. introducing the xcot strategy leads to substantial improvements across all languages. for chinese and malay, xcot reduces over-generation errors by encouraging more focused and structured reasoning. however, challenges with domainspecific terminology persist, as these terms are often underrepresented in the training data. in tamil, xcot improves response relevance by providing a clearer reasoning scaffold, but some hallucinations remain, often manifesting as topic drift or vague associations. when xcot is combined with crosslingual contrastive learning, we observe further improvements in all three languages, particularly in handling terminology. in medium-resource settings, hallucinations are reduced to minor factual inaccuracies, such as incorrect years or road names, indicating a transition from gross misunderstanding to finer-grained errors. in tamil, hallucinations become more diverse but notably less frequent, suggesting better control over output semantics despite limited language-specific training signals. these findings highlight the importance of tailoring multilingual llm interventions based on language resource availability. for mediumresource languages, enhancing factual precision and grounding in culturally specific terminology should be prioritized. for low-resource languages like tamil, future improvements will likely require more robust semantic alignment mechanisms and adaptive reasoning strategies to address a broader range of hallucination types and maintain answer relevance. additional qualitative examples are included in section a.5 to further illustrate these patterns. ## conclusion this paper present ccl-xcot, an efficient twostage framework for cross-lingual knowledge transfer in multilingual llms. our approach combines curriculum-based contrastive learning during continued pretraining with xcot strategy during instruction fine-tuning. together, these components align semantic representations across languages and scaffold reasoning through high-resource language traces, substantially reducing hallucinations in low-resource question answering tasks. extensive experiments show that contrastive learning improves semantic alignment and consistency, while xcot enables more accurate and faithful generation across diverse language pairs. our layer-wise analysis reveals that cross-lingual transfer is most effective when concentrated in midlevel layers, offering a practical path toward efficient multilingual adaptation. case studies further demonstrate how hallucination patterns vary by resource level, highlighting the need for differentiated strategies. looking ahead, future work may explore integrating multilingual retrieval augmentation, expanding xcot-style reasoning to additional modalities, and developing lightweight adapters to support broader cross-lingual generalization, especially in zero-shot or resource-sparse scenarios.