Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent? (2025)

Abstract: Cross-lingual consistency should be considered to assess cross-lingual transferability, maintain the factuality of the model knowledge across languages, and preserve the parity of language model performance. We are thus interested in analyzing, evaluating, and interpreting cross-lingual consistency for factual knowledge.To facilitate our study, we examine multiple pretrained models and tuned models with code-mixed coreferential statements that convey identical knowledge across languages. Interpretability approaches are leveraged to analyze the behavior of a model in cross-lingual contexts, showing different levels of consistency in multilingual models, subject to language families, linguistic factors, scripts, and a bottleneck in cross-lingual consistency on a particular layer. Code-switching training and cross-lingual word alignment objectives show the most promising results, emphasizing the worthiness of cross-lingual alignment supervision and code-switching strategies for both multilingual performance and cross-lingual consistency enhancement. In addition, experimental results suggest promising result for calibrating consistency on test time via activation patching.

## introduction frege's theory of reference @xcite indicates that the knowledge conveyed by a sentence depends on the references of the expressions that make up the sentence. a salient aspect of humanity is that, while people may speak different languages, they can share common references and knowledge. thus, references and knowledge must be consistent across languages, and a multilingual model serving as a knowledge base @xcite @xcite should provide consistent knowledge when consulted in * equal contributions. we attempt to answer three questions: 1) do multilingual language models recall factual knowledge for the coreferential statements in a similar manner, 2) how does the mechanism of multilingual language models work on the incorporation between entities or references to convey knowledge in crosslingual settings, and 3) which factors prevent model consistency in multilingual settings? in addition to model consistency in cross-lingual settings, our study is related to a broader linguistic phenomenon of entity-level code-switching and language interference: an entity code-switches between two languages without changing the reference, as we create code-mixed coreferential statements from monolingual statements by substituting a subject entity with an equivalent one in another language that shares the same reference. more recently, we share a similar goal with knowledge incorporation and editing @xcite , since we incorporate a coreferential entity from other languages to recall factual knowledge in cross-lingual settings. our main findings are as follows. on the other hand, parallel samples providing cross-lingual generalization supervision offer limited gains to consistency. our contribution is to offer an understanding of multilingual language models' limitations under cross-lingual settings and highlight potential research directions to address such issues. ## transferable findings to other language bias throughout this paper, studies are conducted on code-mixed coreferential statements between english and other languages. however, references and knowledge are universal. this raises a question: are all findings transferable to other coreferential entities and statements in non-english-centric scenarios? to answer this question, we conduct experiments for llama-3.1-1b-instruct, mt0-base, and xlm-r-base, using fr, vi and hy as the matrix (l 1 ) languages. experimental results in figure 8 are consistent with our main findings, providing evidence that our findings can be transferred to other language bias. in addition, they discovered that language models are able to recall the correct knowledge in the middle layers using the english concept, which is consistent with @xcite ; @xcite . we trace consistent information and knowledge throughout the layers in cross-lingual settings, attempting to understand and interpret how commonly used strategies to improve multilingual models for downstream tasks could impact the crosslingual knowledge consistency. ## conclusion our analysis reveals that knowledge consistency is highly dependent on model architectures, training strategies, deep semantic alignments, and languagespecific information. our layer-by-layer analysis of multilingual models uncovers a consistency bot-tleneck whereby the consistency does not grow monotonically on each layer. our work highlights promising directions in the test-time calibration and training with cross-lingual alignment objectives to achieve knowledge consistency across languages, which will better preserve parity of language model performance and also alleviate such bottleneck. cross-lingual representations, shared scripts and parallel samples might contribute to the cross-lingual consistency but are not a sufficient condition to achieve it. we encourage researchers to work on representation learning approaches that induce crosslingual alignment inductive bias explicitly to enhance alignments between coreferential entities. we also suggest test-time approaches that calibrate output distributions for knowledge consistency across languages. these methods can alleviate the consistency bottleneck and enhance alignments between coreferential entities, potentially improving both multilingual performance and crosslingual consistency.