Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models (2023)

Abstract: Multilingual large-scale Pretrained Language Models (PLMs) have been shown to store considerable amounts of factual knowledge, but large variations are observed across languages. With the ultimate goal of ensuring that users with different language backgrounds obtain consistent feedback from the same model, we study the cross-lingual consistency (CLC) of factual knowledge in various multilingual PLMs. To this end, we propose a Ranking-based Consistency (RankC) metric to evaluate knowledge consistency across languages independently from accuracy. Using this metric, we conduct an in-depth analysis of the determining factors for CLC, both at model level and at language-pair level. Among other results, we find that increasing model size leads to higher factual probing accuracy in most languages, but does not improve cross-lingual consistency. Finally, we conduct a case study on CLC when new factual associations are inserted in the PLMs via model editing. Results on a small sample of facts inserted in English reveal a clear pattern whereby the new piece of knowledge transfers only to languages with which English has a high RankC score. All code and data are released at https://github.com/Betswish/Cross-Lingual-Consistency.

## introduction large-scale pre-trained language models (plms) have demonstrated powerful capabilities in tasks where factual knowledge plays an important role @xcite . while most previous work on probing factual knowledge in plms has focused on english @xcite @xcite @xcite , a few notable studies have extended the evaluation to a number of other languages @xcite @xcite . the results of these studies show a large variation in 1 all code and data released at https://github.com/ betswish/cross-lingual-consistency the extent to which factual knowledge generalizes across languages, revealing yet another facet of language inequality in modern nlp technologies @xcite . assessing factual knowledge across languages, however, is not a trivial endeavor. ensuring comparability of the results requires that a single set of 'universal' facts is queried in all languages, but the choice of that set is likely to be biased to specific world regions that are more represented in popular knowledge bases like wikidata. 2 conversely, facts that are more relevant in other regions of the world (e.g., information about locations or important persons in a given region) are less likely to occur in the benchmarks, which makes it hard to interpret the results of such evaluations. in this work, we take a different stance: instead of measuring the amount of factual knowledge encoded by a plm in each language, we focus on its consistency across languages. as illustrated in figure 1 , the multilingual bloom-3b model @xcite outputs consistently correct completions of the first prompt when queried in english, spanish, and vietnamese, but not in hungarian and greek. the model also outputs consistent, though wrong, answers to the second query in english, spanish, and vietnamese (but not in hungarian and greek), suggesting the first three languages share relevant knowledge representations within the model. the study of cross-lingual consistency (clc) is important for at least two reasons: firstly, true knowledge of a fact implies encoding of its meaning regardless of a given surface form @xcite . thus, if a model knows that the city of beijing is the capital of china, it should return the same answer when asked the same question in different languages. from a practical point of view, clc is crucial to ensure users have a similar experience when interacting with the same model in different languages. secondly, studying clc is important to understand whether and how knowledge acquired in a language gets implicitly transferred to another within multilingual plms. besides scientific relevance, this has practical implications for the incorporation of external knowledge into multilingual plms. in fact, while a prolific line of work has focused on model editing as a way to insert new factual associations in plms in various dataand computation-efficient manners @xcite @xcite , no one, to our knowledge, has looked yet at how this affects the factual knowledge in languages other than the one to which editing is directly applied. we conduct the first in-depth study of clc of factual knowledge in multilingual plms and make the following contributions: (i) we propose a novel ranking-based consistency (rankc) metric which assesses knowledge consistency independently from accuracy. (ii) we filter the existing unbalanced datasets @xcite to form a multi-parallel clc benchmark, balanced multilingual language model analysis (bmlama), which has the same set of prompts translated into all languages. (iii) we apply the new metric to bmlama to assess clc in various encoder-only, decoder-only, and encoderdecoder plms, including xlm-roberta-large, mt5-large, and bloom series. we analyze a number of language properties that correlate with clc and provide new insights into how factual knowledge percolates among languages. finally (iv) we present a case study with a state-of-the-art model editing technique based on neuron interpretability @xcite , providing preliminary evidence that clc is predictive of whether a fact inserted in language x will transfer to language y. ## related work probing factual knowledge in plms since first proposed by lama @xcite , promptbased probing has become the main technique to assess factual knowledge in plms @xcite @xcite @xcite . given the knowledge represented in a tuple (subject, relation, object), a query q is formed by filling the subject into a relation-specific template, which is fed into the plm. if the prediction is in line with the object, the model is considered to possess this knowledge. for instance, given a candidate set of city names, when queried with 'the capital of people's republic of china is _', the plm is considered to capture this piece of knowledge if it gives the highest probability to the correct answer 'beijing', among all candidates. ## measuring cross-lingual consistency task definition each language @xmath0.e. prompts) defined as q l . for each query q @xmath1, there are n i corresponding candidates @xcite . for example, the query 'steve jobs worked for __' has 10 candidates: apple, nintendo, google, wwe, alexandria, germany, yahoo, berlin, bbc, microsoft. each query is fed to the plm, and the returned probabilities are used to compute a ranking score for each of the candidate words. the specific score calculation depends on the type of model (encoder-only, encoder-decoder, or decoder-only) and on the way a candidate word gets segmented into subwords (see details in appendix b). after being sorted by ranking scores, the candidate set for q i is represented as {c 1 i , . . . , c n i i }, where c 1 i has the highest prediction probability and c n i i has the lowest. note that the existing multilingual datasets for knowledge probing (x-factr @xcite and mlama @xcite ) have different numbers of queries in different languages, which is problematic for measuring consistency. ## experimental setup datasets as explained in sect. 3.2, rankc requires that the queries and their candidates are translated into all evaluated languages. we therefore extract from x-factr @xcite and mlama @xcite all queries that fulfill this criterion. we call the resulting multiparallel dataset balanced multilingual language model analysis (bmlama), and release it in two versions: bmlama-17 including 6.7k queries in 17 languages (close to x-factr which includes 23 languages), and bmlama-53 including 3k queries in 53 languages (same as mlama). the detailed statistics are shown in models previous work on multilingual knowledge probing @xcite focused on encoder-only plms, such as mbert @xcite or xlm-roberta @xcite . however, since decoder-only plms have become mainstream in the current nlp era, our experiments also include the decoder-only bloom series (560m, 1.1b, 1.7b, 3b parameters) @xcite and the encoder-decoder mt5large (1.2b) @xcite , in addition to the encoder-only xlm-roberta-large (354m). en fr nl es ru ja zh ko vi el hu he tr ca ar uk fa bloom-3b avg: 25% en fr nl es ru ja zh ko vi el hu he tr ca ar uk fa en fr nl es ru ja zh ko vi el huhe tr ca ar uk fa mt5-large avg: 32% en fr nl es ru ja zh ko vi el hu he tr ca ar uk fa en fr nl es ru ja zh ko vi el hu he tr ca ar uk fa xlm-roberta-large avg: 32% en fr nl es ru ja zh ko vi el hu he tr ca ar uk fa ## main consistency results before looking at consistency, we present in figure 2 the factual probing accuracy results of the three plms on bmlama-17. 5 we first notice that the encoder-only xlm-roberta-large and encoderdecoder mt5-large models outperform the whole decoder-only bloom series in terms of average probing accuracy. the trend across languages is similar among the three models, however, bloom stands out with an english accuracy that is much higher than all other languages. regarding model size (bloom series, green bars), we find that increasing the number of parameters leads to slight but consistent improvements in factual probing accuracy, which is in line with previous work @xcite . our xlm-roberta-large results are consistent with those reported by @xcite on x-factr, which demonstrates the reliability of our multi-parallel dataset bmlama. 5 the experiments in this section use bmlama-17 as it enables better visualization. results for bmlama-53 are given in appendix h and discussed in the analysis of section 6. ## factors of cross-lingual consistency in this section, we examine the different degrees of clc that are observed in different language pairs, and the factors that may explain such differences. ## case study: cross-lingual consistency and knowledge incorporation previous work @xcite @xcite and our probing results indicate the amount of knowledge in low-resource languages is limited. simply training a new plm on larger non-english corpora is time-consuming and the cost is not affordable by most universities and other research institutions @xcite . a promising solution is to incorporate external knowledge through fine-tuning methods @xcite or directly editing the plm's weights in a very targeted way @xcite . to make the process feasible in multilingual scenarios and avoid unexpected effects, it is important to understand whether and how inserting a piece of knowledge in one language affects other languages within the plm, including the most and least susceptible languages. in this section, we conduct a first case study on this problem and its interplay with clc. ## conclusion we analyzed the cross-lingual consistency (clc) of factual knowledge in multilingual large-scale plms. we proposed a new metric, rankc, to quantify consistency independently from accuracy, and applied it to a cross-lingually balanced factual knowledge benchmark. our comprehensive analysis shows that (i) average clc is low across different plms and not visibly affected by model size; taken together, our findings suggest that factual knowledge may primarily propagate across languages in a shallow way, that is, through shared word embeddings rather than through deeper language-agnostic representations. while this result has negative implications on the generalization abilities of modern plms, it could also be turned into an advantage to design efficient methods for knowledge incorporation in multilingual scenarios.