Tracing Multilingual Factual Knowledge Acquisition in Pretraining (2025)

Abstract: Large Language Models (LLMs) are capable of recalling multilingual factual knowledge present in their pretraining data. However, most studies evaluate only the final model, leaving the development of factual recall and crosslingual consistency throughout pretraining largely unexplored. In this work, we trace how factual recall and crosslingual consistency evolve during pretraining, focusing on OLMo-7B as a case study. We find that both accuracy and consistency improve over time for most languages. We show that this improvement is primarily driven by the fact frequency in the pretraining corpus: more frequent facts are more likely to be recalled correctly, regardless of language. Yet, some low-frequency facts in non-English languages can still be correctly recalled. Our analysis reveals that these instances largely benefit from crosslingual transfer of their English counterparts – an effect that emerges predominantly in the early stages of pretraining. We pinpoint two distinct pathways through which multilingual factual knowledge acquisition occurs: (1) frequency-driven learning, which is dominant and language-agnostic, and (2) crosslingual transfer, which is limited in scale and typically constrained to relation types involving named entities. We release our code and data to facilitate further research at https://github.com/cisnlp/multilingual-fact-tracing .

## introduction despite being predominantly trained on englishcentric data, llms exhibit surprisingly strong multilingual capabilities across a wide range of tasks @xcite @xcite . notably, they can recall factual knowledge in multiple languages @xcite @xcite . however, these models frequently exhibit crosslingual inconsistencies -answering a factual query correctly in one language but failing to do so in another @xcite @xcite . although bilinguals typically recall information more effectively when the language of encoding matches the language of retrieval, they can usually recall factual knowledge learned in one language using their other proficient language @xcite -highlighting a flexibility that contrasts with the inefficiencies seen in llms. understanding this discrepancy requires deeper insight into how multilingual factual knowledge is acquired. while prior work has investigated mechanisms of (multilingual) factual recall @xcite @xcite @xcite and analyzed sources of crosslingual inconsistency @xcite , these studies have largely focused on final models, drawing conclusions solely from the end of pretraining. as a result, the developmental process by which llms acquire factual knowledge across languages remains poorly understood. to address this gap, we trace the dynamics of multilingual factual recall and crosslingual consis-tency throughout pretraining. rather than treating factual recall as a static outcome, we analyze its emergence across checkpoints using olmo-7b @xcite , an english-centric decoder-only llm pretrained on dolma @xcite . our analysis evaluates both accuracy within individual languages and consistency across languages for facts that are parallel in all languages. in addition, we investigate the key factors that contribute to correct multilingual factual recall. prior work has shown that the frequency of an instance can significantly influence performance relating to it, including factual prediction @xcite @xcite . motivated by these findings, we hypothesize that fact frequency in the pretraining corpus plays a central role in multilingual factual recall. to test this, we compute the frequency of each fact and systematically link it to factual recall across languages and pretraining stages. we summarize the key findings of this paper: ## multilingual factual recall dynamics we begin our analysis by tracing how factual recall performance evolves throughout pretraining across different languages. specifically, we examine both accuracy and crosslingual consistency at each checkpoint of olmo (cf. §3.1) using the klar dataset. figure 2 summarizes these results for eight languages (see §b for full results). crosslingual consistency is tightly coupled with non-english performance. we observe that the trajectory of crosslingual consistency in each language l ̸ = eng_latn closely mirrors its own factual accuracy throughout pretraining. this suggests that consistency is primarily driven by whether the fact is correctly recalled in l, which almost always implies that it is also recalled in english. the implication is twofold. (1) for non-english languages, the consistency of a language (co) is effectively gated by its performance (acc). (2) the limited capability of the model to transfer knowledge from english to other languages, referred to as the crosslingual knowledge barrier @xcite , is a persistent problem throughout pretraining. factual knowledge is acquired rapidly in early pretraining phases. we observe that factual recall performance (acc) improves very quickly in the early stages of pretraining for many languages. for example, english reaches approximately 80% script plays a more important role than language family in sustained improvements. languages such as ara_arab, jpn_jpan, and kor_kore, which neither use the latin script nor belong to the indo-european family, reach early saturation in performance -typically even before 2k steps. in contrast, latin-script languages such as cat_latn, fra_latn, and spa_latn, continue to improve with more training steps. interestingly, ell_grek, despite being an indo-european language, saturates early as well, whereas tur_latn, from the turkic family, benefits from extended pretraining. this pattern suggests that surface features like script similarity are more influential for possible crosslingual knowledge transfer than deeper typological relationships, as we further investigate in §6. ## fact frequency as predictor figure 4 : relationship between fact frequency and factual recall for all languages and six pretraining checkpoints. high-frequency facts are more likely to be correctly recalled than rare ones. this frequencycorrectness correlation emerges early in pretraining and becomes more pronounced over time. a form of memorization, where frequent exposure to specific facts in the pretraining corpus enables the model to retrieve them accurately. to investigate this, we approximate the frequency of all facts in the klar dataset (cf. §3.4) and analyze the relationship between frequency and factual recall performance both "globally" -across all languages -and "locally" -within individual languages. ## investigation of transfer effect we observed a substantial number of false negatives when using frequency as a predictor in §5.3, particularly for languages that use non-latin scripts. this is counterintuitive given the strong role frequency typically plays in factual recall. we hypothesize that these cases are due to the crosslingual transfer effect -factual knowledge is primarily learned in english and is successfully transferred to other languages. in the following sections, we present a detailed analysis of these false negatives identified in §5.3 -which we will refer to as surprisingly correct low-frequency predictions (sclfps). ## conclusion we investigate how multilingual factual recall and crosslingual consistency emerge during pretraining, using olmo-7b as a case study. our analysis shows that factual recall improves early and is primarily driven by fact frequency, regardless of language. however, some low-frequency facts in non-english languages can still be recalled, mainly due to crosslingual transfer from english -especially for relations that involve named entities. we therefore conclude that multilingual factual knowledge is gained through both frequency-driven learning and crosslingual transfer starting from early stages.