Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge in Foundation Models (2023)

Abstract: In this work, we assess the ability of foundation models to recall encyclopedic knowledge across a wide range of linguistic contexts. To support this, we: 1) produce a 20-language dataset that contains 303k factual associations paired with counterfactuals, 2) evaluate 5 models in a multilingual test, and 3) benchmark a diverse set of 24 models in an English-only test. Meta’s LLaMA achieves the highest scores in both multilingual and English-only evaluations. Yet, an analysis of LLaMA’s errors reveals significant limitations in its ability to recall facts in languages other than English, plus difficulties related to the location and gender of fact subjects. Overall, our findings suggest that today’s foundation models are far from polyglots.

## introduction can foundation models be used as multilingual knowledge bases? foundation models typify an emerging paradigm that warrants further study; all-purpose large language models (llms) that are trained on internet-scale corpora excel in generalization to some new tasks @xcite @xcite . their widespread adoption and ostensible credibility come with risks, though. for instance, foundation models inherit inaccuracies from training corpora @xcite , which are in turn propagated downstream to the models that are fine-tuned from them @xcite . additionally, foundation models spend the majority of their training phase absorbing information in english; for example, @xcite 's llama devotes two-thirds of its training dataset to an english-only subset of the common-crawl @xcite . thus, foundation models are potentially deficient when performing non-english tasks @xcite . to benchmark how robustly llms learn factual associations, @xcite and @xcite evaluated the encyclopedic knowledge of models like bert @xcite and roberta @xcite using rank-based approaches. our study builds off this research in a few ways. we utilize a contrastive scoring approach, which tests the extent to which a model grasps a concept with more rigor than rank-based methods, as detailed below. additionally, we inspect a diverse group of causal and masked language models rather than testing a single architecture to capture a more representative view of the field. ## task we formulate the polyglot of not? test with cloze statements: given some context, we prompt an llm to predict the next token. factual associations are formalized as the triplet ⟨s, r, o⟩ where s and o denote the subject and object entity and r is a linking relation, in line with @xcite . thus, the fact "paris is the capital of france" is represented by ⟨p aris, capital of, f rance⟩ where "paris" corresponds to s, "capital of" corresponds to r, and "france" corresponds to o. we then prompt a model m using the original natural language sentence with o masked out. to assess if m has correctly encoded an association, we calculate p m (o | s, r). prior knowledge assessments employed a rank-based reward @xcite where a model is thought to understand the association if o has a high chance of occurring as the next token (relative to all other options). however, this practice has pitfalls such as an inability to parse unsatisfactory outcomes for questions with numerous correct answers and a lack of insight into the llm's confidence in its response. to address these issues, our work uses a variant of the contrastive knowledge assessment (cka) from prior work @xcite . erroneous "counterfactuals" ⟨s, r, o ′ ⟩ like ⟨p aris, capital of, italy ′ ⟩ are used to assess a model m 's understanding of ⟨s, r, o⟩. simply put, if m truly knows the fact, p m (o | s, r) should be larger than p m (o ′ | s, r) @xcite . formally, cka measures whether m correctly knows a fact ⟨s, r, o⟩ via calculating: when cka m (s, r, o) > 1, the model is said to understand the factual association. this approach alleviates the issues that arise from ranking a model's vocabulary-wide token probabilities at inference; using counterfactuals elicits connections across different languages and contexts which forces the model to demonstrate generalized understanding of a given concept. furthermore, examining the contrast allows us to quantify the confidence level with more nuance. our work builds off @xcite by applying cka to a multilingual dataset for the first time. to carry out the test by language, we solicited cloze completions for each of the associations contained in the dataset. the percentage of factcompletions that m recalls correctly is calculated by tallying up the number of completions where cka m (s, r, o) > 1 and dividing by the total number of completions. we accommodated different tokenizers by removing special tokens from text generation and ensuring that the completion probing corresponded to the first token to the right of cloze. additionally, all evaluated models are fully open-source. 2 while we would have 2 llama weights were accessed with meta's permission liked to test proprietary llms such as gpt-4 (openai, 2023), these models don't currently provide vocabulary token probabilities at inference, a prerequisite for cka (see assessing open vs. proprietary llms for details). ## dataset the dataset includes 303k knowledge statements in 20 languages. 3 each row includes: dataset_id (primary key), stem, true, f alse, relation, subject, and object. in total, the dataset contains 31 unique relation categories, 76,036 unique subjects, 18,837 unique objects, 18,503 unique trues, and 88,224 unique falses. masked true/false objects consistently appear on the right-hand side of the statement to support masked and causal llms. the translated subset for each language contains different amounts of statements due to varying syntactic capacities to support this requirement. on average, a given fact appears in 12 of the 20 languages tested. to construct the dataset, we first merged two english-language datasets from @xcite and @xcite that share common lineage in the t-rex wikidata @xcite project. we then improved the dataset by filtering out inaccuracies and grammatical errors, as well as de-duplicating the ⟨s, r, o⟩ triplets, as detailed by dataset preprocessing in the appendix. after preprocessing, the dataset contained 26,254 knowledge statements in english. we then used the google translate api to translate the data into 19 target languages: bg, ca, cs, da, de, en, es, f r, hr, hu, it, nl, pl, pt, ro, ru, sl, sr, sv, and uk (iso 639-1 codes). our translation approach mirrors prior multilingual studies such as the programmatic translation of mmlu @xcite prompts when analyzing gpt-4. additionally, work from @xcite shows minimal practical differences when using machine versus manually translated cloze statements. ## future work there are many directions left to pursue in this domain. model weight editing in a multilingual setting presents a novel next step for our project since our data finds its roots in two projects @xcite ) that explore how to remedy inaccuracies located in llms. also, applying the test to future open-source models will fortify this work's impact and relevance for future researchers (see testing new models in the appendix for details). we can also add languages that use neither cyrillic nor latin scripts; we are working with native hindi and japanese speakers to create cloze statements in these languages. there is also work to be done regarding the variable difficulty of a given fact based on the availability of training data in that language; the values from our wikipedia analysis could be used as prior probabilities in a future iteration of cka. additionally, we could analyze more facets of training corpora metadata. perhaps it's possible to causally connect a model erring on a particular fact to artifacts in its training data rather than the measured, associative approach we adopt. current work @xcite affords helpful scaffolding for this endeavor. ## conclusion here, we present a multilingual contrastive knowledge assessment of encyclopedic facts. our original evaluation benchmarks 5 foundation models in a multilingual test and two dozen in an englishonly test. meta's llama demonstrated superior performance in both settings. accompanying analyses reveal that llama struggles to operate in non-english languages, particularly in cyrillic script, suggesting an absence of robust crosslingual knowledge transfer. these findings vouch for the utility of high-quality, multilingual datasets for training the next-generation of foundation models. our hope is that this project motivates future interrogations of foundation model data sources and provides a roadmap for others to conduct transparent evaluations. by doing so, llms can be better equipped for broad application across diverse linguistic contexts.