DLAMA : A Framework for Curating Culturally Diverse Facts for Probing the Knowledge of Pretrained Language Models (2023)

Abstract: A few benchmarking datasets have been released to evaluate the factual knowledge of pretrained language models. These benchmarks (e.g., LAMA, and ParaRel) are mainly developed in English and later are translated to form new multilingual versions (e.g., mLAMA, and mParaRel). Results on these multilingual benchmarks suggest that using English prompts to recall the facts from multilingual models usually yields significantly better and more consistent performance than using non-English prompts. Our analysis shows that mLAMA is biased toward facts from Western countries, which might affect the fairness of probing models. We propose a new framework for curating factual triples from Wikidata that are culturally diverse. A new benchmark DLAMA-v1 is built of factual triples from three pairs of contrasting cultures having a total of 78,259 triples from 20 relation predicates. The three pairs comprise facts representing the (Arab and Western), (Asian and Western), and (South American and Western) countries respectively. Having a more balanced benchmark (DLAMA-v1) supports that mBERT performs better on Western facts than non-Western ones, while monolingual Arabic, English, and Korean models tend to perform better on their culturally proximate facts. Moreover, both monolingual and multilingual models tend to make a prediction that is culturally or geographically relevant to the correct label, even if the prediction is wrong.

## introduction transfer learning paradigms such as fine-tuning, few-shot learning, and zero-shot learning rely on pretrained language models (plms), that require having large compilations of raw data @xcite @xcite . these plms showed some ability to model different linguistic phenomena @xcite in addition to memorizing facts related to real-world knowledge. while there is a drive to have multilingual models, english is still the language that is better supported due to the abundance of large english raw corpora, diverse datasets, and benchmarks. moreover, monolingual non-english plms are still being pretrained for other high-resource languages. as a way to probe the non-english and multilingual plms, researchers tend to translate english benchmarks into other languages, which might degrade the quality of the samples especially if the translation is performed automatically. while translating english benchmarks saves the time and money needed to build new language-specific benchmarks, it might introduce unintended biases or artifacts into the benchmarks. lama @xcite and pararel @xcite are two benchmarks developed to quantify the factual knowledge of the english plms. they used a setup in which a language model is said to know a specific fact if it can predict the right object for a prompt in a fill-the-gap setup (e.g., for the prompt "the capital of england is [mask]", the model needs to fill the masked gap with "london"). multilingual versions of these benchmarks namely: mlama @xcite , and mpararel @xcite were released to evaluate the performance of multilingual plms by translating lama and pararel into 53 and 46 languages respectively. the subjects and objects of the triples within these benchmarks were translated using their multilingual labels on wikidata, while the templates were automatically translated from the english ones used in the original benchmarks. these templates transform triples into textual natural language prompts for probing the models. x-factr is another benchmark sharing the same setup, and is built for 23 different languages @xcite . all three benchmarks sample factual triples in the form of (subject, relation predicate, object) from t-rex, a dump of wikidata triples aligned to abstracts extracted from the english wikipedia @xcite . the way t-rex is constructed might make it more representative of the facts related to western cultures, which might introduce an unnoticed bias to the benchmarks based on it. we hypothesize that having a fair representation of the different cultures within a benchmark is vital for fairly probing models pretrained for multiple languages. the main contributions of our paper can be summarized as follows: similarly, jiang et al. (2020) created x-factr by sampling relation triples from t-rex for 46 different wikidata predicates. the multilingual wikidata labels were used to translate the subjects and objects of the triples. they compared multiple decoding methods. moreover, they employed different templates to generate prompts having the correct number/gender agreement with the subjects of the triples. english prompts still outperformed prompts written in 22 other languages. pararel and its multilingual version mpararel are benchmarks created by sampling triples from t-rex for 38 relation predicates @xcite . their aim is to measure the consistency of the model in making the same prediction for different paraphrases of the same template. results on both benchmarks showed that the multilingual mbert and xlm-r models are less consistent than the monolingual english bert model, especially when these multilingual models are prompted with non-english inputs. from a model diagnostics perspective, @xcite found that english plms might be biased to making specific predictions based on a predicate's template irrespective of the subjects used to populate this template. thereafter, @xcite designed a causal framework for modeling multiple co-occurrence statistics that might cause english plms to achieve high scores on some of lama's predicates. we focus on why a non-english plm might fail to recall facts and hypothesize the following possible reasons: while the first two factors are studied in the literature, we believe that the third factor is a major quality issue that previous work has overlooked. randomly sampling the triples from t-rex might introduce a representation bias toward western cultures, since only facts aligned to english wikipedia abstracts are considered. we investigate the presence of such bias ( §3). moreover, we empirically demonstrate how better model diagnostics can be performed when the benchmark is formed using two diverse and contrasting sets of facts ( §5). ## cultural bias in mlama probing plms using prompts is an analysis tool attempting to understand how they behave. a biased probing benchmark might be deceiving, as both a good-performing model and a model sharing the same bias found in the benchmark would achieve good performance. in this section, we investigate if the facts within mlama might be biased toward western cultures, which can affect the reliability of the performance scores achieved by plms when probed using mlama. ## building dlama our methodology aims at building a culturally diverse benchmark, which would allow for a fairer estimation of a model's capability of memorizing facts. within dlama, query parameters form un-derlying sparql queries that are used to retrieve wikidata triples as demonstrated in figure 2 . to operationalize the concept of cultures, we use countries as a proxy for the cultures of interest. for instance, countries that are members of the arab league are considered representatives of arab cultures. conversely, western countries mentioned in §3.1 represent western cultures. furthermore, china, indonesia, japan, malaysia, mongolia, myanmar, north korea, philippines, singapore, south korea, taiwan, thailand, and vietnam are 13 countries from east asia, and southeast asia foot_5 representing asian cultures, while argentina, bolivia, brazil, chile, colombia, ecuador, guyana, paraguay, peru, suriname, uruguay, venezuela represent south american cultures. for predicates in which the subject is a person, we add a filter to the sparql query which limits the country of citizenship of the person to a specific set of countries (i.e., a specific culture). for predicates in which the subject is a place, we limit the values of the places to those located in a country within the predefined set of countries related to the target culture. we implemented a python interface to simplify the process of querying wikidata triples. currently, 20 relation predicates are supported. the userfriendly interface allows the addition of new relation predicates and filters, which we hope would encourage contributions to dlama. query parameters -subject: person -object: country -predicate: p27 (country of citizenship) -countries of interest: usa -languages of labels: english sparql query select ?person ?country ?subject_article_en where { values ?country {wd:q30} . # country is usa ?person wdt:p27 ?country . optional {?subject_article_en schema:about ?person . ?subject_article_en schema:inlanguage "en" . ?subject_article_en schema:@xmath0://en.wikipedia.org/ > .} . } (s2b) sort by article size (s2a) query articles' sizes (s1a) form query (s3) query all valid objects (s4) query labels input output sub uri obj uri article url q81324 q30 bret_hart q81328 q30 harrison_ford q65645 q30 matthias_pintscher ... ... ... (s1b) query wikidata sub uri obj uri article url article size q81324 q30 bret_hart 201353 q81328 q30 harrison_ford 81422 q65645 q30 matthias_pintscher 6923 ... ... ... ... sub label obj label boris johnson united states of america united kingdom donald trump united states of america tom brady united states of america ... ... sub uri obj uri article url article size q180589 q30 q145 boris_johnson 433128 q22686 q30 donal_trump 417652 q313381 q30 tom_brady 408608 ... ... ... ... sub uri obj uri article url article size q180589 q30 boris_johnson 433128 q22686 q30 donal_trump 417652 q313381 q30 tom_brady 408608 ... ... ... ... ## conclusion previous work suggested that english prompts are more capable of recalling facts from multilingual pretrained language models. we show that the facts within the underlying probing benchmark (mlama) are skewed toward western countries, which makes them more relevant to english. hence, we propose a new framework (dlama) that permits the curation of culturally diverse facts directly from wikidata. three new sets of facts are released as part of the dlama-v1 benchmark containing factual triples representing 20 relation predicates comprising facts from (arab-western), (asian-western), and (south american-western) countries, with a more balanced representation between the countries within each pair. the results of probing plms on the dlama-v1 support that mbert has a better performance recalling western facts than non-western ones irrespective of the prompt's language. monolingual arabic and korean models on the other hand perform better on culturally proximate facts. we believe the probing results are more trustable and fairer when the underlying benchmark is less skewed toward specific countries, languages, or cultures. moreover, we find that even when the model's prediction does not match any of the correct labels, the model might be making an educated guess relevant to the culture of the underlying facts. this finding augments previous experiments which showed that models tend to have a language bias, by which a model tends to overgenerate a specific prediction for each prompting language irrespective of the triple's subject used to fill in the prompt. finally, our framework is opensourced for the community to contribute new pairs to the dlama benchmark in the future.