G eo MLAMA : Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models (2022)

Abstract: Recent work has shown that Pre-trained Language Models (PLMs) store the relational knowledge learned from data and utilize it for performing downstream tasks. However, commonsense knowledge across different regions may vary. For instance, the color of bridal dress is white in American weddings whereas it is red in Chinese weddings. In this paper, we introduce a benchmark dataset, Geo-diverse Commonsense Multilingual Language Models Analysis (GeoMLAMA), for probing the diversity of the relational knowledge in multilingual PLMs. GeoMLAMA contains 3125 prompts in English, Chinese, Hindi, Persian, and Swahili, with a wide coverage of concepts shared by people from American, Chinese, Indian, Iranian and Kenyan cultures. We benchmark 11 standard multilingual PLMs on GeoMLAMA. Interestingly, we find that 1) larger multilingual PLMs variants do not necessarily store geo-diverse concepts better than its smaller variant; 2) multilingual PLMs are not intrinsically biased towards knowledge from the Western countries (the United States); 3) the native language of a country may not be the best language to probe its knowledge and 4) a language may better probe knowledge about a non-native country than its native country.

## introduction pre-trained language models (plms) @xcite @xcite are increasingly used in various natural language processing (nlp) applications. pre-trained on large-scale text corpora, they are shown to store relational knowledge @xcite @xcite , e.g., commonsense knowledge @xcite @xcite . they have been used to construct knowledge bases while requiring limited human effort for rule crefor each concept (e.g., color of wedding dress), there are multiple masked multilingual prompts (english, hindi, swahili, etc.) with specified country information [x] querying geo-diverse knowledge about the concept. we test multilingual plms by examining the extent to which masked word predictions align with the gold answers in [mask] columns. in this paper, we concentrate on evaluating multilingual plms @xcite @xcite . studying geodiversity naturally involves multilinguality. people in different regions may speak different languages, and it is natural to assume that geo-specific knowledge is better represented in its native language. moreover, pre-trained on a collection of multilingual corpora, multilingual plms accumulate the knowledge from various languages. therefore, we posit that knowledge in multilingual plms is more diverse than that in models trained on a single language. centered around multilingual plms, we follow the original knowledge probing task language model analysis (lama) @xcite and introduce a new geo-diverse probing benchmark geomlama. as shown in figure 1 , given a masked geo-diverse prompt with a particular country name @xcite , such as "in traditional [x] weddings, the color of wedding dress is usually @xcite .", and a corresponding candidate answer list, {"red", "white", "black", "blue", ...}, multilingual plms are required to predict the masked word @xcite from the candidate list. the characteristics of geomlama are summarized as follows. 1) diverse answers across countries: each prompt is designed based on geodiverse concept (e.g., color of traditional wedding dress in figure 1 ) and gold answers for masked word are different across countries. 2) broad coverage of geo-diverse concepts: geomlama encompasses comprehensive geo-diverse topics including habits and personal choices, cultures and customs, policies and regulations, and geography. 3) coverage of multiple countries and languages: ge-omlama involves knowledge about the united states, china, india, iran, and kenya, and is constructed by the native languages of the five countries, english, chinese, hindi, persian, and swahili. overall, there are 3,125 prompts in our benchmark. we perform in-depth probing analysis on 11 multilingual plms, including mbert @xcite , xlm @xcite , xlm-r @xcite , mt5 @xcite , and xglm @xcite . in general, we observe that multilingual plms significantly outperform random guess, suggesting that multilingual plms are capable of storing geo-diverse commonsense to some extent. we then conduct fine-grained investigation across three dimensions. we first study the correlation between model performance and model size. contrary to our intuition, we notice that the largest models do not necessarily have the best performance on our benchmark. we further study the best language to probe the knowl-edge about a particular country. surprisingly, we find that the best language is not the native language of the given country (e.g., english is not the best language to probe knowledge about the us). we also explore the knowledge that can be most accurately probed by a particular language. similarly, we find that the most accurately probed knowledge is not the one about indigenous country of the language (e.g., the country for which chinese prompts provide the most accurate predictions is not always china). lastly, we find evidence of reporting bias that might explain such observations. ## geomlama benchmark construction to build a geo-diverse commonsense probing benchmark, we recruit annotators from five different countries, the united states, china, india, iran, and kenya to participate in annotation. the annotation process is separated into four stages. 1) we first ask the annotators to list geo-diverse concepts. ## conclusions we propose a knowledge probing benchmark, ge-omlama, to evaluate the extent of multilingual plms to store geo-diverse commonsense. results show that multilingual plms can achieve significantly higher performance than random guess, suggesting that they are capable of storing geo-diverse knowledge. we also find that fed with prompts without any country cues, multilingual plms are not intrinsically biased towards knowledge about the united states. we further investigate the best language to probe the knowledge about a particular country, and the country best probed with prompts in a certain language. surprisingly, we notice that the best language is not the country's native language, and the best probed country is not the indigenous country of the language. we connect this to reporting bias issue in geo-diverse context: one country's commonsense is seldom recorded in the text by people living in that country as it is too trivial and not worth mentioning for them.