Factual Consistency of Multilingual Pretrained Language Models (2022)

Abstract: Pretrained language models can be queried for factual knowledge, with potential applications in knowledge base acquisition and tasks that require inference. However, for that, we need to know how reliable this knowledge is, and recent work has shown that monolingual English language models lack consistency when predicting factual knowledge, that is, they fill-in-the-blank differently for paraphrases describing the same fact. In this paper, we extend the analysis of consistency to a multilingual setting. We introduce a resource, mParaRel, and investigate (i) whether multilingual language models such as mBERT and XLM-R are more consistent than their monolingual counterparts;and (ii) if such models are equally consistent across languages. We find that mBERT is as inconsistent as English BERT in English paraphrases, but that both mBERT and XLM-R exhibit a high degree of inconsistency in English and even more so for all the other 45 languages.

## introduction pretrained language models (plms) enable highquality sentence and document representations @xcite @xcite and encode world knowledge that can be useful for downstream tasks, e.g. closed-book qa @xcite , and commonsense reasoning @xcite , to name a few. recent work has used language models as knowledge bases @xcite @xcite and as the basis of neural databases @xcite . such usage of plms relies on the assumption that we can generally trust the world knowledge that is induced from these models. consistency is a core quality that we would like models to have when we use their stored factual knowledge. we want models to behave consistently on semantically equivalent inputs @xcite , and to be consistent in their believes @xcite . moreover we want them to be fair across languages or in other words to exhibit a consistent behaviour across languages @xcite . nonetheless, recent work on consistency in plms has shown that models are brittle in their predictions when faced to irrelevant changes in the input @xcite @xcite . these works only considered english plms, while @xcite studied the consistency of korean plms. there are, to the best of our knowledge, no resources available to measure the consistency of multilingual plms. ## probing consistency we use the same probing framework as defined by @xcite and refined by @xcite , and query plms with cloze-test statements created from subject-relation-object wiki-data triples @xcite . that is, we have a set of different relations {r}, and each r has a set of templates or patterns {t} and a set of subjectobject tuples {(s, o)}. each template t describes its corresponding relation r between the pairs (s, o). e.g. a relation r can be born-in, and two patterns could be {t 1 ="[x] was born in [y]", t 2 ="[x] is originally from [y]"} (where [x] is the subject and [y] the object to be replaced). then the corresponding subject-object tuples {(s, o)} are used to query and evaluate the model by replacing the subject and masking the object. we study the consistency of a plm by querying it with cloze-test paraphrases and measuring how many of the predictions of the paraphrases are the same (details in §4). ## mpararel we used the paraphrases in the pararel dataset @xcite , which has 38 relations in total and an average of 8.6 english templates per relation. we translated these using the procedure below, obtaining paraphrases for 46 languages. translations we relied on five different machine translation models: google translate foot_1 , microsoft translator foot_2 , a pretrained mbart model that translates between 50 languages @xcite , a pretrained mixture of transformers that translates between 100 languages @xcite , and opus-mt @xcite . we fed models with templates, e.g.,"[x] died in [y]" foot_3 , automatically checking if the translation contained [x] and @xcite . we considered as valid: (1) translated paraphrases that were agreed upon by two or more different models, and (2) the translations from the microsoft translator, as they were found to be of good quality in several languages as per manual inspection by native speakers. so for languages that microsoft supports, we will have a template t from the microsoft translator, as well as any other translation agreed upon by two or more other translators foot_4 . finally, we also include the templates in the mlama dataset @xcite . translations of subject-object entities were obtained from wikidata, using the entity identifiers. we kept only the languages that (i) covered at least 60% of the manually reviewed languages are underlined. the order is given by the consistency results (see figure 2 ). ## experiments we ran experiments with mbert @xcite , a multilingual bert model of 110m parameters trained on 104 languages using wikipidea, and xlm-roberta @xcite , a multilingual roberta model of 560m parameters trained on 100 languages using 2.5tb of common-crawl data. ## results and discussion ## conclusion in this work, we measured the consistency of multilingual pretrained language models when queried to extract factual knowledge. we constructed a high-quality multilingual dataset containing 46 different languages, to assess the consistency of models predictions in the face of language variability. finally, we experimented with mbert and xlm-r and concluded that their consistency is poor in english, but even worse in other languages.