Prix- LM : Pretraining for Multilingual Knowledge Base Construction (2022)

Abstract: Knowledge bases (KBs) contain plenty of structured world and commonsense knowledge. As such, they often complement distributional text-based information and facilitate various downstream tasks. Since their manual construction is resource- and time-intensive, recent efforts have tried leveraging large pretrained language models (PLMs) to generate additional monolingual knowledge facts for KBs. However, such methods have not been attempted for building and enriching multilingual KBs. Besides wider application, such multilingual KBs can provide richer combined knowledge than monolingual (e.g., English) KBs. Knowledge expressed in different languages may be complementary and unequally distributed: this implies that the knowledge available in high-resource languages can be transferred to low-resource ones. To achieve this, it is crucial to represent multilingual knowledge in a shared/unified space. To this end, we propose a unified representation model, Prix-LM, for multilingual KB construction and completion. We leverage two types of knowledge, monolingual triples and cross-lingual links, extracted from existing multilingual KBs, and tune a multilingual language encoder XLM-R via a causal language modeling objective. Prix-LM integrates useful multilingual and KB-based factual knowledge into a single model. Experiments on standard entity-related tasks, such as link prediction in multiple languages, cross-lingual entity linking and bilingual lexicon induction, demonstrate its effectiveness, with gains reported over strong task-specialised baselines.

## introduction multilingual knowledge bases (kbs), such as db-pedia @xcite , wikidata @xcite , and yago @xcite , provide structured knowledge expressed in multiple languages. those kbs are modeled as knowledge graphs (kgs) that possess two types of knowledge: monolingual triples which describe relations of entities, and cross-lingual links which match entities across languages. the knowledge stored in such kgs facilitates various downstream applications such as question answering @xcite @xcite , recommendation @xcite @xcite , and dialogue systems @xcite @xcite . manually constructing large-scale knowledge bases has been labor-intensive and expensive @xcite , leading to a surge of interest in automatic knowledge base construction @xcite . recent research @xcite @xcite proposes to generate structured knowledge using pretrained lan-guage models (plms; @xcite , where missing elements in kb facts (i.e., triples) can be completed (i.e., filled in) by the plm. while these methods arguably perform well for english, such automatic kb construction has not yet been tried for multilingual kbs -improving the knowledge in multilingual kbs would have a positive impact on applications in other languages beyond english. moreover, kbs in multiple languages may possess complementary knowledge, and knowledge bases in low-resource languages often suffer severely from missing entities and facts. this issue could be mitigated by propagating knowledge from multiple well-populated highresource languages' kbs (e.g., english and french kbs) to the kbs of low-resource languages, this way 'collectively' improving the content stored in the full multilingual kb. 1 however, training lms to capture structural knowledge independently for each language will fall short of utilizing complementary and transferable knowledge available in other languages. therefore, a unified representation model is required, which can capture, propagate and enrich knowledge in multilingual kbs. in this work, we thus propose to train a language model for constructing multilingual kbs. starting from xlm-r @xcite as our base model, we then pretrain it on the multilingual dbpedia, which stores both monolingual triples and cross-lingual links (see figure 1 ). we transform both types of knowledge into sequences of tokens and pretrain the language model with a causal lm objective on such transformed sequences. the monolingual triples infuse structured knowledge into the language model, while the cross-lingual links help align knowledge between different languages. this way, the proposed model prix-lm (pre-trained knowledge-incorporated cross-lingual language model) is capable of mapping knowledge of different languages into a unified/shared space. we evaluate our model on four different tasks essential for automatic kb construction, covering both high-resource and low-resource languages: link prediction, cross-lingual entity linking, bilingual lexicon induction, and prompt-based lm 1 this intuition is illustrated by the example in figure 1 . consider the prediction of facts (e.g., genre) about the oldest japanese novel the tale of genji. english dbpedia records its genre only as monogatari (story), whereas complementary knowledge can be propagated from the japanese kb, which provides finer-grained genre information, including love story, royal family related story, and monogatari. ## prix-lm we now describe prix-lm, first outlining the data structure and pretraining task, and then describing its pretraining procedure in full ( §2.1), and efficient inference approaches with prix-lm ( §2.2). pretraining task. we rely on multilingual dbpedia, but note that prix-lm is also applicable to other kbs. dbpedia contains two types of structured knowledge: monolingual knowledge triples, and cross-lingual links between entities. the monolingual triples represent (relational) facts expressed in a structured manner. each triple is denoted as {e 1 , r, e 2 }: the elements of a triple are identified as the subject entity e 1 , relation (or predicate) r, and object entity e 2 , respectively (see also figure 1 for examples). for instance, the fact "the capital of england is london" can be represented as {england, capital, london}. the cross-lingual links, denoted as {e a , e b }, represent the correspondence of 'meaning-identical' entities e a and e b in two different languages: e.g., the english entity london is mapped to londres in spanish. we treat both types of knowledge using the same input format {s, p, o}, where @xmath0, @xmath1, @xmath2, and @xmath3, @xmath4, @xmath5. the pretraining task is then generating o given s and p. this objective is consistent with the link prediction task and also benefits other entity-related downstream tasks, as empirically validated later. ## conclusion we have proposed prix-lm, a unified multilingual representation model that can capture, propagate and enrich knowledge in and from multilingual kbs. prix-lm is trained via a casual lm objective, utilizing monolingual knowledge triples and cross-lingual links. it embeds knowledge from the kb in different languages into a shared representation space, which benefits transferring complementary knowledge between languages. we have run comprehensive experiments on 4 tasks relevant to kb construction, and 17 diverse languages, with performance gains that demonstrate the effectiveness and robustness of prix-lm for automatic kb construction in multilingual setups. the code and the pretrained models will be available online at: https://github.com/luka-group/prix-lm .