Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triples (2022)

Abstract: Knowledge-enhanced language representation learning has shown promising results across various knowledge-intensive NLP tasks. However, prior methods are limited in efficient utilization of multilingual knowledge graph (KG) data for language model (LM) pretraining. They often train LMs with KGs in indirect ways, relying on extra entity/relation embeddings to facilitate knowledge injection. In this work, we explore methods to make better use of the multilingual annotation and language agnostic property of KG triples, and present novel knowledge based multilingual language models (KMLMs) trained directly on the knowledge triples. We first generate a large amount of multilingual synthetic sentences using the Wikidata KG triples. Then based on the intra- and inter-sentence structures of the generated data, we design pretraining tasks to enable the LMs to not only memorize the factual knowledge but also learn useful logical patterns. Our pretrained KMLMs demonstrate significant performance improvements on a wide range of knowledge-intensive cross-lingual tasks, including named entity recognition (NER), factual knowledge retrieval, relation classification, and a newly designed logical reasoning task.

## introduction pretrained language models (plms) such as bert @xcite and roberta @xcite have achieved superior performances on a wide range of nlp tasks. existing plms usually learn universal language representations from general-purpose large-scale corpora but do not concentrate on capturing world's factual knowledge. it has been shown that knowledge graphs (kgs), such as wikidata @xcite and freebase @xcite , can provide rich factual information for better language understanding. many studies have demonstrated the effectiveness of incorporating such factual knowledge into monolingual plms @xcite @xcite @xcite . following this, a few recent attempts have been made to enhance multilingual plms with wikipedia or kg triples @xcite @xcite . however, due to the structural difference between kg and texts, existing kg based pretraining often relies on extra relation/entity embeddings or additional kg encoders for knowledge enhancement. these extra embeddings/components may add significantly more parameters which in turn increase inference complexity, or cause inconsistency between pretrain and downstream tasks. for example, mluke @xcite has to enumerate all possible entity spans for ner to minimize the inconsistency caused by entity and entity position embeddings. other methods @xcite also require kg triples to be combined with relevant natural sentences as model input during training or inference. in this work, we propose kmlm, a novel knowledge-based multilingual language model pretrained on massive multilingual kg triples. unlike prior knowledge enhanced models @xcite @xcite , our model requires neither a separate encoder to encode entities/relations, nor heterogeneous information fusion to fuse multiple types of embeddings (e.g., entities from kgs and words from sentences). the key idea of our method is to convert the structured knowledge from kgs to sequential data which can be directly fed as input to the lm during pretraining. specifically, we generate three types of training data -the parallel knowledge data, the code-switched knowledge data and the reasoning-based data. the first two are obtained by generating parallel or code-switched sentences from triples of wikidata @xcite , a collaboratively edited multilingual kg. the reasoning-based data, containing rich logical patterns, is constructed by converting cycles from wikidata into word sequences in different languages. we then design pretraining tasks that are operated on the parallel/code-switched data to memorize the factual knowledge across languages, and on the reasoning-based data to learn the logical patterns. compared to existing knowledge-enhanced pretraining methods @xcite @xcite , kmlm has the following key advantages. (1) kmlm is explicitly trained to derive new knowledge through logical reasoning. therefore, in addition to memorizing knowledge facts, it also learns the logical patterns from the data. (2) kmlm does not require a separate encoder for kg encoding, and eliminates relation/entity embeddings, which enables kmlm to be trained on a larger set of entities and relations without adding extra parameters. the token embeddings are enhanced directly with knowledge related training data. (3) kmlm does not rely on any entity linker to link the text to the corresponding kg entities, as done in existing methods @xcite @xcite . this ensures kmlm to utilize more kg triples even if they are not linked to any text data, and avoids noise caused by incorrect links. (4) kmlm keeps the model structure of the multilingual plm without introducing any additional component during both training and inference stages. this makes the training much easier, and the trained model is directly applicable to downstream nlp tasks. we evaluate kmlm on a wide range of knowledge-intensive cross-lingual tasks, including ner, factual knowledge retrieval, relation classification, and logical reasoning which is a novel task designed by us to test the reasoning capability of the models. our kmlm achieves consistent and significant improvements on all knowledgeintensive tasks, meanwhile it does not sacrifice the performance on general nlp tasks. ## related work knowledge-enhanced language modeling aims to incorporate knowledge, concepts and relations into the plms @xcite @xcite , which proved to be beneficial to language understanding @xcite . the existing approaches mainly focus on monolingual plms, which can be roughly divided into two lines: implicit knowledge modeling and explicit knowledge injection. previous attempts on implicit knowledge modeling usually consist of entity-level masked language modeling @xcite , entity-based replacement prediction @xcite , knowledge embedding loss as regularization @xcite and universal knowledge-text prediction @xcite . in contrast to implicit knowledge modeling, the methods of explicit knowledge injection separately maintain a group of parameters for representing structural knowledge. such methods @xcite usually require a heterogeneous information fusion component to fuse multiple types of embeddings obtained from the text and kgs. zhang et al. (2019) and poerner et al. (2020) employ external entity linker to discover the entities in the text and perform feature interaction between the token embeddings and entity embeddings during the encoding phase of a transformer model. @xcite borrow the pre-computed knowledge embeddings as the supporting features of training an internal entity linker. @xcite insert an adapter component @xcite in each transformer layer to store the learned factual knowledge. extending knowledge based pretraining methods to the multilingual setting has received increasing interest recently. @xcite propose an auto-regressive model trained on knowledge triples for multilingual kg completion. @xcite ; @xcite attempt improving multilingual entity representation via wikipedia hyperlink prediction, however, their methods add a large amount of parameters due to the reliance on extra entity embeddings. for example, the mluke base @xcite model initialized with xlm-r base doubles the number of parameters (586m vs 270m). similar to us, @xcite al. (2022) also utilize kg for plm pretraining. they employ kg and wikipedia entity descriptions to inject knowledge into multilingual lm, but relation embeddings are also required to assist learning. moreover, the above methods only focus on memorizing the existing facts but ignore the reasoning over the unseen/implicit knowledge that id language label aliases q1420 english motor car auto, autocar, . . . spanish automóvil coche, carro, . . . hungarian autó gépkocsi, személyautó, . . . . . . . . . . . . ## framework in this section, we describe the proposed framework for knowledge based multilingual language model (kmlm) pretraining. we first describe the process to generate knowledge-intensive multilingual training data, followed by the pretraining tasks to train the language models to memorize factual knowledge and learn logical patterns from the generated data. ## experiments we first describe the pretraining details of our kmlms. then we verify its effectiveness on the knowledge-intensive tasks. finally, we examine its performance on general cross-lingual tasks. in all of the tasks except x-factr @xcite , the plms are fine-tuned on the english training set and then evaluated on the target language test sets. the evaluation results are averaged over 3 runs with different random seeds. x-factr does not require fine-tuning, so the plms are directly evaluated using the official code. the results of the baseline models are reproduced in the same environment. ## conclusions in this paper, we have presented a novel framework for knowledge-based multilingual language pretraining. our approach firstly creates a synthetic multilingual corpus from the existing kg and then tailor-makes two pretraining tasks, multilingual knowledge oriented pretraining and logical reasoning oriented pretraining. these multilingual pretraining tasks not only facilitate factual knowledge memorization but also boost the capability of implicit knowledge modeling. we evaluate the proposed framework on a series of knowledgeintensive cross-lingual tasks and the comparison results consistently demonstrate its effectiveness.