Adapters for Enhanced Modeling of Multilingual Knowledge and Text (2022)

Abstract: Large language models appear to learn facts from the large text corpora they are trained on. Such facts are encoded implicitly within their many parameters, making it difficult to verify or manipulate what knowledge has been learned. Language models have recently been extended to multilingual language models (MLLMs), enabling knowledge to be learned across hundreds of languages. Meanwhile, knowledge graphs contain facts in an explicit triple format, which require careful and costly curation and are only available in a few high-resource languages, restricting their research and application. To address these issues, we propose to enhance MLLMs with knowledge from multilingual knowledge graphs (MLKGs) so as to tackle language and knowledge graph tasks across many languages, including low-resource ones. Specifically, we introducea lightweight adapter set to enhance MLLMs with cross-lingual entity alignment and facts from MLKGs for many languages. Experiments on common benchmarks show that such enhancement benefits both MLLMs and MLKGs, achieving: (1) comparable or improved performance for knowledge graph completion and entity alignment relative to baselines, especially for low-resource languages (for which knowledge graphs are unavailable); and (2) improved MLLM performance on language understanding tasks that require multilingual factual knowledge; all while maintaining performance on other general language tasks.

## introduction knowledge graphs serve as a source of explicit factual information for various nlp tasks. however, language models @xcite , which capture implicit knowledge from vast text corpora, are already being used in knowledgeintensive tasks. recently, language models have 1 our code, models, and data (e.g., integration corpus and extended datasets) are available at https://github.com/yifanh/multilingual_space . been successfully extended to multilingual language models (mllms) that integrate information sourced across hundreds of languages @xcite @xcite . however, as with most neural networks, the information is encoded in a diffused and opaque manner that is difficult to interpret, verify or utilize @xcite . meanwhile, multilingual knowledge graphs (mlkgs) require careful curation of explicit facts and annotation of entities that occur in languages (cross-lingual entity alignment), making knowledge graphs expensive and time-consuming to extend to new languages, restricting knowledge graph research to a few high-resource languages. further, open-source mlkgs such as wordnet @xcite and wikidata @xcite suffer from incompleteness as many true facts (or triples) and entity alignments are missing @xcite . in this work, we propose to overcome the above limitations of each knowledge source by integrating mlkgs into mllms (as shown in figure 1 ), to enable (i) the transfer of mlkg knowledge from high-resource languages to low-resource languages; and (ii) explicit knowledge of mlkgs to supplement mllms for knowledge-intensive language tasks, one of the key challenges in mllms @xcite . while this idea seems intuitive, there is no easy way to incorporate the explicit knowledge of mlkgs into the parametrically stored information of mllms. existing knowledge integration methods utilize language models and knowledge graphs in two ways: (1) training knowledge graph embeddings individually and combining the embeddings corresponding to linked entities in sentences with the language model representations (e.g., know-bert @xcite and ernie @xcite ); or (2) absorbing the knowledge in knowledge graphs into the language model's parameters via joint training (e.g., k-bert @xcite and k-adapter @xcite ). the first method requires embedding knowledge graph entities and accurately extracting entities in sentences across hundreds of languages, which is highly challenging. the second method typically suffers from the curse of multilinguality @xcite @xcite and catastrophic forgetting @xcite due to limited model capacity. most importantly, both methods integrate knowledge implicitly such that it is difficult to access and extend to low-resource languages @xcite . furthermore, both methods require large sets of aligned sentences and knowledge triples, which is costly to gather and accurately annotate across hundreds of languages. to address above issues, we first collect and clean multilingual data from wikidata foot_0 and wikipedia foot_1 for the enhancement, where rich factual knowledge and cross-lingual alignments are available. then, we propose to enhance mllms with the mlkg information by using a set of adapters @xcite , which are lightweight, collectively having only around 0.5% extra parameters than the mllm. each adapter integrates information from either mlkg triples (i.e. facts) or cross-lingual entity alignments, and is trained on either phrase or sentence level data. each of the resulting four adapters (ep/tp/es/ts) is trained individually to learn information sup-plemental to that already learned by the mllm. adapter outputs are combined by a fusion mechanism @xcite . training objectives are similar to those for mlkg embedding @xcite instead of mask language modeling, which are more efficient with large corpus. we conduct experiments on various downstream tasks to demonstrate the effectiveness of our approach. for mlkg tasks, following the data collection methods of two existing benchmarks @xcite , we extended them from 2-5 languages to 22 languages, including two rare languages. 4 results show that our method obtains comparable performance to existing state-of-theart baselines on the knowledge graph completion benchmark, and significantly better performance on the entity alignment benchmark. more importantly, we can perform these knowledge graph tasks in lowresource languages for which no knowledge graph exists, and achieve comparable results to the highresource languages. improvements over baseline mllms are significant. the results demonstrate that our proposed method integrates the explicit knowledge from mlkgs into mllms that can be used across many languages. our method also improves existing mllms noticeably on knowledgeintensive language tasks, such as cross-lingual relation classification, whilst maintaining performance on general language tasks such as named entity recognition (ner) and question answering (qa). ## multilingual knowledge integration in this paper, we fuse knowledge from a mlkg into a mllm. following previous works @xcite , we make use of an entity tagged corpus of text (called a knowledge integration corpus) for knowledge integration. we formally introduce these concepts below. ## adapters and adapter fusion in this section, we first describe how we incorporate adapters into language models and how they can be used to enhance them with different sources of knowledge from knowledge graphs. adapter. adapters have become a popular choice for parameter-efficient finetuning of language models on downstream tasks @xcite due to their flexibility, effectiveness, low cost and scalability @xcite . adapters are new modules that are added between layers of language models foot_3 , the parameters of which are updated only during finetuning while the language model parameters are frozen. an adapter is a bottleneck layer composed of two feed-forward layers with one non-linear activation function. for h m , the hidden representation of token t l i at layer m, the adapter acts as here, w down and w up are weight matrices, which map the hidden representations to the lowdimensional space and then map them back. b down and b up are bias parameters, and σ is a nonlinear activation function. adapter fusion. we follow the architecture of @xcite , but instead of using adapters for finetuning, we use them to enhance mllms with knowledge. our approach is similar to @xcite , but our adapters supplement and augment the existing implicit knowledge of mllms (into the explicit geometric properties of hidden representations), and our approach is more lightweight, with only c.0.5% additional parameters (@xmath0. as shown in figure 2 (left), still considering the m-th layer, the output representations of the feedforward layer (denoted h m as in eq. 1) are input to the adapters. a fusion layer aggregates all adapter outputs a n (h m ) (@xmath1...n } indexes each adapter) and the un-adapted representations with a multiplicative attention mechanism: here, a 0 (•) is the identity function; q m , k m , v m are parameters in the multiplicative attention mechanism; and ⊗ is the hadamard product. the additional knowledge to be learned by the adapters comes from knowledge triples and entity alignments, each provided in both phrase and sentence format (hence @xmath2. as shown in figure 2 (center), for a given entity in two languages l and l ′ , adapter-ep. learns to align the two (multilingual) representations of e l i and e l ′ i , e.g., zurich is aligned with zurigo. adapter-tp. learns knowledge triples, e.g., predicting switzerland given entity and relation (zurich, is located in,). besides these non-contextualized settings, entities within context can be considered also (mllm corpus). thus, adapter-es. and adapter-ts. have the similar objectives but use contextualized representations from input sentences. ## knowledgeable adapters next, we design objectives with corresponding knowledge integration datasets to train a set of adapters. similar to mlkg embedding @xcite , we aim to encode knowledge into the geometric properties of the adapted mllm representations, i.e., the mllm and adapters collectively act as an mlkg embedding model. specifically, we use cosine distance within the contrastive learning loss of @xcite : adapter-ep. we use wikidata @xcite to enhance mllms with the knowledge of cross-lingual entity alignments. inspired by the idea that languages are aligned implicitly in a universal space in mllms @xcite , we train the aligned entities to have closer representations. denoting the mllm with this adapter as lm(•), the objective used to train ep is: adapter-tp. we train this adapter using the knowledge triples in wikidata. inspired by previous knowledge graph embedding algorithms (e.g. @xcite , for a given fact triple, we train the (adapted) object entity embedding to be close to the (adapted) joint embedding of the subject entity and relation. the objective used to train tp is quite different from existing mask language modeling-based ones: ## experiments this section first introduces the general experimental settings ( §5.1). we then show that our adapter set can enhance mllms with the knowledge of mlkgs and, in particular, that the enhanced mllms generalize well to perform mlkgrelated tasks in low-resource languages ( §5.2). we also show that enhancing mllms with mlkgs improves their performance on knowledge-intensive language tasks ( §5.3). we compare our approach with the only existing mlkg integration work ( §5.4). finally, we present an ablation study of the adapter set to demonstrate the effectiveness of each adapter ( §5.5). ## conclusion in this paper we propose an approach to enhance mllms with mlkgs using a set of knowledge adapters, where explicit knowledge from mlkgs is integrated into the implicit knowledge learned by mllms. in experiments, we show that enhanced mllms can conduct mlkg-related tasks and achieve better performance on knowledge-intensive tasks, especially on low-resource languages where knowledge graphs are not available.