Adapting Multilingual LLM s to Low-Resource Languages with Knowledge Graphs via Adapters (2024)

Abstract: This paper explores the integration of graph knowledge from linguistic ontologies into multilingual Large Language Models (LLMs) using adapters to improve performance for low-resource languages (LRLs) in sentiment analysis (SA) and named entity recognition (NER). Building upon successful parameter-efficient fine-tuning techniques, such as K-ADAPTER and MAD-X, we propose a similar approach for incorporating knowledge from multilingual graphs, connecting concepts in various languages with each other through linguistic relationships, into multilingual LLMs for LRLs. Specifically, we focus on eight LRLs — Maltese, Bulgarian, Indonesian, Nepali, Javanese, Uyghur, Tibetan, and Sinhala — and employ language-specific adapters fine-tuned on data extracted from the language-specific section of ConceptNet, aiming to enable knowledge transfer across the languages covered by the knowledge graph. We compare various fine-tuning objectives, including standard Masked Language Modeling (MLM), MLM with full-word masking, and MLM with targeted masking, to analyze their effectiveness in learning and integrating the extracted graph data. Through empirical evaluation on language-specific tasks, we assess how structured graph knowledge affects the performance of multilingual LLMs for LRLs in SA and NER, providing insights into the potential benefits of adapting language models for low-resource scenarios.

## introduction in recent years, the advancement of multilingual large language models (llms) @xcite @xcite has revolutionized the field of natural language processing (nlp), enabling impressive performance across various languages. however, these models often struggle with low-resource languages (lrls), where limited data availability affects their effectiveness @xcite . to address this limitation, researchers have explored integrating external knowledge sources into multilingual llms to enhance their performance in both high-resource and low-resource contexts @xcite @xcite via adapters @xcite and full fine-tuning. adapters, introduced by @xcite , are small modules inserted between the layers of a model and trained while the model is kept frozen. previous work has used such adapters to integrate external knowledge into llms. for instance, @xcite demonstrated improvements in relation classification, entity typing, and question answering tasks by integrating graph knowledge from wikidata @xcite into roberta @xcite using adapters. similarly, @xcite enhanced bert @xcite with graph knowledge from conceptnet @xcite , achieving significant performance gains on tasks requiring common-sense knowledge. however, these efforts primarily focused on the english language. in contrast, @xcite addressed low-resource languages by integrating textual knowledge from wikipedia into xlm-r @xcite ) via language adapters. their approach demonstrated improvements over the baseline model for named entity recognition (ner) task. motivated by recent advancements in the integration of graph knowledge into language models, particularly for english, this paper investigates the incorporation of graph knowledge from linguistic ontologies, specifically conceptnet, into multilingual llms particularly for lrls. injecting such data might be beneficial due to the scarcity of training data for these languages and the additional semantic and multilingual information provided by knowledge graphs @xcite . our focus is on a subset of lrls, aiming to extend the success observed in graph knowledge integration to linguistically diverse and resource-scarce contexts. we work with maltese, bulgarian, indonesian, nepali, javanese, uyghur, tibetan, and sinhala, identified as low-resource according to @xcite . our primary objective is to evaluate whether injecting multilingual graph knowledge, connecting various languages through linguistic relationships, into pretrained multilingual llms through adapters improves performance for lrls. we train languagespecific adapters on conceptnet data using different objective functions, including standard masked language modeling (mlm) @xcite , mlm with full-word masking @xcite , and mlm with targeted masking, and evaluate the downstream performance of the adapted model on sentiment analysis (sa) and ner tasks. our work extends existing advancements by proposing an approach that utilizes adapters to integrate graph knowledge specifically for lrls, following a modular design similar to the one introduced by @xcite . our contributions include: figure 1: proposed method. one of the wiki or con-ceptnet language adapters is used during inference. the outputs then go to a task adapter, which is followed by a classification head. if fusion is specified, the fusion mechanism is activated. our study provides insights into the potential benefits of adapting multilingual llms for low resource scenarios, contributing to the ongoing exploration of multilingual language model adaptation and graph knowledge integration. the complete code for our experiments is publicly available on github foot_0 . ## related work our work extends recent advancements in integrating external graph knowledge into pre-trained llms @xcite and adapting pre-trained multilingual llms to specific languages @xcite . we use the methodologies proposed in these studies as a foundation for enhancing the performance of multilingual llms on downstream tasks, particularly for lrls. to this end, we provide the overview of these methods as well as other strategies of adapting multilingual llms to lrls @xcite @xcite . ## injecting external knowledge into llms for lrls this section describes our approach for enhancing multilingual llms for lrls by injecting external knowledge. we discuss the use of language adapters trained on conceptnet and wikipedia data, explore adapter fusion @xcite for combining knowledge sources, and describe task adapters for fine-tuning multilingual llms for specific tasks. our proposed method is illustrated in figure 1 . ## experiments in this section, we detail the experiments and data used for conducting the study. ## results and discussion the experimental results, summarized in ## conclusion in this study, we investigated the integration of structured graph knowledge into multilingual llms for lrls using language adapters and taskspecific adapters. we explored the use of con-ceptnet and wikipedia data for training language adapters, and we examined adapter fusion as a method to combine knowledge sources. additionally, we implemented task adapters for fine-tuning llms for specific downstream tasks such as sentiment analysis (sa) and named entity recognition (ner). our experiments revealed insights into the effectiveness of different model configurations in improving sa and ner tasks performance across lrls. we observed a positive effect of incorporating external graph and textual knowledge through language adapters for a number of languages, including bulgarian, indonesian, maltese, nepali, uyghur, tibetan, and sinhala, some of which did not possess extensive data for training both language adapters and task adapters. fusion of knowledge sources yielded improvements in less cases, suggesting the need for further refinement in this area. overall, our findings underscore the importance of parameter-efficient fine-tuning methods and the potential benefits of leveraging external knowledge for enhancing multilingual llms in low resource contexts. however, there are limitations to our approach, including the choice of objective functions and the need for tasks better suited to leverage external knowledge.