Parameter-Efficient Domain Knowledge Integration from Multiple Sources for Biomedical Pre-trained Language Models (2021)

Abstract: Domain-specific pre-trained language models (PLMs) have achieved great success over various downstream tasks in different domains. However, existing domain-specific PLMs mostly rely on self-supervised learning over large amounts of domain text, without explicitly integrating domain-specific knowledge, which can be essential in many domains. Moreover, in knowledge-sensitive areas such as the biomedical domain, knowledge is stored in multiple sources and formats, and existing biomedical PLMs either neglect them or utilize them in a limited manner. In this work, we introduce an architecture to integrate domain knowledge from diverse sources into PLMs in a parameter-efficient way. More specifically, we propose to encode domain knowledge via adapters , which are small bottleneck feed-forward networks inserted between intermediate transformer layers in PLMs. These knowledge adapters are pre-trained for individual domain knowledge sources and integrated via an attention-based knowledge controller to enrich PLMs. Taking the biomedical domain as a case study, we explore three knowledge-specific adapters for PLMs based on the UMLS Metathesaurus graph, the Wikipedia articles for diseases, and the semantic grouping information for biomedical concepts. Extensive experiments on different biomedical NLP tasks and datasets demonstrate the benefits of the proposed architecture and the knowledge-specific adapters across multiple PLMs.

## introduction in the past few years, large pre-trained language models (plms) have demonstrated superior performance over various downstream tasks in natural language processing (nlp), such as bert @xcite , roberta @xcite , albert @xcite , @xcite , etc. these plms mainly depend on selfsupervised pre-training on large amounts of textual data, e.g., wikipedia, and can be conveniently applied to downstream tasks via fine-tuning. despite the great success of these general plms, their performance over domain-specific texts is relatively poor due to domain shifts @xcite . consequently, recent studies construct domain-specific plms through fine-tuning or pre-training from scratch over domain corpora, such as biobert @xcite , clinicalbert @xcite , scibert @xcite , etc. since these plms are mostly pre-trained on unstructured free texts, a common issue among the aforementioned general and domain-specific plms is their lack of specific structured knowledge, which results in their incompetence on knowledgedriven tasks @xcite . for instance, some studies point out plms are insufficient to well capture factual knowledge from text @xcite @xcite . to enrich plms with external knowledge, some efforts have been made recently @xcite @xcite @xcite . a common theme among these approaches is the incorporation of an auxiliary knowledge-driven training objective. for instance, kg-bert @xcite integrates world/factual knowledge from wikipedia via knowledge graph completion; kepler @xcite introduces a knowledge embedding objective and combine it with the language modeling objective for joint optimization. despite the improved performance of these knowledge-enriched plms over downstream tasks, there are three limitations. first, these approaches, either training from scratch or fine-tuning over off-the-shelf checkpoints, need to optimize the entire model, which is quite expensive. second, they mostly focus on single-source knowledge incorporation, e.g., an encyclopedia, and neglect knowledge from multiple sources. this limits the utilization of potential knowledge, especially for knowledge-sensitive ar-eas such as the biomedical domain where knowledge is stored in multiple sources and formats @xcite . third, most of existing knowledge integration approaches focus on general domain knowledge, while domain knowledge infusion for plms is underexplored. to address these limitations, we propose to perform knowledge integration for plms via adapters @xcite @xcite @xcite . basically, adapters are lightweight neural networks that are placed inside plms. when fine-tuning a plm, the original parameters of the plm are fixed and only the adapters are fine-tuned. this makes adapters a parameter-efficient alternative to full model finetuning. another benefit of adapters is their independent nature, where multiple adapters can be trained independently without interfering with each other. as such, we propose to enrich plms with adapters that are independently pre-trained for different sources of domain knowledge. in this paper, we propose an architecture that aims to integrate domain knowledge from multiple sources via knowledge-specific adapters to enrich plms. we take the biomedical domain as a case study, as it is a knowledge-sensitive area where domain-knowledge is essential for various nlp applications. specifically, we explore three knowledge-specific adapters for plms based on the umls metathesaurus graph, the wikipedia articles for diseases, and the semantic grouping information for biomedical concepts. we also incorporate an attention-based knowledge controller module that aims to adaptively adjust the activation levels of the adapters, which also brings some explainability as it shows the importance of the adapters for a task. the experimental results show that by equipping plms with domain knowledge from multiple sources via the proposed architecture, their overall performance gets consistently improved across tasks and datasets. moreover, the pre-trained adapters can be directly integrated with multiple plms, demonstrating transferability of the architecture. the contributions of this work can be summarized as follows: this study is essentially related to two lines of research: knowledge integration for plms and domain-specific plms (biomedical plms in particular). there has been a surge of research on knowledge injection for plms in recent years @xcite @xcite @xcite @xcite @xcite . these studies aim to integrate knowledge from an external knowledge source, e.g., wikipedia, into plms by augmenting the training objective with a knowledge-driven regularization. as mentioned above, these methods are limited in the sense that they mostly focus on single-source knowledge, and require full model training. k-adapter @xcite addresses some of these issues by introducing linguistic and factual adapters into roberta, but the adapters are treated equally in their work. also, general domain knowledge, such as factual knowledge @xcite @xcite , commonsense knowledge @xcite , and linguistic knowledge @xcite are prioritized in these studies, while domain knowledge is somewhat underexplored @xcite . biomedical nlp continues to be an active area of research in the past few years. there have been several biomedical plms proposed and have proven to be successful in various domain tasks @xcite @xcite . as variants of bert @xcite in the biomedical domain, these plms are mostly pre-trained on large amounts of domain-specific corpora, such as the pubmed texts @xcite and clinical notes @xcite , and do not explicitly incorporate domain knowledge in the pre-training stage. this work differs from the aforementioned studies in that we are the first to integrate biomedical domain-specific knowledge from multiple sources into plms via an adapter-based architecture. the knowledge integration process is flexible, efficient and transferable. ## diverse adapters for knowledge integration (daki) in this section, we introduce a mechanism, i.e., daki, that encodes domain knowledge from diverse sources into plms via knowledge-specific adapters. we first introduce the adapter module along with the overall architecture of daki, and then discuss the knowledge-specific adapters for the biomedical domain. in the end we explain the attention-based knowledge controller that is leveraged to adaptively integrate these adapters. ## experiments in this section, we evaluate the daki architecture over three knowledge-driven downstream tasks in biomedical nlp, where we aim to show the effectiveness of the knowledge integration method. we also investigate some desirable properties of the architecture. ## conclusion in this paper, we propose daki, an adapter-based architecture that adaptively integrates knowledge from multiple sources into pre-trained language models. we take the biomedical domain as a case study, and specifically explore three different sources of biomedical knowledge and integrate them with daki. the experimental results prove the effectiveness of the architecture, and also show that the architecture demonstrates parameterefficiency, transferability, and explainability to some degree. the objective of this work is not to update state-of-the-art results on the benchmarks, but to provide an alternative method of domain knowledge integration for plms, especially from multiple sources of knowledge. the views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of aro, odni, iarpa, the department of defense, or the u.s. government. the u.s. government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. this document does not contain technology or technical data controlled under either the u.s. international traffic in arms regulations or the u.s. export administration regulations.