A Unified Knowledge Graph Augmentation Service for Boosting Domain-specific NLP Tasks (2023)

Abstract: By focusing the pre-training process on domain-specific corpora, some domain-specific pre-trained language models (PLMs) have achieved state-of-the-art results. However, it is under-investigated to design a unified paradigm to inject domain knowledge in the PLM fine-tuning stage. We propose KnowledgeDA, a unified domain language model development service to enhance the task-specific training procedure with domain knowledge graphs. Given domain-specific task texts input, KnowledgeDA can automatically generate a domain-specific language model following three steps: (i) localize domain knowledge entities in texts via an embedding-similarity approach; (ii) generate augmented samples by retrieving replaceable domain entity pairs from two views of both knowledge graph and training data; (iii) select high-quality augmented samples for fine-tuning via confidence-based assessment. We implement a prototype of KnowledgeDA to learn language models for two domains, healthcare and software development. Experiments on domain-specific text classification and QA tasks verify the effectiveness and generalizability of KnowledgeDA.

## introduction although general nlp models such as @xcite have demonstrated great potential, they may not consistently perform well in domain-specific tasks like healthcare @xcite and programming @xcite . this is because most pre-trained language models are trained on general-domain corpora, e.g., openweb-text @xcite and c4 @xcite . however, the words or knowledge entities frequently used in a specific domain are typically different from those in a general domain. for instance, scientific texts use different words than general texts, with only a 42% overlap @xcite . consequently, general plms struggle to capture many important domain entities that rarely appear in general corpora. therefore, it is necessary to develop a suitable training mechanism for domain-specific nlp tasks. in general, two steps are needed for domainspecific nlp model development: (i) language model pretraining and (ii) task-specific model training @xcite . most existing studies focus on pretraining. in particular, to learn domainspecific word embeddings, they retrain plms with domain-specific corpora, including clinicalbert @xcite , biobert @xcite , scibert @xcite , etc. in contrast, how to improve the second step (i.e., task-specific training) is under-investigated. a common practice is directly fine-tuning the task-specific model with annotated data @xcite . however, it is difficult to obtain abundant annotated data for a domain-specific task, as labeling often requires domain experts' knowledge @xcite ; without sufficient data, direct fine-tuning may not lead to a satisfactory performance due to overfitting @xcite . some studies propose task-dependent methods to train task-specific models by introducing some types of domain knowledge @xcite , but they are hard to be generalized to other tasks @xcite . then, a research question appears: can we introduce domain knowledge to task-specific model training in a unified way? to answer the question, two main issues need to be addressed: (i) where to find a unified format of domain knowledge? (ii) how to improve the task-specific training of various domains' models in a unified way? on one hand, the domain knowledge graph (kg) is an effective and standardized knowledge base for a specific domain @xcite . kgs have been constructed for various domains such as cybersecurity @xcite , social-impact funding @xcite , and healthcare @xcite , which emphasizes the wide availability of domain kgs. hence, domain kg could be a feasible source for unified domain knowledge. on the other hand, data augmentation (da) is a data-space approach to enrich training data to avoid overfitting regardless of the task-specific model structure. they are often taskagnostic @xcite , i.e., not specified to any particular task. this property inspires us that it may be possible to design a unified da process to introduce domain knowledge to task-specific model training. however, current da methods in nlp are mostly proposed for general texts @xcite , and the performance on domain-specific tasks is limited @xcite . in general, domain-specific da is still an under-researched direction @xcite . to fill this research gap, by exploiting domain kgs, we propose knowledgeda, a novel and unified three-step procedure to perform domainspecific da: (i) domain knowledge localization to map phrases in the text to entities in the domain kg; (ii) domain knowledge augmentation to fully utilize the kg and the training data to achieve domain-specific augmentation; and (iii) augmentation quality assessment to single out highquality augmented data for fine-tuning the taskspecific model. specifically: (i) to the best of our knowledge, this is one of the pioneering efforts toward proposing a unified development process for domain-specific nlp models, especially focusing on task-specific model training. (ii) knowledgeda consists of three core steps, domain knowledge localization, domain knowledge augmentation, and augmentation quality assessment. we implement a prototype of knowledgeda, which can automatically learn domain-specific models given domainspecific texts, especially in healthcare domain. (iii) experiments are run on text classification and qa tasks (english and chinese) mainly in healthcare. results show that knowledgeda can obtain ∼ 4% improvement compared to direct fine-tuning, and significantly outperform existing da methods @xcite . the source codes are available 1 . ## conclusions in this paper, we present knowledgeda, a unified knowledge graph service to boost domain-specific nlp tasks. the intrinsic technical novelty is a three-step framework of task-specific data augmentation process based on domain kgs. the experiments on healthcare-related texts both in english and chinese verify the effectiveness and generality of knowledgeda. we also confirm that it can be flexible and effective to incorporate other generatorbased da methods on few-shot tasks. in the future, we can further investigate how to better combine knowledgeda and generator-based da methods and add kg quality inspection methods to avoid the negative impact of errors in kg.