KITLM : Domain-Specific Knowledge I n T egration into Language Models for Question Answering (2023)

Abstract: Large language models (LLMs) have demon- strated remarkable performance in a wide range of natural language tasks. However, as these models continue to grow in size, they face sig- nificant challenges in terms of computational costs. Additionally, LLMs often lack efficient domain-specific understanding, which is par- ticularly crucial in specialized fields such as aviation and healthcare. To boost the domain- specific understanding, we propose, KITLM 1 , a novel knowledge base integration approach into language model through relevant informa- tion infusion. By integrating pertinent knowl- edge, not only the performance of the lan- guage model is greatly enhanced, but the model size requirement is also significantly reduced while achieving comparable performance. Our proposed knowledge-infused model surpasses the performance of both GPT-3.5-turbo and the state-of-the-art knowledge infusion method, SKILL, achieving over 1.5 times improvement in exact match scores on the MetaQA. KITLM showed a similar performance boost in the avi- ation domain with AeroQA. The drastic perfor- mance improvement of KITLM over the exist- ing methods can be attributed to the infusion of relevant knowledge while mitigating noise. In addition, we release two curated datasets to accelerate knowledge infusion research in specialized fields: a) AeroQA, a new bench- mark dataset designed for multi-hop question- answering within the aviation domain, and b) Aviation Corpus, a dataset constructed from unstructured text extracted from the National Transportation Safety Board reports. Our re- search contributes to advancing the field of domain-specific language understanding and showcases the potential of knowledge infusion techniques in improving the performance.

## introduction large pre-trained language models (plms) @xcite have succeeded remarkably in various nlp downstream tasks. their achievements can be attributed to two key factors: extensive pretraining on diverse text sources and the ability to fine-tune domain-specific data. plms undergo extensive pre-training on vast amounts of text data from various sources such as books, articles, and websites. this process allows them to develop a profound understanding of language and capture a comprehensive range of linguistic patterns and contextual information. furthermore, plms can be fine-tuned on domain-specific datasets, enabling them to specialize and adapt to a particular domain. this fine-tuning process refines the models' knowledge and performance, allowing them to excel in tasks specific to those domains. however, recent research has highlighted the efficacy of incorporating knowledge graphs into language models using diverse techniques @xcite @xcite . our paper shows that incorporating relevant structured knowledge from knowledge graphs can further enhance language model performance and domain-specific understanding. various studies have explored different methods for infusing knowledge into language models. one popular approach involves verbalizing triples in the knowledge base and continually pretrain the llm using a training criteria such as masked language modeling. however, this approach can be computationally demanding. other methods like qa-gnn @xcite and @xcite rely on knowledge graph embeddings @xcite to obtain the domain knowledge which requires additional training. the two critical factors in a knowledge infusion method are: i) the quality of infused knowledge, which allows for achieving strong empirical performance, and ii) the simplicity of the architecture. these underscore the need for a knowledge infusion technique that is computationally efficient while maintaining high quality and simplicity. our paper presents an innovative framework for integrating knowledge into language models like t5 @xcite through fine-tuning. the experimental results demonstrate that the checkpoints trained using the proposed approach on @xcite and wikimovies @xcite outperforms the t5 baselines, state-of-the-art skill @xcite and gpt-3.5-turbo on metaqa @xcite and our curated multihop qa dataset, aeroqa. instead of introducing additional parameters to pre-trained language models (plms) or modifying their architectures, the proposed framework employs a novel knowledge integration objective. this objective entails verbalizing the kg triples, extracting pertinent triples for each question-answer pair using colbertv2 @xcite , and incorporating them during both the training and testing phases of the language model. we conducted a comprehensive study to enhance our proposed framework, kitlm, by exploring the impact of different formats of external knowledge on a language model. we incorporated unstructured general corpora, domain-specific corpora, and structured knowledge (triples) into the t5 model for question-answering. to evaluate the effectiveness of these settings, we employed the skill approach @xcite and compared t5, t5 + unstructured text, t5 + kg triples, and t5 + unstructured text + kg triples on the aeroqa and metaqa datasets. our proposed approach, kitlm, outperformed all other settings, underscoring the importance of integrating relevant knowledge alongside llms while mitigating noise for enhancing question-answering capabilities. our contributions are: it contains 34k qa pairs, with 21k 1-hop pairs and the rest being 2-hop pairs. (b) aviation corpus, comprising 665,000 lines of clean english text from 4,000 ntsb foot_0 reports. it is specifically curated for continual pre-training to facilitate knowledge infusion tasks in language models. ## motivation aviation-related datasets are scarce and highly sought after, posing challenges for building question-answering (qa) systems capable of reasoning over knowledge graphs like aviationkg @xcite . to address this, we have developed a valuable multi-hop reasoning qa dataset derived from the national transportation safety board reports in the aviation domain. this dataset is valuable for the aviation industry and researchers, facilitating information retrieval and qa tasks. its creation aims to provide deeper insights into aircraft accidents and contribute to developing preventive measures to enhance aviation safety. large language models @xcite have demonstrated efficient performance across various downstream nlp tasks. however, the high computational requirements associated with llms have raised concerns. furthermore, llms are typically trained on generic datasets, so their suitability for domain-specific tasks is limited. our study provides evidence that computational resources can be conserved by employing smaller language models for specific tasks. additionally, we highlight the importance of integrating relevant knowledge in the lm to address the needs of domain-specific tasks. the background and related work for our paper "kitlm" can be found in the appendix. prevalent state-of-the-art models like bert @xcite , @xcite , and t5 @xcite have emerged as powerful tools for various tasks. these models are typically pre-trained on unstructured text data, allowing them to comprehend language within a contextual framework. however, knowledge about the real world is crucial to gain a comprehensive understanding of a statement. this world knowledge is frequently represented as triples within a knowledge graph. ## methodology this section details the following methodologies for knowledge integration: 1. our novel framework kitlm, designed for multi-hop question answering, depicted in figure 1 . ## dataset this section introduces aeroqa, a benchmark dataset specifically designed for questionanswering tasks in the aviation domain. additionally, an aviation-related text dataset called aviation corpus, similar to c4, is created. the rest of the datasets used for experimentation is explained in appendix. ## results and analysis in the t5-large model is used as the baseline, while the state-of-the-art knowledge infusion method, described in @xcite , is referred to as the skill pre-trained combined model. this model combines t5 with additional input, which is unstructured corpora denoted as x, and also incorporates triples as part of the knowledge infusion process. the t5 + x + kg demonstrates improved performance compared to t5, t5 + x, and t5 + kg for both metaqa and aeroqa datasets. in metaqa, the performance is adversely affected when using only non-verbalized kg-triples, leading to a decrease in scores compared to the c4 text experiment. although the t5 + kg-triples model exhibits a slight improvement in the aeroqa, it fails to surpass the t5 baseline due to catastrophic interference, despite its inherent domain-specific benefits. the t5 + kg-triples result underscores the importance of incorporating unstructured corpora for language models, as relying solely on triples can result in catastrophic forgetting. but this doesn't mean that the integration of triples is irrelevant, and infusing only text into the language model will help. we observed a decline in the performance for the tasks after utilization of only c4 text compared to the baseline. the rationale behind the ongoing use of text-only pre-training is explained in @xcite . it is proposed that repeatedly training on c4 text could potentially lead to a decrease in performance for a t5 model. although incorporating the aviation corpus in the aeroqa task significantly improves the score compared to the c4 text, this improvement can be primarily attributed to the domain similarity between the corpus and the task itself. to conclude, considering the individual inclusion of unstructured text and triples, the outcomes vary, with some cases showing a decline in performance while others exhibit slight improvements. however, when both text and triples are combined using the skill, the results demonstrate enhancements. it is important to acknowledge that explaining the performance of these approaches can be complex, and their effectiveness relies on the particular corpora employed, such as c4 or the aviation corpus. furthermore, the continual pre-training demands higher computational resources. to address the aforementioned limitations, we developed the kitlm approach, demonstrating the best performance across all tasks. the reason why kitlm performs better is as it selects the most relevant triples from the knowledge base after mitigating noise. the triples serve as the context for answering the question. kitlm is a selective approach that helps to remove confusing elements compared to existing methods, such as skill. this resulted in kitlm to achieve a significant improvement of 30% and 20% in the exact match score for the aeroqa 1-hop and metaqa 1-hop tasks, respectively. attributing to similar reasons, kitlm demonstrates better performance for the 2-hop and 3-hop qa. the reason kitlm surpasses other methods is due to its adeptness in handling multi-hop question-answering tasks, which demand advanced reasoning abilities for accurate answers. unlike previous infusion methods that relied on continual pre-training, which can result in catastrophic forgetting, kitlm addresses this challenge effectively by removing the pre-training procedure to infuse knowledge. another advantage of kitlm is its ability to mitigate the computational power requirements associated with knowledge infusion methods, which typically arise from pre-training. kitlm addresses this issue by selecting triples n times for n-hop qa, eliminating the need to train the model repeatedly. this approach not only reduces computational costs but also improves performance. the significance of the kitlm is amplified by its capability to adapt to diverse domains and tasks, eliminating the necessity for domain-specific corpora in the process of knowledge infusion. performance on gpt-3 and chatgpt. in our experiments on metaqa and aeroqa datasets, we observed that chatgpt, despite being a powerful model, encountered challenges in producing accurate results. in following the evaluation of chatgpt's performance with one-shot learning, we fine-tune the gpt-3 model for the qa tasks. however, the results obtained for aeroqa 1-hop and metaqa 1hop were relatively poor, with em scores of 24.3% and 18.73% respectively. also, the 2-hop and 3hop tasks performed worse than kitlm. our results using gpt-3 and chatgpt highlight the significant advantage of knowledge infusion for smaller models compared to large language models such as the gpt series. through the incorporation of knowledge infusion techniques, we achieved superior performance across various tasks, demonstrating the effectiveness of leveraging domainspecific knowledge to enhance the capabilities of smaller models. ## conclusion and future work our framework, kitlm, addresses the challenge of providing context for multi-hop question answering by leveraging a knowledge base to filter and select relevant information. our study underscores the ongoing importance of incorporating domain-specific knowledge and context, even with the availability of large language models like gpt-3 and chatgpt, to enhance their performance in specialized domains. we have also introduced the aeroqa dataset, designed specifically for multihop question-answering in the aviation domain, and the aviation corpus, which serves as a valuable resource for knowledge infusion tasks in language models. our future work aims to enhance nlp tasks like sentiment analysis and summarization by leveraging knowledge bases' reasoning capabilities. we'll address existing knowledge source limitations and explore ways to integrate domain-specific knowledge directly into language models, potentially replacing traditional knowledge bases.