There is No Big Brother or Small Brother:Knowledge Infusion in Language Models for Link Prediction and Question Answering (2022)

Abstract: The integration of knowledge graphs with deep learning is thriving in improving the performance of various natural language processing (NLP) tasks. In this paper, we focus on knowledge-infused link prediction and question answering using language models, T5, and BLOOM across three domains:Aviation, Movie, and Web. In this context, we infuse knowledge in large and small language models and study their performance, and find the performance to be similar. For the link prediction task on the Aviation Knowledge Graph, we obtain a 0.2 hits@1 score using T5-small, T5-base, T5-large, and BLOOM. Using template-based scripts, we create a set of 1 million synthetic factoid QA pairs in the aviation domain from National Transportation Safety Board (NTSB) reports. On our curated QA pairs, the three models of T5 achieve a 0.7 hits@1 score. We validate our findings with the paired student t test and Cohen’s kappa scores. For link prediction on Aviation Knowledge Graph using T5-small and T5-large, we obtain a Cohen’s kappa score of 0.76, showing substantial agreement between the models. Thus, we infer that small language models perform similar to large language models with the infusion of knowledge.

## introduction a large number of pre-trained language models (lms) are used for downstream tasks, such as question answering (qa). generally, these language models are trained on generic domain data, such as web data and news forums. recently, lms are used for downstream tasks in domain-specific fields, namely, healthcare @xcite , radiology @xcite , and aviation @xcite . for tasks such as information extraction (ie) and question answering (qa), knowledge graphs (kgs) are used as a source of * equal contribution external knowledge to boost the performance of models. to a great extent, researchers focus on the synergy of knowledge graph and deep learning @xcite @xcite . with the increase in data, it is observed that larger models are preferred for different tasks across various domains. the large language models (llms) are preferred to obtain better results than small or nonpre-trained models as they have a vast number of parameters and have been trained on a large amount of data. but, the larger model increases the need for computation power and training time. in this paper, we show that small and large models perform likewise with the infusion of knowledge. we can use non-pre-trained models for different tasks across domains that require less computation power and time and still attain the same performance as pre-trained models. we validate our hypothesis with the llms, i.e., t5 @xcite ) & bloom foot_0 . we perform two tasks: a) link prediction, and b) question answering on different datasets: a) aviation knowledge graph (aviationkg) @xcite , and aviation qa pairs (section 4.4), b) movie knowledge base (moviekb) & metaqa (a set of qa pairs), both present in the metaqa dataset @xcite , and c) complex web questions (cwq) @xcite , which uses subsets of freebase @xcite . we perform hypothesis testing to validate our hypothesis. we use paired student t-test and attempt to reject our hypothesis that models have a negligible difference in performance. but, we were not able to repudiate our hypothesis. to strengthen our findings, we use cohen's kappa measure and show significant agreement between models. our contributions are as follows: ## motivation as stated earlier, in section 1, lms are trained on generic datasets. so, knowledge from different sources, i.e., kgs, are used to perform downstream tasks in specific domain areas. llms infused with knowledge are required to perform such tasks, namely, qa and link prediction, which increases the need for computation power and time. we show that computational resources can be saved by using smaller language models for tasks. it is rare to obtain datasets related to the aviation domain, which is in increased demand. we scrape ntsb reports from ntsb's website foot_2 and create qa pairs that can be used by the aviation industry and researchers for information retrieval (ir) and qa purposes. the created dataset will help find insights into aircraft accidents and develop solutions to prevent accidents. ## background & related work a knowledge graph is a collection of entities and relations represented in the form of triplets (subject, relation, object). querying kg in natural language (nl) is a long-standing work. early work focused on rule-based and pattern-based systems @xcite . recently, the work is shifted to seq2seq architecture @xcite and pre-trained models with the advent of neural networks. querying kgs remains a challenge because of the conversion of nl to the graph query language, namely, sparql, cypher, etc. with the value increase of knowledge in the world, the popularity of the kg has escalated. researchers are keenly interested in the synergy of knowledge graphs and deep learning. several methods are exploited considering synergy: a) integrating triplets of kg into the neural network @xcite , b) computing the relevance of entity and relations in a kg using a neural network @xcite . deep learning models use representations of entities and relations to integrate triplets of kg. knowledge graph embeddings are widely used to obtain representations @xcite . the kg embedding models are trained on link prediction over triplets to obtain representations @xcite . recent work has focused on using fine-tuned language models over kge models for link prediction to reduce the number of parameters required to obtain the representations @xcite . lms and kgs are extensively used to improve task-specific performance. still, no study has been done to understand the characteristics of a language model during the synergy of kg and dl. in this paper, we observe the behavior of language models after knowledge infusion with different domain datasets. ## results and analysis this section analyzes the performance of two models: t5 and bloom. ## hypothesis testing we attempt to contradict our hypothesis (1) that the difference in scores for the two models is negligible. we choose paired student t-test @xcite to refute our hypothesis. in our testing, the significance level (p-value) is 0.1, and the sample size is 20% of the test set selected randomly. in comparing the pair of models (section 4.6), we predicted t5-large to perform better than t5-base & t5-small and bloom to perform better than all three models of t5 because of its large size. but, 7 out of 10 times student t-test was unable to reject our hypothesis, and the significance level among the pair of models was greater than 0.1. after not being able to reject the hypothesis, our next step was to strengthen it, so, we calculate cohen's kappa @xcite ) score of the pair of models with different datasets ( thus, the testing supports our hypothesis, and we prove that the level of performance of different models with the infusion of knowledge remains the same. ## conclusion and future work we have successfully created a million factoid qa pairs from the ntsb aircraft accident reports. the qa pairs are used in our experiments with avia-tionkg. we have validated our claim that with the infusion of knowledge to language models, the per-formance of the small language model is similar to the large language model. we substantiate with different language models and a diversity of datasets. our investigation will benefit researchers in selecting the appropriate language model when working with knowledge and save computation power and time. the future line of work is to investigate the performance of models with incomplete and noisy knowledge graphs and study the extent to which the models can outright the domain knowledge.