SKILL : Structured Knowledge Infusion for Large Language Models (2022)

Abstract: Large language models (LLMs) have demonstrated human-level performance on a vast spectrum of natural language tasks. However, it is largely unexplored whether they can better internalize knowledge from a structured data, such as a knowledge graph, or from text. In this work, we propose a method to infuse structured knowledge into LLMs, by directly training T5 models on factual triples of knowledge graphs (KGs). We show that models pre-trained on Wikidata KG with our method outperform the T5 baselines on FreebaseQA and WikiHop, as well as the Wikidata-answerable subset of TriviaQA and NaturalQuestions. The models pre-trained on factual triples compare competitively with the ones on natural language sentences that contain the same knowledge. Trained on a smaller size KG, WikiMovies, we saw 3x improvement of exact match score on MetaQA task. The proposed method has an advantage that no alignment between the knowledge graph and text corpus is required in curating training data. This makes our method particularly useful when working with industry-scale knowledge graphs.

## introduction large pre-trained language models, such as bert @xcite , gpt-3 @xcite , t5 @xcite , realm @xcite and ernie @xcite have become the state-of-the-art technology for many tasks. they are commonly pre-trained using unstructured text corpora, on tasks such as next word prediction, next sentence prediction (nsp) or masked language modelling (mlm). especially for t5, selfsupervised learning on unlabelled text corpus with mlm has been a common pre-training recipe @xcite . this is normally followed by a fine-tuning step on the task of interest @xcite , although large language models have also proved useful without this task-specific finetuning @xcite . beyond the capacity of contextual understanding, human-level language understanding pivots on the knowledge about the world. the world knowledge is often expressed as factual triples (c.f. @xcite , in the form of (subject entity, relation, object entity). a knowledge graph (kg) defined by a set of factual triples consists of the subjects and objects as vertices/nodes, and the relations forming the edges connecting them. most of the large scale kgs (e.g. wikidata, @xcite are stored in triple format. llms demonstrate some capacity of learning world knowledge from the natural text corpus @xcite , but it is unclear to what degree they are also able to learn and memorize new knowledge directly from structured kg triples, or from text describing them explicitly. in order to infuse knowledge into a llm, one option is to generate a textual version of the knowledge base, and apply the standard training objectives, e.g. mlm. this is unfortunately highly nontrivial. one can either align sentences with kg triples, as done in ernie @xcite , or generate sentences from triples, as done in kelm @xcite . these approaches are unfortunately hard to port to knowledge graphs with different schemas. these processes are also lossy in that not every triple can be aligned or produce a valid sentence, and there is not a good understanding whether this can introduce unnecessary selection biases on top of biases existing in the original kg. in this work, we propose a method of knowledge infusion for large language models (skill), where llms directly learns from knowledge triples. experiment results shows the checkpoints trained with proposed method on wikidata kg outperform the t5 baselines on four standard closed-book question-answering (qa) tasks. with a smaller kg, wikimovies, the proposed method gain 3× exact match score performance improvement on metaqa task. the models learning directly from knowledge triples performs competitively with the ones with the aligned natural sentences that contain the same amount of knowledge. being able to learn directly from knowledge triples enables easy addition of structured knowledge into language modeling pre-training. ## related work previous works that use knowledge graphs to enhance the quality of knowledge-intensive downstream tasks can be divided into two groups: using knowledge graphs at the inference time, and infusing knowledge into the model weights at the pre-training time. the proposed method falls in the latter group. ## method there are two components of knowledge infusion for llms (skill): the corpus and the training method. we introduce the method based on wikidata kg, but it can be applied to any other kgs. training corpus. we use two corpora with different knowledge representations: wikidata kg @xcite in triple format, and kelm corpus foot_1 @xcite as synthetic natural language sentences converted from wikidata kg. the kelm corpus contains 15, 628, 486 synthetic sentences. to ensure two corpora share the same knowledge, we take the snapshot of the wikidata kg used to created the kelm corpus, which contains 35, 697, 715 triples. to prevent the degradation of model performance on natural language understanding, we mix the wikidata corpus or kelm corpus with natural text from c4 @xcite , 50 : 50, for the knowledge infusion training data. training method. t5 @xcite was trained through masked-language modelling with random span corruption on the c4 corpus. @xcite found that masking salient terms @xcite in pre-training t5 models, instead of masking random token spans, could significantly improve the performance on downstream tasks, e.g. closed-book qa. we apply salient span masking for unsupervised learning in our knowledge-infusing training. to mask the same amount of information is for both corpora, the following method is applied. for a knowledge triple, we mask either the subject or object entity. for a kelm sentence, we identify the aligned triple, with details in appendix a, and mask the full spans corresponding to the subject or object in the triple. the relation tokens are never masked, as there is no robust way to map the abstract relation in knowledge triples to natural ## experiments we assess skill by training and evaluating the knowledge infused models on closed-book qa tasks, where questions are provided without supporting context and external knowledge. ## conclusion we proposed a method to directly infuse knowledge from knowledge graphs into t5 models through pre-training. empirical results show that t5 can learn directly from structured data and apply the learned knowledge to improve closed-book qa results. we also demonstrated that the models pre-trained on factual triples perform competitively with the ones on natural language sentences that contain the same knowledge. by enabling knowledge infusion directly from triples, this method can be very easily applied to industry-scale kgs.