Infusing Knowledge into Large Language Models with Contextual Prompts (2023)

Abstract: Knowledge infusion is a promising method for enhancing Large Language Models for domainspecific NLP tasks rather than pre-training models over large data from scratch. These augmented LLMs typically depend on additional pre-training or knowledge prompts from an existing knowledge graph, which is impractical in many applications. In contrast, knowledge infusion directly from relevant documents is more generalisable and alleviates the need for structured knowledge graphs while also being useful for entities that are usually not found in any knowledge graph. With this motivation, we propose a simple yet generalisable approach for knowledge infusion by generating prompts from the context in the input text. Our experiments show the effectiveness of our approach which we evaluate by probing the fine-tuned LLMs.

## introduction unifying large language models (llms) and knowledge graphs (kgs) is an active area of research for several reasons. @xcite presents a survey of different approaches, including knowledge infusion into large language models from knowledge bases. common techniques for infusing knowledge involve pre-training over a factually-rich corpus prepared from structured knowledge bases or learning soft knowledge prompts pertaining to entities using factual triples from a knowledge base for improving entityspecific inference @xcite . infusing knowledge directly into the model from knowledge bases, though more efficient than retraining llms from scratch, is unrealistic in many real-world applications. maintaining knowledge graphs about customers, organizations and events mentioned in documents is cumbersome and incurs an overhead of maintaining privacy. further, all entities are not equal in knowledge bases, with some entities being highly under-represented. this degrades the performance over downstream tasks for such entities, post-knowledge infusion, due to lack of sufficient knowledge. such entities may be more frequent in domain-specific corpora, which can provide the relevant context for knowledge infusion. as an example, while entities such as barack obama and michelle obama are well known and represented in structured knowledge bases such as wikidata, entities such as michelle obama's great aunt robbie shields terry, being relatively lesser known, exist only in the wikipedia page of the subject and not as an entity in wikidata. motivated by the above, we propose to exploit contextual text from a relevant domain-specific corpus to infuse knowledge into large language models. infusing knowledge directly from documents without having to create knowledge graphs is not only efficient, but also more general. figure 1 outlines the four steps of our proposed approach. given an input prompt composed of a task instance and accompanying instructions, we retrieve relevant context from an indexed corpus by identifying sentences mentioning involved entities. knowledge is then infused into a pre-trained model by fine-tuning over input prompts augmented with the identified context. the knowledge-infused model obtained after fine-tuning can be leveraged for knowledge-intensive downstream tasks. we compare and contrast our knowledge-infusion method against other infusion techniques involving prompting with factual triples @xcite and natural sentences from triples @xcite . our approach offers significant advantages over existing knowledge infusion methods. the proposal alleviates the need for structured knowledge sources by relying solely on domain corpora. using fine-tuning makes our approach simple, scalable, and employable in low-resource settings where excessive compute resources and data sources are unavailable. the method permits seamless integration of structured knowledge graphs, if available, to further enhance the identification of relevant context from corpora via entity linking and disambiguation. our method is also extensible to other modalities like tabular data and graphs. ## knowledge infusion with context our motivation is to create a knowledge-prompting approach that draws on documents and large language models rather than only kgs. accordingly, our approach leverages relevant context retrieved from a domain-specific fine-tuning corpus to infuse knowledge into a pre-trained language model. this is in contrast to approaches utilizing factual triples @xcite or corresponding natural sentences @xcite from knowledge bases, or soft knowledge prompts prepared in an entity-specific manner @xcite as the source for knowledge to be infused. we prepare a pre-trained model by infusing knowledge using full fine-tuning over a specific downstream task, such as tail prediction in triples, question answering, or translation. for this purpose, along with a suitable dataset comprising of downstream task instances, we also identify a domain-specific fine-tuning corpus composed of relevant documents that contain information pertaining to involved entities, which serves as our source of knowledge. primarily, we formulate instances from a training dataset into prompts, by prepending brief task-specific instructions alongside the instance data. for example, for a tailprediction task, alongside a factual triple without the tail entity, an instruction describing the task of tail-prediction is prepended. next, for a given prompt, we identify named entities present in the prompt and retrieve relevant context from the domain-specific fine-tuning corpus. this context is composed of useful information that could provide knowledge about the involved entities to aid in enhancing performance for the task. we infuse knowledge from the context surrounding the entity by augmenting the context alongside the text prompt. for each entity, our contextual information is the phrase, sentence, or paragraph surrounding the entity in a document index. since we are doing full fine-tuning, the model parameters are updated, and the error is propagated based on the task. during inference, task-specific prompts comprising of task instructions and the query to process are given, for which responses are generated based on the knowledge infused. unlike the fine-tuning stage, relevant context is not retrieved and utilized for inference. as an example, for the tail and link prediction task, a dataset for fine-tuning comprising triples is identified, alongside which we identify a relevant domain-specific corpus for the source of knowledge. we formulate the factual triple as a prompt for the llm, post-masking one of the entities or the relation at random. for the unmasked entities and relation, we retrieve relevant context from the corpus and append it to the generated input prompt to obtain the contextual prompt finally used to fine-tune the llm. post fine-tuning, at inference the model is prompted with factual triples formulated as a prompt in the same format but without retrieved context. unlike the approach in dos santos et al. ( 2022 ), neither entity linking via an external knowledge graph nor inference based on information/prompts derived from structured knowledge bases is performed. further, the prompt formulation contrasts from @xcite and @xcite , where for entity and link prediction explicit mentions of the masked positions is present and additional context is not leveraged during fine-tuning. in this work, we directly utilize retrieved-context alongside the input prompt, which is comprised of tokens and is therefore discrete in nature as opposed to soft prompts in a continuous latent space. to include larger amounts of information and bypass the context length restrictions of llms, con-textual prompts can be prepared based on context embedded as soft prompts, allowing for further improvements to efficiency and performance. further, for tasks where structured knowledge sources or information in other modalities, such as tabular data is available, our approach could be extended to leverage the information from these sources and create appropriate contextual prompts for knowledge infusion. ## experiments we conduct experiments with pre-trained flan-t5 models @xcite to demonstrate the role of contextual text as discrete text prompts. we fine-tune the model on two downstream tasks, tail prediction in triples and question answering. tail prediction, as the name indicates, predicts the object in subject, predicate, and object triples. our experiments leverage the following variants of flan-t5 models: flan-t5-small with 80m parameters and flan-t5-base with 250m parameters. fine-tuning and inference were performed in fp-16 mixed-precision mode @xcite over the task-specific datasets using a single nvidia a4000 gpu with 16gbs of vram available. additionally, gradient accumulation and gradient check-pointing @xcite techniques were employed to reduce memory consumption.