Robust Retrieval Augmented Generation for Zero-shot Slot Filling (2021)

Abstract: Automatically inducing high quality knowledge graphs from a given collection of documents still remains a challenging problem in AI. One way to make headway for this problem is through advancements in a related task known as slot filling. In this task, given an entity query in form of [Entity, Slot, ?], a system is asked to ‘fill’ the slot by generating or extracting the missing value exploiting evidence extracted from relevant passage(s) in the given document collection. The recent works in the field try to solve this task in an end-to-end fashion using retrieval-based language models. In this paper, we present a novel approach to zero-shot slot filling that extends dense passage retrieval with hard negatives and robust training procedures for retrieval augmented generation models. Our model reports large improvements on both T-REx and zsRE slot filling datasets, improving both passage retrieval and slot value generation, and ranking at the top-1 position in the KILT leaderboard. Moreover, we demonstrate the robustness of our system showing its domain adaptation capability on a new variant of the TACRED dataset for slot filling, through a combination of zero/few-shot learning. We release the source code and pre-trained models.

## introduction slot filling is a sub-task of knowledge base population (kbp), where the goal is to recognize a pre-determined set of relations for a given entity and use them to populate infobox like structures. this can be done by exploring the occurrences of the input entity in the corpus and gathering information about its slot fillers from the context in which it is located. a slot filling system processes and indexes a corpus of documents. then, when prompted with an entity and a number of relations, 1 our source code is available at: https://github. com/ibm/kgi-slot-filling it fills out an infobox for the entity. some slot filling systems provide evidence text to explain the predictions. figure 1 illustrates the slot filling task. many kbp systems described in the literature commonly involve complex pipelines for named entity recognition, entity co-reference resolution and relation extraction @xcite . in particular, the task of extracting relations between entities from text has been shown to be the weakest component of the chain. the community proposed different solutions to improve relation extraction performance, such as rule-based @xcite , supervised @xcite , or distantly supervised @xcite . however, all these approaches require a considerable human effort in creating hand-crafted rules, annotating training data, or building well-curated datasets for bootstrapping relation classifiers. recently, pre-trained language models have been used for slot filling @xcite , opening a new research direction that might provide an effective solution to the aforementioned problems. in particular, the kilt benchmark @xcite , standardizes two zero-shot slot filling tasks, zsre @xcite and t-rex @xcite , providing a competitive evaluation framework to drive advancements in slot filling. however, the best performance achieved by the current retrieval-based models on the two slot filling tasks in kilt are still not satisfactory. this is mainly due to the lack of retrieval performance that affects the generation of the filler as well. in this work, we propose kgi (knowledge graph induction), a robust system for slot filling based on advanced training strategies for both dense passage retrieval (dpr) and retrieval augmented generation (rag) that shows large gains on both t-rex (+38.24% kilt-f1) and zsre (+21.25% kilt-f1) datasets if compared to previously submitted systems. we extend the training strategies of dpr with hard negative mining @xcite , demonstrating its importance in training the context encoder. in addition, we explore the idea of adapting kgi to a new domain. the domain adaptation process consists of indexing the new corpus using our pretrained dpr and substituting it in place of the original wikipedia index. this enables zero-shot slot filling on the new dataset with respect to a new schema, avoiding the additional effort needed to rebuild nlp pipelines. we provide a few additional examples for each new relation, showing that zeroshot performance quickly improves with a few-shot learning setup. we explore this approach on a variant of the tacred dataset @xcite ) that we specifically introduce to evaluate the zero/fewshot slot filling task for domain adaption. the contributions of this work are as follows: section 2 present an overview of the state of the art in slot filling. section 3 describes our kgi system, providing details on the dpr and rag models and describing our novel approach to hard negatives. our system is evaluated in sections 4 and 5 which include a detailed analysis. section 6 concludes the paper and highlights some interesting direction for future work. ## related work the use of language models as sources of knowledge @xcite @xcite , has opened tasks such as zero-shot slot filling to pre-trained transformers. furthermore, the introduction of retrieval augmented language models such as rag @xcite and realm @xcite also permit providing textual provenance for the generated slot fillers. kilt @xcite was introduced with a number of baseline approaches. the best performing of these is rag @xcite . the model incorporates dpr @xcite to first gather evidence passages for the query, then uses a model initialized from bart @xcite to do sequence-to-sequence generation from each evidence passage concatenated with the query in order to generate the answer. in the baseline rag approach only the query encoder and generation component are fine-tuned on the task. the passage encoder, trained on natural questions @xcite is held fixed. interestingly, while it gives the best performance of the baselines tested on the task of producing slot fillers, its performance on the retrieval metrics is worse than bm25 @xcite . this suggests that fine-tuning the entire retrieval component could be beneficial. another baseline in kilt is bart large fine-tuned on the slot filling tasks but without the usage of the retrieval model. in an effort to improve the retrieval performance, multi-task dpr @xcite used the multi-task training of the kilt suite of benchmarks to train the dpr passage and query encoder. the top-3 passages returned by the resulting passage index were then combined into a single sequence with the query and a bart model was used to produce the answer. this resulted in large gains in retrieval performance. densephrases @xcite is a different approach to knowledge intensive tasks with a short answer. rather than index passages which are then consumed by a reader or generator component, it indexes the phrases in the corpus that can be potential answers to questions, or fillers for slots. each phrase is represented by the pair of its start and end token vectors from the final layer of a transformer initialized from spanbert @xcite . @xcite addresses the retrieval task in kilt slot filling by using a sequence-to-sequence transformer to generate the title of the wikipedia page where the answer can be found. this method can produce excellent scores for retrieval but it does not address the problem of producing the slot filler. it is trained on blink @xcite and all kilt tasks jointly. open retrieval question answering (orqa) @xcite introduced neural information retrieval for the related task of factoid question answering. like dpr, the retrieval is based on a biencoder bert @xcite model. unlike dpr, orqa projects the bert @xcite vector to a lower dimensional (128) space. it also uses the inverse cloze pre-training task for retrieval, while dpr does not use retrieval specific pre-training. ## knowledge graph induction figure 2 shows kgi, our approach to zero-shot slot filling, combining a dpr model and rag model, both trained for slot filling. we initialize our models from the natural questions @xcite trained models for dpr and rag available from hugging face @xcite foot_0 . we then employ a two phase training procedure: first we train the dpr model, i.e. both the query and context encoder, using the kilt provenance ground truth. then we train the sequence-to-sequence generation and further train the query encoder using only the target tail entity as the objective. it is important to note that the same query encoder component is trained in both phases. query encoder generator passage encoder dpr rag kilt knowledge source head [sep] relation tail passages ann index ## kilt experiments since the transformers for passage encoding and generation can accept a limited sequence length, we segment the documents of the kilt knowledge source (2019/08/01 wikipedia snapshot) into passages. the ground truth provenance for the slot filling tasks is at the granularity of paragraphs, so we align our passage segmentation on paragraph boundaries when possible. if two or more paragraphs are short enough to be combined, we combine them into a single passage and if a single paragraph is too long, we truncate it. ## conclusion in this paper, we presented kgi, a novel approach to zero-shot slot filling. kgi improves dense passage retrieval using hard negatives from the dense index, and implements a robust training procedures for retrieval augmented generation. we evaluated kgi on both t-rex and zsre slot filling datasets, ranking at top-1 position in the kilt leaderboard with a net improvement of +38.24 and +21.25 percentage points in kilt-f1, respectively. moreover, we proposed and release a new benchmark for zero/few-shot slot filling based on tacred to evaluate domain adaptation where our system obtained much better zero-shot results compared with the baselines. in addition, we have observed significant improvement in results for kgi when rapidly fine-tuned in a few-shot setting. this work opens promising future research directions for slot filling and other related tasks. we plan to apply dpr with dense negative sampling to other tasks in the kilt benchmark, including dialogue and question answering. likewise, an in-depth investigation on more effective strategies for domain adaptation, such as the combination of zero-shot and few-shot learning involving human-in-the-loop techniques, would be another interesting direction to explore.