KGI : An Integrated Framework for Knowledge Intensive Language Tasks (2022)

Abstract: In this paper, we present a system to showcase the capabilities of the latest state-of-the-art retrieval augmented generation models trained on knowledge-intensive language tasks, such as slot filling, open domain question answering, dialogue, and fact-checking. Moreover, given a user query, we show how the output from these different models can be combined to cross-examine the outputs of each other. Particularly, we show how accuracy in dialogue can be improved using the question answering model. We are also releasing all models used in the demo as a contribution of this paper. A short video demonstrating the system is available at https://ibm.box.com/v/emnlp2022-demos .

## introduction recently, we proposed re 2 g @xcite , the core of our kgi (knowledge graph induction) system. re 2 g combines both neural initial retrieval and reranking into a bart-based sequenceto-sequence generation. we show that the endto-end reranking component also permits merging retrieval results from sources with incomparable scores, enabling an ensemble of bm25 and neural initial retrieval. moreover, to train our system end-to-end, we introduce a novel variation of knowledge distillation to train the initial retrieval, reranker, and generation using only ground truth on the target sequence output. we find large gains in four diverse tasks: zero-shot slot filling, question answering, fact-checking, and dialog, with relative gains of 9% to 34% over the previous state-of-theart on the kilt leaderboard @xcite 1 . in this work, we describe the complete kgi system, which is an enhancement of our previous work. we demonstrate how users can asynchronously interact with the system in real-time, not only for completing triples (aka slot filling), but also for dialogue, fact-checking, and open-domain question answering. we empirically show that our system is the state of the art for these tasks on the kilt leaderboard. in addition, we show how dialog accuracy can be improved by exploiting the question answering model, a novel approach demonstrated in this paper. there are several different intended usages of our system. for example, kgi allows users to interact with different levels of verbosity. also, it enables users to cross-examine results through different kilt tasks that are part of the same gui. we are releasing our best kgi core models (i.e. re 2 g) that we used in this paper at https: //huggingface.co/ibm . ## system architecture the kgi system (figure 1 ) is a web-based application that enables users to asynchronously interact with the system in real-time and allows users to obtain results from four different task-specific models simultaneously in different tabs in the gui. these models are trained using re 2 g model as shown in figure 2 . there is a corresponding ann (approxi- @xcite mate nearest neighbors) index, in this case hnsw (hierarchical navigable small world) @xcite using the open source faiss library @xcite 2 . these indexes contain the passage vectors for the source corpus of the corresponding tasks. more details about the kgi core models, particularly how a model can be trained and a corresponding index can be created, can be found in @xcite . the kgi core model, such as re 2 g, takes a textual query as input and returns a set of generated texts as results together with a set of passages as supporting evidences/references for each of the tasks such as slot filling or fact-checking. refer to ## conclusions in this work, we present our kgi system and show how a user can asynchronously interact with it in real time simultaneously for four nlp tasks. this allows users to interact with same system with different levels of verbosity. the kgi core models (re 2 g) is still the state of the art (in some cases, by wide margin) for 4 out these 5 tasks in the kilt leaderboard. we are releasing those models to the community as part of this paper. in addition, we show how dialog accuracy can be improved by exploiting open domain qa where both models are grounded in the same snapshot of knowledge source. in future work, we would like to exploit results from fact checking and slot filling to further improve accuracy of the response conversation 1: human: i think a lot of young people are addicted to social media platforms. kgi (both dialog): i think so too. social media can be used to promote a product or service, but it can also be used as a way to get information and ideas. human: i sometimes check facebook and post photos there but i don't use it very often. kgi (both dialog): i don't use it as much myself, but i do see it as a way to get information and ideas. human: do you know when was facebook first launched? kgi (hybrid dialog): february 4, 2004 . kgi (conventional dialog): i'm not sure, but i do know that it was founded by mark zuckerberg in the 1980s. conversation 2: human: i use my mobile phone all the time, it is really useful. kgi (both dialog): i love my mobile phone, it is a portable telephone that can make and receive calls over a radio frequency link while the user is moving within a telephone service area. human: yes, i can't imagine how we managed to work together when we didn't have mobile phones. kgi (both dialog): modern mobile telephone services use a cellular network architecture, and, therefore, mobile telephones are called "cellular telephones" or "cell phones" in north america. human: interesting. who is the inventor of cell phones? kgi (hybrid dialog): william rae young, jr. kgi (conventional dialog): i am not sure, but i do know that the first handheld mobile phone was envisioned in the early stages of radio engineering. conversation 3: human: i ate two doughnuts for breakfast today. kgi (both dialog): i love doughnuts. they are a type of fried dough confectionery. human: yes, they are so delicious. i love them too.