KILT : a Benchmark for Knowledge Intensive Language Tasks (2021)

Abstract: Challenging problems such as open-domain question answering, fact checking, slot filling and entity linking require access to large, external knowledge sources. While some models do well on individual tasks, developing general models is difficult as each task might require computationally expensive indexing of custom knowledge sources, in addition to dedicated infrastructure. To catalyze research on models that condition on specific information in large textual resources, we present a benchmark for knowledge-intensive language tasks (KILT). All tasks in KILT are grounded in the same snapshot of Wikipedia, reducing engineering turnaround through the re-use of components, as well as accelerating research into task-agnostic memory architectures. We test both task-specific and general baselines, evaluating downstream performance in addition to the ability of the models to provide provenance. We find that a shared dense vector index coupled with a seq2seq model is a strong baseline, outperforming more tailor-made approaches for fact checking, open-domain question answering and dialogue, and yielding competitive results on entity linking and slot filling, by generating disambiguated text. KILT data and code are available at https://github.com/facebookresearch/KILT .

## introduction there has been substantial progress on natural language processing tasks where the inputs are short textual contexts such as a sentences, paragraphs, or perhaps a handful of documents. critically, we have seen the emergence of general-purpose architectures and pre-trained models that can be applied to a wide range of such tasks @xcite . however, for many real world problems, processing at this local level is insufficient. for example, 1 and at https://huggingface.co/datasets? @xmath0. fact checking a claim @xcite requires models to find evidence, often on the web. in knowledgeable open dialogue @xcite , models need access to knowledge from large corpora to sustain informed conversations. in general, solving knowledge-intensive tasks requires-even for humans-access to a large body of information. like in information retrieval (ir) this involves satisfying an information need leveraging large collections of text @xcite . however, while ir focuses of finding relevant material (usually documents), the tasks we consider focus on more fine-grained behavior, such as producing specific answers to queries. for such knowledge-intensive tasks, general infrastructure and architectures across tasks have yet to emerge, and fundamental research questions remain open. for example, while it was long assumed that nonparametric and explicit memory accessed through retrieval is strictly required for competitive results @xcite , recent large pre-trained sequence-to-sequence models such as t5 @xcite and bart @xcite store all knowledge in their parameters while performing remarkably well @xcite . likewise, while the classical approach of information extraction for populating a knowledge base (kb, @xcite seems outof-fashion, recent results show that they remain contenders @xcite . while there are numerous datasets for knowledge-intensive tasks (e.g. @xcite @xcite , to name just a few), it is difficult to answer the above questions generally across them. each dataset comes in a different format, is pre-processed with different assumptions, and requires different loaders, evaluations, and analysis tools. critically, they all use different knowledge sources, from different versions of wikipedia to entirely different corpora. this makes task-to-task comparisons difficult and substantially increases computational overhead. for example, one cannot easily assess whether the same knowledge representation can be re-used if each dataset is tied to a different source. moreover, if one decides to work with different sources across different tasks, many approaches require re-indexing and re-encoding large numbers of documents. if a language model is pre-trained on one snapshot of wikipedia to capture its knowledge, tasks that use other snapshots might require re-training. to facilitate research on models that must access specific information in a knowledge source, we introduce kilt, a benchmark and library for knowledge intensive language tasks. kilt aims to lower the entry barrier for such research by formulating several knowledge-intensive nlp tasks with respect to a common interface and the same unified knowledge source-a single wikipedia snapshot. the kilt benchmark consists of eleven datasets spanning five distinct tasks, and includes the test set for all datasets considered. 2 an important aim of kilt is to cover many different ways of seeking knowledge. for this reason, we select tasks that provide a variety of ways to formulate both the input query (e.g., a claim to verify, a text chunk to annotate, a structured query, a natural question or a conversation) and the expected output (e.g., discrete, extractive, or abstractive). moreover, while some tasks are factoid in nature (e.g., slot filling), others require using background knowledge to answer more complex questions (e.g, eli5) or to sustain a conversation (e.g,. wizard of wikipedia). the format of the kilt benchmark is model-agnostic, so any system capable of producing a textual output given a textual input is eligible to participate. kilt is an in-kb resource @xcite , i.e., the evidence required to answer each of the ~3.2m instances in kilt is present somewhere in the knowledge source. hence there are no unanswerable instances in kilt. although recognizing unanswerable instances is important, we believe the in-kb setting already poses an hard challenge to current state-of-the-art techniques, and thus leave unanswerable instances as future work. kilt enables researchers to develop generalpurpose models and evaluate them across multiple domains, testing hypotheses around task-agnostic memory and knowledge representations without indexing different large-scale textual corpora or writing new io routines. furthermore, the kilt library provides general building blocks to ease research on knowledge intensive nlp. we provide various state-of-the-art information retrieval systems (both neural and non-neural) coupled with different models that read text in the knowledge source and make predictions for different tasks. we evaluate several state-of-the-art models that represent diverse approaches to knowledgeintensive nlp, and find that a hybrid approach combining a neural retriever with a pretrained sequenceto-sequence model outperforms most task-specific solutions when trained end-to-end. we additionally evaluate whether systems can provide evidence for their predictions. with this aim, we augment every instance in kilt with provenance information in the form of textual spans in specific wikipedia pages to corroborate the output. we additionally perform an annotation campaign via amazon mechanical turk to increase the provenance coverage. lastly, in addition to evaluating downstream performance with popular metrics we formulate novel kilt variants for those that award points only if systems find provenance wikipedia pages for the output given the input. the poor absolute performance of our baselines for those metrics indicates the need for focused research on systems able to explain their decisions. in summary, we contribute: ## knowledge source a main feature of the kilt benchmark is the use of a unified knowledge source that contains all information necessary for all tasks. defining a unified knowledge source is a challenging problem -although all tasks use wikipedia, they consider different snapshots. as wikipedia pages are constantly modified, added, and removed, the knowledge can differ drastically from snapshot to snapshot. concretely, the kilt knowledge source is based on the 2019/08/01 wikipedia snapshot and contains 5.9m articles. we describe how each dataset is represented in kilt, and our mapping strategy for aligning data to our chosen snapshot. additional details are in the appendix. ## tasks we consider five tasks that use wikipedia as a knowledge source for kilt: fact checking, open domain question answering, slot filling, entity linking, and dialogue. the diversity of these tasks challenge models to represent knowledge flexibly. some tasks require a discrete prediction (e.g., an entity), others, such as extractive question answering, can copy the output directly from a wikipedia page, while still other tasks must synthesize multiple pieces of knowledge in an abstractive way to produce an output. kilt also provides a variety of ways to seek knowledge, from a claim to verify to a text chunk to annotate, from a structured or natural question to a conversation (see ## provenance annotation campaign we perform an amazon mechanical turk campaign on the nq and eli5 datasets for the dev and test splits. while for the nq our aim is to increase the provenance coverage (i.e., we already have a provenance page for each qa pair) for eli5 we want to collect provenance information from scratch. for each question we ask annotators to indicate if four pre-determined passages contain enough evidence to answer the question and additionally highlight a salient span in them. we select the passages to annotate using our baseline retrieval models, namely tf-idf, dpr, rag and blink + flair (details in the appendix). 9 we only consider passages with some tokens overlap with the gold answers (at least 10%). for nq, we additionally include gold passages among those to annotate, with the twofold objective of controlling the quality of the annotation process and filter out questions that can't be an-swered given the kilt wikipedia snapshot. 10 if no passage is selected by an annotator we ask to provide either another one from wikipedia or an explanation. we collect three annotations for each passage, and insert the passage as new provenance for the question if at least two annotators found enough evidence to answer in it. the average interannotator agreement is 0.3 and 0.1 cohen's kappa for nq and eli5 respectively. note that eli5 questions are in general more complex than nq ones, the required answer is not an extracted span from a page but a free-form explanation that not always can be grounded in wikipedia. to make eli5 data more robust we computed the overlap between provenance passages and answers for each instance using rouge-l and manually annotate instances with low overlap (rouge-@xmath1.15). overall, we were able to collect provenance information for 1507 dev instances (3000 annotated) and 600 test instances (2000 annotated) for eli5, with an average of 1.18 wikipedia pages as provenance per instance. for nq, we filter out on average 8% of data (258 dev and 110 test instances) and include on average 1.57 wikipedia pages as provenance per instance. additional details in the appendix, ## evaluation metrics various tasks in the kilt benchmark need to be evaluated differently, which can make task-wide comparison challenging. further, there are multiple aspects of each system that we want to assess, namely (1) downstream results, (2) performance in retrieving relevant evidence to corroborate a prediction and (3) a combination of the two. we report different metrics to capture these aspects. 11 downstream performance. we consider different metrics to capture the uniqueness of the different tasks in kilt and mimic the typical way to assess performance for each dataset. we use accuracy for tasks that require a discrete output (e.g., an entity); exact match (em) for tasks with extractive (i.e., natural questions, triviaqa) or short abstractive output format (i.e., hotpotqa); finally, for tasks with long abstractive output format, we use rouge-l @xcite for eli5 and f1-score for wizard of wikipedia. for em and f1-score we follow standard post-processing to lowercase, model #parameters trans memnet @xcite 15.5m bert (base) @xcite 110m nsmn @xcite 199m +93m nt t5 (base) @xcite 220m dpr @xcite 220m +15b idx bert (large) @xcite 340m bart (large) @xcite 406m rag @xcite 626m +15b idx blink @xcite 680m +6b idx retrieval. we adopt a page-level formulation and measure the ability of a model to provide a set of wikipedia pages as evidence for a prediction. 12 for most datasets in kilt a single page is enough to provide complete evidence, with the exception of fever (~12% which requires more than one page) and hotpotqa (two pages are always required). we consider the following retrieval metrics in kilt: r-precision, calculated as r r , where r is the number of wikipedia pages inside each provenance set and r is the number of relevant pages among the top-r retrieved pages. for most of the datasets @xmath2. concretely, r-@xmath3. we report the maximum value among all provenance sets for any given input. recall@k, calculated as w n , where n is the number of distinct provenance sets for a given input and w is the number of complete provenance sets among the top-k retrieved pages. for datasets that require more than one page of evidence (e.g., fever and hotpotqa), we use the lowest ranked page in each provenance set to determine its position and remove the other pages in the set from the rank. for both metrics, we report the mean over all test datapoints. ## baselines the kilt tasks provide a dual challenge of retrieving information and conditioning upon that to create an output. various directions could be applied to these. for example, the wikipedia knowledge could be represented explicitly, as natural language or in a structured form, or represented implicitly, as knowledge stored in model parameters. models could be discriminative, extractive, where a specific span is selected as output, or generative, where the model writes an output. we consider retrieval, task-specific, and general baselines for kilt (see ## results we summarize the main results in three tables: downstream performance in by formulating entity linking within kilt, we can evaluate the ability of seq2seq models at this task. they perform surprisingly well, even without any explicit access to knowledge (i.e., bart and t5). these solutions are able to link entity mentions by either leaving them untouched (if they match the correct wikipedia title), completely altering mention text (e.g., "european cup" → "uefa champions league"), or adding disambiguation tokens (e.g., "galatasaray" → "galatasaray s.k. (football)"). we report an example in figure 4 . when considering retrieval alone ( finally, the kilt scores formulation allows us to systematically assesses the performance for output and provenance jointly ( ## discussion there are custom solutions that can easily simplify the slot filling task. for instance, subject entities can be used for lookups by title in wikipedia to retrieve knowledge (this heuristic will always work for zsre), and structured human-curated resources (such as wikidata 13 ) could be used to get all answers right. nevertheless, we are interested in testing if a general model can extract attributes about specific entities from a large body of text. the provenance to justify each system prediction can come from anywhere, including a different system, and this is difficult to detect. moreover our provenance might not be exhaustive-given the redundancy of information in wikipedia there could be other pages with the knowledge needed to solve a kilt instance. we conduct an annotation campaign to mitigate the problem. 2018), triviaqa-web @xcite , quasar-t @xcite , webquestions @xcite and complexwebquestions @xcite -in kilt we exclusively consider publicly available wikipedia-based datasets in order to merge and unify the knowledge source. ## conclusion we introduce kilt, a benchmark for assessing models that need to condition on specific knowledge in a defined snapshot of wikipedia to solve tasks spanning five domains. the goal is to catalyze and facilitate research towards general and explainable models equipped with task-agnostic representations of knowledge. our experiments show promising results for a general solution combining dense retrieval and seq2seq generations, although there is large room for improvements. in particular, we find that provenance of current models is generally low.