IELM : An Open Information Extraction Benchmark for Pre-Trained Language Models (2022)

Abstract: We introduce a new open information extraction (OIE) benchmark for pre-trained language models (LM). Recent studies have demonstrated that pre-trained LMs, such as BERT and GPT, may store linguistic and relational knowledge. In particular, LMs are able to answer “fill-in-the-blank” questions when given a pre-defined relation category. Instead of focusing on pre-defined relations, we create an OIE benchmark aiming to fully examine the open relational information present in the pre-trained LMs. We accomplish this by turning pre-trained LMs into zero-shot OIE systems. Surprisingly, pre-trained LMs are able to obtain competitive performance on both standard OIE datasets (CaRB and Re-OIE2016) and two new large-scale factual OIE datasets (TAC KBP-OIE and Wikidata-OIE) that we establish via distant supervision. For instance, the zero-shot pre-trained LMs outperform the F1 score of the state-of-the-art supervised OIE methods on our factual OIE datasets without needing to use any training sets.

## introduction pre-trained language models (lm), such as bert @xcite and @xcite , have revolutionized nlp over the last several years and advanced the state-of-theart results in a wide set of downstream nlp tasks. recent studies show that a considerable amount of linguistic @xcite and relational knowledge @xcite @xcite has been captured by the pretrained lms via pre-training on large-scale textual corpora. these approaches often design "fill-in-theblank" questions based on pre-defined relations. for example, a question "bob dylan was born in _" is manually created for the lms to answer the "birthplace" relation of "bob dylan". most existing approaches that evaluate what pretrained lms have learned are based on benchmarks with pre-defined relation categories. yet, the benchmarks present two limitations. first, most benchmarks only cover a limited number of pre-defined relations. therefore, it is unclear whether the pretrained lms have stored general open relation information. for example, the google-re in lama benchmark @xcite includes only three relations (i.e., "birthplace", "birthdate", and "deathplace"), while there are hundreds of relations available in the real world scenario. second, a majority of benchmarks evaluate lms in a close manner. this means that the gold relation is given to the models. for example, "was born in" is given as the model's input. besides, the existing benchmarks often provide a single gold relation for each input sentence. however, an input sentence may indicate multiple relations, e.g., containing both "birthplace" and "birthdate" information about an argument or entity. we are curious: instead of the limited relational information, can we systematically benchmark the general information stored in the pre-trained lms? in this work, we set up a new open information extraction (oie) benchmark, called ielm, towards testing the general and open relational information stored in pre-trained lms. we refer to oie as it is a task that is designed to extract open relations from massive corpora without requiring a pre-defined relation category. as shown in figure 1 , we successfully convert pre-trained lms to zero-shot oie systems. we apply them to two standard oie datasets, including carb @xcite and re-oie2016 @xcite , as well as two new large-scale factual oie datasets in our ielm benchmark. we show that the zero-shot pre-trained lms outperform the fully supervised state-of-the-arts on factual oie datasets. standard oie datasets rely on human annotations and often consist of thousands of gold triples and sentences. unlike those datasets, we create two large-scale oie datasets, namely tac kbp-oie and wikidata-oie, via distant supervision from knowledge graphs. for example, wikidata-oie is constructed via aligning english wikipedia to wikidata triples, resulting in millions of triples and documents. the design of zero-shot lms for oie is important: by encoding the noun chunks as arguments in the input, we only make use of the parameters of pre-trained lms to decode the predicates (or relations) between the arguments. to the best of our knowledge, this is the first attempt to systematically evaluate pre-trained lms in a zero-shot oie setting. to summarize, our key contributions are the following. the two new oie datasets are called tac kbp-oie and wikidata-oie, which are constructed via distant supervision from two knowledge graphs (tac kbp and wikidata). our benchmark is a general oie benchmark, helping the development of future oie systems. ## language models as zero-shot information extractors for open information extraction (oie), we take an input as a np-chunked sentence and output a set of triples. below is an example. input dylan np was born in minnesota np , and was awarded nobel prize np . output (dylan; born in; minnesota), (dylan; awarded; nobel prize). np denotes the noun phrase. ## results in this section, we show that pre-trained lms are effective zero-shot oie systems, and exceed the previous state-of-the-art oie systems on our largescale factual oie datasets in ielm benchmark. to keep our evaluation as simple as possible, the hyperparameters and settings are shared across datasets. more experimental details are described in the appendix. ## related work pre-trained language models (lm), e.g., bert @xcite , gpt @xcite , and large lms over 100b parameters @xcite @xcite contain growing amount of linguistic and factual knowledge obtained via pre-training on large-scale corpora. to evaluate their abilities, researchers have created many knowledge benchmarks. lama leverages manually created prompts @xcite . recent studies have also developed soft prompts @xcite ) for fact retrieval. kilt @xcite proposes a knowledge-intensive benchmark concerning several downstream tasks to evaluate lms' ability in capturing knowledge. wang et al. (2022) have utilized a set of knowledge-intensive structure prediction tasks to evaluate the knowledge in pre-trained lms. shen et al. (2022) have adapted kg completion as a benchmark to evaluate lms. besides relational knowledge, closedbook openqa (roberts et al., 2020) benchmarks (in which lms answer the open-domain questions without retrieving contexts) have also been adopted as a way to evaluate lms' knowledge. while the existing benchmarks evaluate lms in an implicit open information extraction systems, e.g., ol-lie @xcite , reverb @xcite , stanford openie @xcite , ope-nie 5 @xcite , rnnoie @xcite , and openie 6 @xcite aim to extract triples from web corpora for open schema kgs. besides, nell @xcite , deepdive @xcite , knowledge vault @xcite extract information based on a fixed schema or ontology, where humans help improve the accuracy of the extractions. probase @xcite produces taxonomies instead of rich typed relations in general kgs. our benchmark first evaluates lms' unsupervised information extraction ability on common open information extraction datasets such as carb @xcite and re-oie2016 (zhan and zhao, 2020), and then aligns the extracted triples to kg triples for large-scale knowledge extraction benchmark construction. our algorithm is similar to the generation algorithm in deepex @xcite . the focus of this work is to benchmark the zero-shot oie performance of pre-trained lms on both standard and factual oie datasets. to further improve the oie performance, the ranking module in deepex can be useful. the structure pre-training proposed in @xcite can also be helpful. ## conclusion we benchmark the general relational information in pre-trained language models (lm) in an open information extraction (oie) setup. we find that the pretrained lms contain a considerable amount of open relational information through large-scale evaluation on both standard oie datasets and newly created large-scale factual oie datasets in our ielm benchmark. we are able to turn pre-training lms into zero-shot oie systems to efficiently deliver the benchmark results. the reach of this result is broad and has potential downstream utility for deep neural network interpretation, information extraction, and knowledge graph construction. although the results are promising, we argue that our results just indicate a lower bound about what the lms have. we hope our results will foster further research in the lm oie benchmark direction.