LSOIE : A Large-Scale Dataset for Supervised Open Information Extraction (2021)

Abstract: Open Information Extraction (OIE) systems seek to compress the factual propositions of a sentence into a series of n-ary tuples. These tuples are useful for downstream tasks in natural language processing like knowledge base creation, textual entailment, and natural language understanding. However, current OIE datasets are limited in both size and diversity. We introduce a new dataset by converting the QA-SRL 2.0 dataset to a large-scale OIE dataset LSOIE. Our LSOIE dataset is 20 times larger than the next largest human-annotated OIE dataset. We construct and evaluate several benchmark OIE models on LSOIE, providing baselines for future improvements on the task. Our LSOIE data, models, and code are made publicly available.

## introduction open information extraction (oie) @xcite aims to automatically extract all factual propositions of a sentence into a series of n-ary tuples. for example, the sentence "the cook baked and ate the cake" would produce two extractions representing the two basic propositions of the sentence: (the cook, ate, the cake) and (the cook, baked, the cake). in oie, extraction arguments are required to be contiguous spans from the sentence and the resulting tuple should be intelligible as natural text when read in order. the schemafree nature of oie provides a flexible framework in which to capture semantic relations between entities in natural language text. open information extraction tuples are useful to a variety of downstream tasks including knowledge base creation @xcite , textual entailment @xcite , and other natural language understanding tasks @xcite . open information extraction relations may be explicitly stated by verbal predicates, or implicitly stated through nominalizations. in this paper, we focus only on explicit extractions. with the original goal of oie as web scale information extraction @xcite , an oie system can focus solely on explicit extractions because the redundancy of language will inevitably display implicit information elsewhere. the interest in oie has grown: both in terms of the types of models that can be applied to tackle oie @xcite @xcite , and in terms of the downstream applications to which oie can be applied @xcite . as the interest in oie grows, however, so too should the scale of the corpora available for training and evaluating oie models. in this paper, we expand the reach and quality of oie data by developing a new dataset, lsoie, which is built by converting the qa-srl bank 2.0 dataset @xcite to the task of oie. our new dataset contains almost ten times as many extractions and about 20 times as many sentences as previous oie datasets built from human anno-figure 1 : an example annotated sentence from qa-srl 2.0 @xcite . in this case, the annotations are derived from the question and answers: -where does someone provide something? in asian countries. who provides something? physicians. what is being provided? drugs. the extracted tuple in our new lsoie dataset is (physicians, provide, drugs, in asian countries). ## the lsoie dataset our work expands upon and addresses the shortcomings present in @xcite and @xcite . we apply a similar conversion processes used for oie2016 on the qa-srl bank 2.0 dataset. in addition, we implement novel conversion heuristics to ensure data quality and order arguments. the result is lsoie, an oie dataset that is much larger and diverse than prior work. ## discussion figure 3 shows precision and recall curves on the lsoie-wiki test set, accompanied by the ls oie model's estimated confidence. the oie modeling task is difficult. results on both evaluation sets show that the bert model and the crf output layer improve over the baseline model. training with the lsoie improves model performance. when science is the target domain, the * sci models are preferable, as they have slightly higher in-domain performance, showing the value of the domain split in lsoie. ## conclusion in this paper, we introduced the lsoie dataset as a resource for supervised oie. we have algorithmically re-purposed the qa-srl bank 2.0 into a new oie dataset, lsoie, which contains over 70,000 sentences and over 150,000 extraction tuples. to benchmark the new dataset, we trained and evaluated a series of supervised oie models, providing baselines for future research on the oie modeling task. the code and datasets introduced in this paper can be found at https://github.com/ jacobsolawetz/large-scale-oie .