Abstractive Open Information Extraction (2023)

Abstract: Open Information Extraction (OpenIE) is a traditional NLP task that extracts structured information from unstructured text to be used for other downstream applications. Traditionally, OpenIE focuses on extracting the surface forms of relations as they appear in the raw text, which we term extractive OpenIE. One of the main drawbacks of this approach is that implicit semantic relations (inferred relations) can not be extracted, compromising the performance of downstream applications. In this paper, we broaden the scope of OpenIE relations from merely the surface form of relations to include inferred relations, which we term abstractive OpenIE. This new task calls for the development of a new abstractive OpenIE training dataset and a baseline neural model that can extract those inferred relations. We also demonstrate the necessity for a new semantics-based metric for evaluating abstractive OpenIE extractions. Via a case study on Complex QA, we demonstrate the effectiveness of abstractive OpenIE.

## introduction open information extraction (openie) is the task of extracting relation tuples from unstructured text @xcite @xcite . unlike traditional information extraction, openie is open domain, intended to be easy to deploy in different domains without fine-tuning. these relations can then be used in downstream applications like summarization @xcite , question-answering @xcite , and knowledge base population @xcite . in order to support these applications, openie needs to extract as many different types of relations as possible. one particular relation type of interest is "inferred relations". we define an "inferred 1 code and models are available at https://github.com/ kevinpei/abstractiveopenie ## abstractive openie abstractive openie is defined as a task that generates ordered tuples in the form of (subject, predicate, object) for all possible relations (inferred or non-inferred) within a given sentence. in this section, we will describe all the pieces required to accomplish this task. ## results and analysis for this section, we focus on the sentence-tuple semantic score because it offers a holistic comparison of the extracted relations and the sentence and does not rely upon potentially incomplete or faulty gold relations. full tables with our empirical results including other metrics can be found in appendix a. we first compare performance on all relations in figure 1 . in general, abstractive openie leads to better performance the higher the proportion of inferred relations in the test set. this is expected because multi 2 oie can not extract inferred relations at all. when considering the full benchmarks, of the data augmentation methods we use, sure augmentation works the best. training on backtranslated oie4 degrades the performance compared to the base extractive oie4 data. this may be because back translation reduces the amount of training data. additionally, back translation often just replaces the gold standard predicate with a synonym instead of changing the syntax of the sentence, which does not help in the extraction of inferred relations. to demonstrate the complementary nature of abstractive openie to extractive openie, we combine their extractions. when combining their extractions, we remove redundant relations by removing relations that are entailed by any other relations. if two relations entail each other, then we keep the longer one. a comparison of combined models can be found in figure 2 . when combining model predictions, we observe that back translation actually helps more than sure augmentation. this suggests that sure augmentation helps extractive openie relations, while back translation is more useful for increasing the recall to inferred relations that could not be extracted by multi 2 oie. the more inferred relations in the benchmark, the more beneficial merging extractions are. on only the inferred relations within each benchmark. to do this, we remove non-inferred relations from the gold standards. we can only measure the resulting recall of the models because the models are trained to generate both inferred and noninferred relations and the metrics we use penalize the precision when there are too many predicted relations for a given sentence, which would be the case for any sentence that had non-inferred relations. figure 3 shows the results of these experiments. as before, the more inferred relations in the benchmark, the better suited an abstractive openie model is for the task. upon a manual examination of the generated relations of each model, we observe that fine-tuning t5 on sure-augmented data results in generated relations replacing some of its predicates with the predicates from sure. ## conclusion in this paper, we introduce abstractive openie, an alternative to what we call extractive openie, the paradigm all current openie models currently follow, in order to address the problems of inferred relations and surface form extraction. we find that existing openie datasets and metrics are ill-suited for this task. as a result, we introduce abstractive training set, model, and metrics. we then compare our models trained on different abstractive training sets and the state-of-the-art extractive ope-nie model using preexisting openie benchmarks. overall, we find that our models achieve higher performance on inferred relations, which extractive openie models have previously struggled with. we believe abstractive openie has potential as a task that will greatly benefit downstream applications that use openie in their pipeline.