When to Use What: An In-Depth Comparative Empirical Analysis of O pen IE Systems for Downstream Applications (2023)

Abstract: Open Information Extraction (OpenIE) has been used in the pipelines of various NLP tasks. Unfortunately, there is no clear consensus on which models to use in which tasks. Muddying things further is the lack of comparisons that take differing training sets into account. In this paper, we present an application-focused empirical survey of neural OpenIE models, training sets, and benchmarks in an effort to help users choose the most suitable OpenIE systems for their applications. We find that the different assumptions made by different models and datasets have a statistically significant effect on performance, making it important to choose the most appropriate model for one’s applications. We demonstrate the applicability of our recommendations on a downstream Complex QA application.

## introduction open information extraction (openie) is the task of extracting relation tuples from plain text @xcite . in its simplest form, openie extracts information in the form of tuples consisting of subject(s), predicate(p), object(o), and any additional arguments(a). openie is an open domain, intended to be easy to deploy in different domains without fine-tuning, with all relations extracted regardless of type. the increasing availability of semi-automatically generated training datasets @xcite as well as significant advances in deep learning techniques have led to the development of state-of-the-art neural models @xcite . since its introduction in @xcite , openie has attracted a large amount of attention by the research community as a tool for a wide range of downstream nlp tasks @xcite . however, there is no real consensus on which openie model is best for each application. one example of this lack of consensus in summarization, where different papers use ollie @xcite , sentence bill gates, former ceo of microsoft, is a harvard dropout. openie extractions (bill gates, was, former ceo of microsoft) (bill gates, is, a harvard dropout) applications ## motivating applications in this section, we identify the properties of openie desired by 5 downstream applications: slot filling, question answering (qa), summarization, event schema induction, and knowledge base population. we survey how openie is used and the properties explicitly desired by papers corresponding to the application, either as motivation for choosing a given openie model or within a case study as a property that would improve performance. the desired properties we observe are higher recall, higher precision, n-ary relation extraction, inferred relation extraction, and fast extraction. we define an "inferred relation" (in ) to be a relation that contains words that are not in the original sentence. for example, given the sentence "bill gates, former ceo of microsoft, is a harvard dropout", the relation (bill gates, was, former ceo of microsoft) can be inferred even though "was" is not in the original sentence. we define an "n-ary relation" (n-ary) to be a relation with more arguments than just (subject, predicate, object). for example, the relation (alice, went, to the store, today) has an additional argument today. slot filling slot filling is a task where an incomplete tuple must be completed using information from a given corpus @xcite . for example, the incomplete tuple (obama, born in, ?) must be completed as openqa involves answering questions given a large database @xcite . complex qa involves using information from multiple sentences to find answers and requires inferring relationships between multiple entities @xcite . @xcite , yin et al. (2015), and clark et al. (2018) are openqa methods that use retrieval-based methods to match openie extractions to questions. by rewriting queries into incomplete tuples, such as rewriting "where was obama born?" into (obama, born in, ?), it is possible to use extracted relations to answer queries by filling in the missing slots in the query. for com-plexqa, khot et al. (2017) and lu et al. (2019) generate graphs from extracted relation tuples, then reason over these graphs to answer questions. in all qa applications surveyed, high recall (hr ) is desired, with lu et al. (2019) using a custom ope-nie method specifically for higher recall. yin et al. (2015)'s case studies state that n-ary would be beneficial while lu et al. (2019) uses a custom openie method that supports in. summarization openie addresses the problems of redundancy and fact fabrication in summarization. redundancy is when a fact is repeated multiple times in the summary. to combat redundancy, ope-nie is used to ensure that the generated summary does not have repeated relations @xcite . fact fabrication is when a fact that is not supported by the text being summarized is in the summary. to combat fact fabrication, openie is used to ensure that the generated summary only contains relations from the original text @xcite . in summarization tasks, hr is useful to ensure summaries contain all information, with ponza et al. (2018) citing greater diversity of extractions as a way to improve performance. high precision (hp ) is also desired by zhang et al. (2021) in order to reduce redundant extractions. event schema induction event schema induction is the automatic discovery of patterns that indicate events, agents, and the agents' roles within that event. extracted relations can be used to find surface forms of events, with redundant tuples being used to induce event schemas. the open nature of openie allows for events to be found regardless of the domain or surface form. hr is useful for event schema induction for the same reason it is useful for slot filling: finding more surface forms allows for more event schemas to be induced (balasubramanian et al., 2013; romadhony et al., 2019; sahnoun et al., 2020). sahnoun et al. (2020) also specifically desire in so that more event schemas can be learned, while balasubramanian et al. (2013) state that n-ary would improve performance. knowledge base population the relations extracted by openie can be used to automatically populate knowledge bases (kbs), creating new nodes and edges. muhammad et al. (2020) and kroll et al. (2021) use learning-based openie models because of their ability to generalize to unseen relations and achieve hr. kroll et al. (2021) also explicitly chooses stanford corenlp and openie6 for their fast extraction times (fe ). ## openie datasets in this section, we discuss the differences between different openie training sets and benchmarks and their shortcomings. we provide statistics about different datasets in ## neural openie models in this section, we describe neural openie models and the properties and assumptions they make that set them apart. neural openie models can be categorized based on how they formulate the openie problem: as a text generation or labeling problem. we provide overviews of the models in ## experiments in this section, we describe how we compare ope-nie models and datasets for the sake of recommendation. to find the best system for different applications, we test whether the properties of ope-nie models and training sets have a statistically significant effect on accuracy in test sets with corresponding properties.we are also interested in how the choice of model affects efficiency in order to satisfy the fast extraction property (fe ). we answer the following questions: we compare spanoie, imojie, multi 2 oie, the 3 stages of openie6: igl-oie, cigl-oie, and openie6, and detie. for each model, we train them with their paper's original dev set and their original hyperparameters. we run all experiments on a quadro rtx 5000 gpu. training datasets: we train models on the spanoie, oie4, imojie, and lsoie training sets. we combine the science and wikipedia domain for both the training and benchmark of lsoie, ensuring there are no duplicate sentences from overlapping sentences in the domains. due to the input structure of spanoie and multi 2 oie, they can not be trained on training datasets with inferred relations. subsequently, we remove any inferred relations from the training sets of those models. similarly, as imojie , openie6, and detie can not extract n-ary relations, we convert all n-ary relations in the training set into binary relations by moving additional arguments into the object. for instance, the relation (alice, went, to the store, today) is converted into (alice, went, to the store today). inferred and n-ary relations were not removed from the gold standards of the test sets. benchmarks: we evaluate all the models on the publicly available english benchmarks oie2016, wire57, reoie2016, carb, and lsoie. ## results in this section, we perform an apples-to-apples comparison among different openie systems to determine the sota openie model and the best general-purpose openie training dataset. ## a case study: complex qa to verify our recommendations, we perform a case study using @xcite , a complex qa method that uses openie to extract entities and predicates from the question and from documents to generate knowledge graphs. the nodes are entities derived from subjects and objects, while the edges are predicates. the knowledge graph is matched to the entities in the question and traversed to find potential answers. because more extractions result in a larger knowledge graph, quest benefits from hr which the authors use their own rule-based openie method to achieve. ## challenges and future directions even with the introduction of neural models, ope-nie systems still have significant room for improvement. in ## conclusion in this paper, we presented an application-focused empirical comparison of recent neural openie models, training sets, and benchmarks. our experiments showed that the different properties of ope-nie models and datasets affect the performance, meaning it is important to choose the appropriate system for a given application and not just choose whatever model is state-of-the-art. we hope that this survey helps users identify the best openie system for their downstream applications and inspires new openie research into addressing the properties desired by downstream applications.