Towards Generalized Open Information Extraction (2022)

Abstract: Open Information Extraction (OpenIE) facilitates the open-domain discovery of textual facts. However, the prevailing solutions evaluate OpenIE models on in-domain test sets aside from the training corpus, which certainly violates the initial task principle of domain-independence. In this paper, we propose to advance OpenIE towards a more realistic scenario: generalizing over unseen target domains with different data distributions from the source training domains, termed Generalized OpenIE. For this purpose, we first introduce GLOBE, a large-scale human-annotated multi-domain OpenIE benchmark, to examine the robustness of recent OpenIE models to domain shifts, and the relative performance degradation of up to 70% implies the challenges of generalized OpenIE. Then, we propose DragonIE, which explores a minimalist expression of textual fact: directed acyclic graph, to improve the OpenIE generalization ability. Extensive experiments demonstrate that DragonIE beats the previous methods in both in-domain and out-of-domain settings by as much as 6.0% in F1 score absolutely, but there is still ample room for improvement.

## introduction open information extraction (openie) aims to mine open-domain facts indicating a semantic relation between a predicate phrase and its arguments from plain text @xcite , without fixed relation vocabulary. openie developments have been demonstrated to benefit various domains and applications, such as knowledge base population @xcite , question answering @xcite , and summarization @xcite recently, openie has seen remarkable advances. regarding different strategies for representing open fact, recent techniques with deep neural models can be subsumed under two categories, i.e., sequencebased and graph-based. sequence-based models * corresponding author. predict the facts one by one in an auto-regressive fashion with iterative labeling or generation framework @xcite @xcite , which is the most classical solution in openie. graph-based method formulates openie as a maximal clique discovery problem based on the span-level text graph @xcite , in which the edge between two spans is defined as the combination of their roles in corresponding fact. to the end, o(m 2 ) edges of o(r 2 ) types are constructed for a fact with m spans of r roles. owning to the exquisite design, both sequencebased and graph-based models can identify complicated facts, thus constantly refreshing performance on benchmarks. nonetheless, it is still unexplored whether these models are sufficient for true open-domain extraction. this doubt comes from that the training and test data in existing ope-nie benchmarks are generally independent and identically distributed, i.e., drawn from the same domain @xcite @xcite . however, this assumption does not hold in practice. built on domainindependence @xcite , openie models have to process diverse text, it is common to observe domain shifts among training and test data in applications. therefore, the performance on indomain benchmarks may not exactly measure the generalization of out-of-domain extraction. starting from this concern, we carry out extensive experiments to investigate whether state-of-theart openie models preserve good performance on unseen target domains. to provide a reliable benchmark, we publicize the first generalized openie dataset containing 110,122 open facts annotated humanly on 20,899 sentences collected from 6 completely different domains. we find out that, there are some noticeable semantic differences between open facts in different domains, posing challenges to the generalization of openie models. because of domain shifts, in sequence-based models, the accuracy in each step prediction declines significantly, and the early errors are magnified later. similarly, in the graph-based model, the reduced edge prediction ability struggles to accurately connect o(m 2 ) edges of o(r 2 ) types especially when the span number m and role number r are both no small in complicated facts. as a result, their f1 scores degrade as much as 70% relatively (from 43% to 13%) when applied to unfamiliar domains, thus cannot work well in real-world extraction. the above observations demonstrate full-fledged open-domain extraction still has a long way to go, and suggest a way for a more generalized openie model: we should reduce the extraction complexity to lower the potential risk of prediction errors in domain shifts. this is essentially the occam's razor principle @xcite : among all functions which fit the training data well, simpler functions are expected to generalize better. therefore, we explore a minimalist expression of open fact: by sequentially connecting the boundary positions of all spans in the fact with their order in the text, each open fact can be simply modeled as a directed acyclic graph. then openie is equivalent to predicting the graph adjacency matrix and decoding facts from the directed graph. this idea leverages the sequential priors to reduce the complexity of function space (edge number and type) in the previous graph-based model from quadratic to linear, while avoiding auto-regressive extraction in sequence-based models, thus improving generalization. we implement it in dragonie, a directed acyclic graph based open information extractor. we perform extensive in-domain and out-ofdomain experiments for openie. on the previous commonly used in-domain evaluation, dragonie outperforms the state-of-the-art method, with substantial gains of up to 3.6% average f1 score, 3x speedup, and 5x convergence. meantime, it reduces the number of edges by 66% and the number of edge types by 88% compared with the previous graph-based method. on our newly proposed outof-domain benchmark, dragonie further improves the performance gap to 6.0%, and still exceeds the previous methods with only 10% training data, showing better generalization. detailed analysis shows that dragonie can effectively represent overlapping, nested, discontinuous, and multiple facts despite its simplicity. we also perform a qualitative analysis that summarizes typical extraction errors and outlines the future directions. ## pilot experiment to quantitatively evaluate the robustness of ope-nie model against domain shifts, we first propose a standard evaluation setup for generalized ope-nie. then, we conduct pilot experiments as well as empirical analyses in this section. ## methodology from the above observations, we know that recent openie models are too complex to generalize. in this section, we propose a simplified expression of open fact: directed acyclic graph. we start with the motivation of our new graph structure, then go through the implementation details. john is the premier and first minister of british columbia john premier first minister of of british columbia john premier of first minister of british columbia subject predicate object (a) undirected maximal clique (b) directed acyclic graph ## experimental results our experiments aim to answer three questions: q1 how does dragonie compare to other methods in both in-domain and out-of-domain settings? q2 does dragonie effectively handle complex extraction scenarios despite its simplicity? q3 what causes the performance gap between outof-domain and in-domain openie? ## conclusion in this paper, we lay out and study generalized ope-nie for the first time. we release globe, a largescale, high-quality, multi-domain benchmark with 110,122 open facts, to evaluate the generalization of openie models. furthermore, we explore the minimalist graph expression of open fact: directed acyclic graph, to reduce the extraction complexity and improve the generalization behavior. experimental results show that our proposed method outperforms state-of-the-art baselines in both indomain and out-of-domain settings. this work is a starting point towards building more practical ope-nie models with stronger generalization, and we also present fine-grained analyses which point out promising avenues for further improvement.