A Survey on Open Information Extraction from Rule-based Model to Large Language Model (2024)

Abstract: Open Information Extraction (OpenIE) represents a crucial NLP task aimed at deriving structured information from unstructured text, unrestricted by relation type or domain. This survey paper provides an overview of OpenIE technologies spanning from 2007 to 2024, emphasizing a chronological perspective absent in prior surveys. It examines the evolution of task settings in OpenIE to align with the advances in recent technologies. The paper categorizes OpenIE approaches into rule-based, neural, and pre-trained large language models, discussing each within a chronological framework. Additionally, it highlights prevalent datasets and evaluation metrics currently in use. Building on this extensive review, this paper systematically reviews the evolution of task settings, data, evaluation metrics, and methodologies in the era of large language models, highlighting their mutual influence, comparing their capabilities, and examining their implications for open challenges and future research directions.

## introduction open information extraction (openie) aims to extract structured information from unstructured text sources @xcite , typically outputting relationships as triplets (arg 1 , rel, arg 2 ). as illustrated in figure 1 , unlike standard ie, which relies on predefined categories to identify relationships, openie operates without such constraints, enabling the extraction of diverse and unforeseen relations. this flexibility makes openie especially valuable for rapidly evolving natural language processing (nlp) tasks such as question answering, search engines, and knowledge graph completion @xcite , as well as for handling large-scale and dynamic data sources like web data. since its inception in 2007, the field of ope-nie has witnessed continual advancements. ini-* these authors contributed equally to this work. to accommodate the technological shift, a variety of methods and task settings have evolved within diversified openie approaches. the emergence of large language models (llms) in 2023 has marked another revolutionary phase, steering openie toward generative information extraction. the robust generalization abilities of these models not only advance the technical prowess of openie systems but also facilitate the convergence of methods and task settings -revisiting the original, straightforward @xmath0. this transition fosters potential integration with standard ie tasks, pointing toward a promising future where extraction tasks are tackled through a unified, multi-task approach. as a result, there has been a decline in openie research in the llm era. is openie research going to its end? how can traditional openie research inspire ie research in the llm era? previous surveys largely focus on pre-llm era models or limit their scope to methodological insights @xcite @xcite @xcite . while recent studies @xcite delve into information extraction in the llm era, they largely bypass openie, concentrating instead on standard ie tasks. we aim to bridge this gap by providing a holistic survey of the openie field from a chronological view, addressing the two research questions above. from a chronological perspective, we summarize all task settings (section 2), data (section 3), evaluation metrics (section 4), and mainstream methods (section 5) before and after llm era. we use a single table to summarize mainstream methods and results from different periods. we emphasize the co-evolution between models and task settings, and the various sources of information used to address open challenges. based on this, we compare the ideas and relative strengths and weaknesses of large models and traditional models (section 6.1), review the impact of large language models on open information extraction and traditional methods (section 6.2), and explore future directions (section 6.3). ## task settings we categorize openie task settings into three groups: open relation triplet extraction (orte), open relation span extraction (orse) and open relation clustering @xcite . orte is the classic task setting, while orse and orc settings are variations developed to cater to diverse models with the advancement of nlp techniques. for all three task settings, openness is shown in the absence of restraints on relation types. figure 2 depicts the workflow for each task setting. orte task: @xmath1, relation, entity 2 ) triplets from a vast corpus of unstructured web text, where entity 1 , entity 2 and relation consist of selected words from input sentences. although the term triplet is more commonly used, the actual extraction tasks are not always limited to triplets and can involve more diverse n-ary relations, such as condition, temporal information, etc. this task setting, irrespective of the learning method or the forms of input and output, represents the most idealized configuration. orse task: entities + @xmath2, open relation span extraction finds relational spans according to previously extracted predicates and entities, aiming to partition complex tasks into easier ones to improve model performance. however, it should be clear that errors in entity extraction steps can accumulate in two-stage pipelines. see open relation extraction (orse) in fig. 2 for an example. orc task: entities + @xmath3, also known as open relation extraction, clusters relation instances (h, t, s), where h and t denote head entity and tail entity respectively, and s denotes the sentence corresponding to two entities. different from the orte, orc does not extract relation from text but uses text between two entities to represent the relation. clustering similar relations is a step forward in labeling specific relations to each relation instance. these task settings outlined above are distinctly characterized by era-specific traits and methodologies, further discussed in section 5. summarized in ## datasets question answering (qa) derived datasets are converted from other crowd-sourced qa datasets. oie2016 @xcite , one of the most popular openie benchmarks, leverages qa-srl @xcite annotations. additional datasets extend from oie2016, such as aw-oie @xcite , re-oie2016 @xcite and carb @xcite . lsoie @xcite , is created by converting the qa-srl 2.0 dataset @xcite to a large-scale openie dataset, which claims to be 20 times larger than the next largest human-annotated openie dataset. crowdsourced datasets are created from direct human annotation, including @xcite , saoke dataset @xcite , and benchie dataset @xcite . knowledge base (kb) derived datasets are established by aligning triplets in kbs with text in the corpus. several works @xcite have aligned the new york times corpus @xcite with freebase @xcite triplets, resulting in several variations of the same dataset, nyt-fb. others are created by aligning relations of given entity pairs @xcite , such as tacred @xcite , fewrel @xcite , t-rex @xcite , t-rex spo and t-rex ds @xcite . coer @xcite , a large-scale chinese kb dataset, is automatically created by an unsupervised open extractor. instruction-based datasets transform ie tasks into instruction-following tasks to harness the capabilities of llms. strategies include integrating ex-isting ie datasets into a unified-format @xcite , and deriving others from wikidata and wikipedia such as instructopen-wiki @xcite , instructie @xcite , and wikidata-oie @xcite . overall, kb derived datasets are mostly used in orc task settings, whereas qa derived, crowdsourced, and instruction-based datasets are usually used in orte and orse task settings. we provide more detailed descriptions in appendix c. ## evaluation evaluation metrics for openie models vary by task setting. in the orte and orse settings, models are assessed using precision, recall, f1 score, and auc, potentially employing various scoring functions. in the orc setting, performance is evaluated using b 3 @xcite , v-measure (rosenberg and hirschberg, 2007), and ari @xcite . to compare the extracted and golden triplets, various datasets employ different matching strategies, typically categorized into token-level and fact-level scorers. token-level scorers focus on individual tokens to ensure precision and semantic accuracy, accommodating linguistic variability @xcite , enhancing conciseness @xcite , and adapting to complex model outputs like those from llms @xcite . fact-level scorers assess the informational faithfulness of extractions to ensure reliable knowledge extraction, validating semantic and information integrity @xcite @xcite to enhance openie evaluations comprehensively. further details are discussed in appendix d. from the perspective of task formulation, tokenlevel scorers are better suited for open relation span extraction (orse), where outputs are succinct, and labeling models in open relation triplet extraction (orte), whose outputs are precise tokens derived from the inputs. conversely, fact-level scorers are more appropriate for generative models in orte, particularly llms, whose outputs exhibit diversity and necessitate semantic evaluation. ## a chronological review of mainstream methods the research approaches for open ie have undergone three significant changes along with technological advancements. we categorize these periods into three eras: the pre-neural era, dominated by rule-based and statistic-based methods; the neural model era, primarily based on neural networks; and the llms era, characterized by the use of llms. chronologically, we will discuss the key models and methods from each period and explore their connections, as depicted in figure 3 . more details on model implementation provided in appendix.a ## discussion this section reviews the diverse sources of information used by openie models and discusses current limitations and future prospects, offering a comprehensive overview of the field's evolving trajectory. ## conclusion we systematically survey the development of ope-nie from a chronological perspective, highlighting historical trends in task settings and model development. we draw important connections and derive lessons from the influence of technology on task settings, examining the advantages and disadvantages of both past and present methods. furthermore, we explore the increasingly blurred distinctions between openie and standard ie. for researchers in llms, past work should not be overlooked; instead, it should serve as a valuable resource for future inquiries.