G en IE : Generative Information Extraction (2022)

Abstract: Structured and grounded representation of text is typically formalized by closed information extraction, the problem of extracting an exhaustive set of (subject, relation, object) triplets that are consistent with a predefined set of entities and relations from a knowledge base schema. Most existing works are pipelines prone to error accumulation, and all approaches are only applicable to unrealistically small numbers of entities and relations. We introduce GenIE (generative information extraction), the first end-to-end autoregressive formulation of closed information extraction. GenIE naturally exploits the language knowledge from the pre-trained transformer by autoregressively generating relations and entities in textual form. Thanks to a new bi-level constrained generation strategy, only triplets consistent with the predefined knowledge base schema are produced. Our experiments show that GenIE is state-of-the-art on closed information extraction, generalizes from fewer training data points than baselines, and scales to a previously unmanageable number of entities and relations. With this work, closed information extraction becomes practical in realistic scenarios, providing new opportunities for downstream tasks. Finally, this work paves the way towards a unified end-to-end approach to the core tasks of information extraction.

## introduction the ability to extract structured semantic information from unstructured texts is crucial for many ai tasks such as knowledge discovery @xcite , knowledge maintenance @xcite , symbolic representation, and reasoning @xcite . the interface between free text and structured knowledge is formalized by knowledge base population (kbp; @xcite , which proposes to represent the information contained in text using (subject, relation, object) fact triplets. in this work, we focus on closed information extraction (cie), the problem of extracting exhaustive sets of fact triplets expressible under the relation and entity constraints defined by a knowledge base (kb) schema. traditionally, cie was approached with pipelines that sequentially combine named entity recognition @xcite , entity linking @xcite , and relation extraction @xcite . entity linking and relation extraction serve as grounding steps, matching entities and relations to numerical identifiers in a kb, e.g., qids and pids for wikidata @xcite . @xcite pointed out that such pipeline architectures suffer from the accumulation of errors and proposed an end-to-end alternative. nevertheless, existing methods are still only prac-tical for small schemas with unrealistically small numbers of relations and entities. alternatively, some works have focused on a simpler syntactic task: open information extraction (oie), which produces free-form triplets from texts. in this setup, the entities and relations are not grounded in a kb and, usually, do not represent facts @xcite . as oie triplets contain only surface relations, they have ambiguous semantics, making them hard to use in downstream tasks @xcite if not first aligned with a kb @xcite . since, in practice, oie often consists of structured substring selection, it has recently been framed as an endto-end sequence-to-sequence problem with great success @xcite . indeed, such autoregressive formulations can exploit the language knowledge already encoded in pre-trained transformers @xcite . for example, some tokens can be more easily recognized as possible entities or relations thanks to the pre-training information. inspired by recent successes in oie, we propose the first autoregressive end-to-end formulation of cie that scales to many entities and relations, making cie practical for more realistic kb schemas (i.e. schemas with millions of entities). 1 we employ a sequence-to-sequence bart model @xcite , and exploit a novel bi-level constrained generation strategy operating on the space of possible triplets (from a fixed schema induced by @xcite to ensure that only valid triplets are generated. our resulting model, genie, performs generative information extraction and combines the advantages of a known schema with an autoregressive formulation. the high-level overview of genie is provided in fig. 1 . the constrained generation encodes the known schema and enables the autoregressive decoder to generate textual tokens but only from the set of allowed entities or relations. ## method in this section we formalize genie, an autoregressive end-to-end model for closed information extraction. let us assume a knowledge base (kb) consisting of a collection of entities e , a collection of relations r, and a set of facts (s, r, o) ∈ e × r × e stored as (subject, relation, object) triplets. additionally, we assume that each entity @xmath0. the wikidata kb @xcite , with wikipedia page titles as entity names, and the wikidata relation labels as relation names satisfy these assumptions. ## discussion unifying the cie spectrum. there is a full spectrum of tasks that are closely related to cie and are central to the field of information extraction. the typical setup assumes a kb associated with entities and relations, and the goal is to either annotate the text with information from the kb, or extract structured unambiguous information from the text. the tasks of entity linking and relation classification, already discussed in sec. 2 and sec. 4.3, are two such examples. another example is slot filling (sf), the task of extracting information for a specific entity and relation (e.g., entity mick jagger, relation member of ) from natural language @xcite . all of these problems rely on the same set of logical tasks: identifying entities from the kb in text and understanding how they interact. therefore, it would be beneficial to assume a single model, or a set of models that share parts of the weights and collectively solve all of the tasks. this would allow for the information from a dataset collected for one task (e.g., rc) to be leveraged for improving the performance of another (e.g., sf or cie). ## conclusion this paper provides a new view on closed information extraction (cie) by casting the problem as autoregressive sequence-to-sequence generation. our method, genie, leverages the autoregressive formulation to capture the fine-grained interactions expressed in the text and employs a bi-level constrained generation strategy to effectively retrieve the target representation from a large, structured, compositional predefined space of outputs. experiments show that genie achieves state-of-the-art performance on cie and can scale to a previously unmanageable number of entities and relations. we believe that our autoregressive formulation of cie, coupled with constrained decoding, is a stepping stone towards a unified approach for addressing the core tasks in information extraction.