Learning to Extract Structured Entities Using Language Models (2024)

Abstract: Recent advances in machine learning have significantly impacted the field of information extraction, with Language Models (LMs) playing a pivotal role in extracting structured information from unstructured text. Prior works typically represent information extraction as triplet-centric and use classical metrics such as precision and recall for evaluation. We reformulate the task to be entity-centric, enabling the use of diverse metrics that can provide more insights from various perspectives. We contribute to the field by introducing Structured Entity Extraction and proposing the Approximate Entity Set OverlaP (AESOP) metric, designed to appropriately assess model performance. Later, we introduce a new Multistage Structured Entity Extraction (MuSEE) model that harnesses the power of LMs for enhanced effectiveness and efficiency by decomposing the extraction task into multiple stages. Quantitative and human side-by-side evaluations confirm that our model outperforms baselines, offering promising directions for future advancements in structured entity extraction. Our source code is available at https://github.com/microsoft/Structured-Entity-Extraction.

## introduction information extraction refers to a broad family of challenging natural language processing (nlp) tasks that aim to extract structured information from unstructured text @xcite @xcite @xcite @xcite @xcite . examples of information extraction tasks include: (i) named-entity recognition @xcite , (ii) relation extraction @xcite , (iii) event extraction @xcite , and (iv) coreference resolution @xcite , as well as higher-order challenges, such as given a text description as well as some predefined schema containing all the candidates of entity types and property keys, we aim to output a structured json for all entities in the text with their information. previous formulations and evaluations of information extraction have predominantly centered around the extraction of ⟨subject, relation, object⟩ triplets. the conventional metrics used to evaluate tripletlevel extraction, such as recall and precision, however, might be insufficient to represent a model's understanding of the text from a holistic perspective. for example, consider a paragraph that mentions ten entities, where one entity is associated with 10 relations as the subject, while each of the other nine entities is associated with only 1 relation as the subject. imagine a system that accurately predicts all ten triplets for the heavily linked entity but overlooks the other entities. technically, this system achieves a recall of more than 50% (i.e., 10 out of 19) and a precision of 100%. however, when compared to another system that recognizes one correct triplet for each of the ten entities and achieves the same recall and precision, it becomes evident that both systems, despite showing identical evaluation scores, offer significantly different insights into the text comprehension. moreover, implementing entity-level normalization within traditional metrics is not always easy due to challenges like coreference resolution @xcite , particularly in scenarios where multiple entities share the same name or lack primary identifiers such as names. therefore, we advocate for alternatives that can offer insights from diverse perspectives. in this work, we propose structured entity extraction, an entity-centric formulation of (strict) information extraction, which facilitates diverse evaluations. we define a structured entity as a named entity with associated properties and relationships with other named-entities. fig. 1 shows an illustration of the structured entity extraction. given a text description, we aim to first identify the two entities "bill gates" and "microsoft". then, given some predefined schema on all possible entity types and property keys (referred to as a strict setting in our scenario), the exact types, property keys, property values on all identified entities in the text are expected to be predicted, as well as the relations between these two entities (i.e., bill gates co-founded microsoft). such extracted structured entities may be further linked and merged to automatically construct kbs from text corpora. along with this, we propose a new evaluation metric, approximate entity set overlap (aesop), with numerous variants for measuring the similarity between the predicted set of entities and the ground truth set, which is more flexible to include different level of normalization (see default aesop in sec. 3 and other variants in appendix a). in recent years, deep learning has garnered significant interest in the realm of information extraction tasks. techniques based on deep learning for entity extraction have consistently outperformed traditional methods that rely on features and kernel functions, showcasing superior capability in feature extraction and overall accuracy @xcite . building upon these developments, our study employs language models (lms) to solve structured entity extraction. we introduce a multistage structured entity extraction (musee) model, a novel architecture that enhances both effectiveness and efficiency. our model decomposes the entire information extraction task into multiple stages, enabling parallel predictions within each stage for enhanced focus and accuracy. additionally, we reduce the number of tokens needed for generation, which further improves the efficiency for both training and inference. human side-by-side evaluations show similar results as our aesop metric, which not only further confirm our model's effectiveness but also validate the aesop metric. in summary, our main contributions are: ## related work in this section, we first review the formulation of existing information extraction tasks and the metrics used, followed by a discussion of current methods for solving information extraction tasks. information extraction tasks are generally divided into open and closed settings. open information extraction (oie), first proposed by @xcite , is designed to derive relation triplets from unstructured text by directly utilizing entities and relationships from the sentences themselves, without adherence to a fixed schema. conversely, closed information extraction (cie) focuses on extracting factual data from text that fits into a predetermined set of relations or entities, as detailed by @xcite . while open and closed information extraction vary, both seek to convert unstructured text into structured knowledge, which is typically represented as triplets. these triplets are useful for outlining relationships but offer limited insight at the entity level. it is often assumed that two triplets refer to the same entity if their subjects match. however, this assumption is not always held. additionally, the evaluation of these tasks relies on precision, recall, and f1 scores at the triplet level. as previously mentioned, evaluating solely on triplet metrics can yield misleading insights regarding the entity understanding. thus, it is essential to introduce a metric that assesses understanding at the entity level through entity-level normalization. in this work, we introduce the aesop metric, which is elaborated on in sec. 3.2. various strategies have been employed in existing research to address the challenges of information extraction. textrunner @xcite initially spearheaded the development of unsupervised methods. recent progress has been made with the use of manual annotations and transformer-based models @xcite . sequence generation approaches, like imo-jie @xcite and gen2oie @xcite , have refined open information extraction by converting it into a sequence-to-sequence task @xcite . genie @xcite focuses on integrating named-entity recognition, relation extraction, and entity linking within a closed setting where a knowledge base is provided. recent work, pivoine @xcite , focuses on improving the language model's generality to various (or unseen) instructions for open information extraction, whereas our focus is on designing a new model architecture for improving the effectiveness and efficiency of language model's information extraction in a strict setting. ## structured entity extraction in this section, we first describe the structured entity extraction formulation, followed by detailing the approximate entity set overlap (aesop) metric for evaluation. we would like to emphasize that structured entity extraction is not an entirely new task, but rather a novel entity-centric formulation of information extraction. ## multi-stage structured entity extraction using language models in this section, we elaborate on the methodology for structured entity extraction using lms. we introduce a novel model architecture leveraging lms, musee, for multi-stage structured entity extaction. musee is built on an encoder-decoder architecture, whose pipeline incorporates two pivotal enhancements to improve effectiveness and efficiency: (i) reducing output tokens through introducing additional special tokens where each can be used to replace multiple tokens, and (ii) multi-stage parallel generation for making the model focus on a sub-task at each stage where all predictions within a stage can be processed parallelly. ## experiments in this section, we describe the datasets used in our experiment, followed by the discussion of baseline methods and training details. ## results as discussed in sec. 4, our musee model is centered around two main enhancements: reducing output tokens and multi-stage parallel generation. by simplifying output sequences, musee tackles the challenge of managing long sequences that often hinder baseline models, like lm-json, genie, imojie, thus reducing errors associated with sequence length. additionally, by breaking down the extraction process into three focused stages, musee efficiently processes each aspect of entity extraction, leveraging contextual clues for more accurate predictions. in contrast, gen2oie's twostage approach, though similar, falls short because it extracts relations first and then attempts to pair entities with these relations. however, a single relation may exist among different pairs of entities, which can lead to low performance with this approach. supplemental ablation study is provided in appendix f. efficiency comparison. as shown in the last column of to better illustrate our model's strength, we show the scatter plots comparing all models with various backbones in fig. 3 on the effectiveness and efficiency. we choose the wikidata-based dataset and the effectiveness is measured by aesop. as depicted, our model outperforms all baselines with a large margin. this advantage makes musee particularly suitable for applications requiring rapid processing of large volumes of data, such as processing web-scale datasets, or integrating into interactive systems where response time is critical. grounding check. as the family of t5 models are pre-trained on wikipedia corpus @xcite , we are curious whether the models are extracting information from the given texts, or they are leveraging their prior knowledge to generate information that cannot be grounded to the given description. we use t5-l as the backbone in this experiment. we develop a simple approach to conduct this grounding check by perturbing the original test dataset with the following strategy. we first systematically extract and categorize all entill m -j so n g en 2o ie im oj ie g en ie m us ee 0 10 20 30 40 50 aesop original perturbed ll m -j so n g en 2o ie im oj ie g en ie m us ee f1 score original perturbed figure 4: grounding check across models on the wikidata-based dataset. musee shows the least performance drop on the perturbed version of data compared to other baselines. ## discussion and conclusion we introduce structured entity extraction (see), an entity-centric formulation of information extraction in a strict setting. we then propose the approximate entity set overlap (aesop) met-ric, which focuses on the entity-level and more flexible to include different level of normalization. based upon, we propose a novel model architecture, musee, that enhances both effectiveness and efficiency. both quantitative evaluation and human side-by-side evaluation confirm that our model outperforms baselines. an additional advantage of our formulation is its potential to address coreference resolution challenges, particularly in scenarios where multiple entities share the same name or lack primary identifiers such as names. models trained with prior triplet-centric formulation cannot solve the above challenges. however, due to a scarcity of relevant data, we were unable to assess this aspect in our current study.