Unified Structure Generation for Universal Information Extraction (2022)

Abstract: Information extraction suffers from its varying targets, heterogeneous structures, and demand-specific schemas. In this paper, we propose a unified text-to-structure generation framework, namely UIE, which can universally model different IE tasks, adaptively generate targeted structures, and collaboratively learn general IE abilities from different knowledge sources. Specifically, UIE uniformly encodes different extraction structures via a structured extraction language, adaptively generates target extractions via a schema-based prompt mechanism – structural schema instructor, and captures the common IE abilities via a large-scale pretrained text-to-structure model. Experiments show that UIE achieved the state-of-the-art performance on 4 IE tasks, 13 datasets, and on all supervised, low-resource, and few-shot settings for a wide range of entity, relation, event and sentiment extraction tasks and their unification. These results verified the effectiveness, universality, and transferability of UIE.

## introduction information extraction (ie) aims to identify and structure user-specified information from unstructured texts @xcite . ie tasks are highly diversified due to its varying targets (entity, relation, event, sentiment, etc.), heterogeneous structures (spans, triplets, records, etc.), and demand-specific schemas @xcite @xcite . currently, most ie approaches are taskspecialized, which leads to dedicated architectures, isolated models, and specialized knowledge sources for different ie task. these taskspecialized solutions greatly hinder the rapid architecture development, effective knowledge sharing, and quick cross-domain adaptation of ie systems. first, it is very complicated to develop dedicated architectures for a large amount of ie tasks/settings/scenarios. second, learning isolated models severely restricts the knowledge sharing between related tasks and settings. finally, it is costly and time-consuming to construct data sets and knowledge sources specialized for different ie tasks. therefore, it will be of great benefit to develop a universal ie architecture that can uniformly model different ie tasks, adaptively predict heterogeneous structures and effectively learn from various resources, which we referred to as universal ie. fundamentally, all ie tasks can be modeled as text-to-structure transformations, with different tasks correspond to different structures. for example, as shown in figure 1 , an entity is a named span structure, an event is a schema-defined record structure. these text-to-structure transformations in ie can be further decomposed into several atomic transformation operations: 1) spotting, which locates the desirable spans concerning to given specific semantic types @xcite . for example, locating span "steve" as a person entity and locating "excited" as a sentiment expression. 2) associating, which connects spans by assigning them with semantic roles in pre-defined schemas @xcite . for example, associating "steve" and "apple" by assigning them as the arg1 and the arg2 of a work-for relation. in this way, different ie tasks can be decomposed into a sequence of atomic text-to-structure transformations, and all ie models share the same underlying spotting and associating abilities. for example, entity extraction can be viewed as spotting mention spans of corresponding entity types, while event detection can be reformulated as spotting triggers spans with event types. and the spotting abilities can be shared between these two tasks. based on the above observations, we propose uie, a unified text-to-structure generation architecture that can universally model different ie tasks, adaptively generate targeted structures, and collaboratively learn general ie abilities from different knowledge sources. specifically, to model heterogeneous ie structures, we design a structural extraction language (sel) that can effectively encode different ie structures into a uniform representation, so that various ie tasks can be universally modeled in the same text-to-structure generation framework. to adaptively generate targeted structures for different ie tasks, we propose structural schema instructor (ssi), a schema-based prompt mechanism which controls what to spot, what to associate, and what to generate in uie. to learn common ie abilities for uie, we pre-train uie on large-scale, heterogeneous datasets mined from easily accessible web sources. the large-scale pretrained uie model provides a solid foundation for knowledge sharing and quick adaptation to new ie settings, and significantly boosts the ie performance in all supervised, low-resource, and fewshot settings. we conduct experiments on 13 datasets of 4 main ie tasks (entity/relation/event/sentiment ex-traction and their unification), and supervised, lowresource, and few-shot settings. experiment results show that uie achieves significant improvements in all settings. on supervised settings, uie achieved 1.42% f1 scores improvements over the state-of-the-art, task-specialized architectures on all datasets. on few-shot and low-resource settings, uie exhibits strong on-demand adaptation ability: it outperforms baselines dramatically by a large margin. these results verified the effectiveness, universality, and transferability of uie across different ie tasks, settings, and scenarios. the main contributions of this paper are: ## unified structure generation for universal information extraction information extraction tasks can be formulated as text-to-structure problems, where different ie tasks correspond to different structures. this paper aims to uniformly model the text-to-structure transformations of different ie tasks via a single framework, i.e., different structure transformations will share the same underlying operations and different transformation abilities in a universal model. formally, given a specific pre-defined schema s and texts x, a universal ie model needs to generate a structure that contains the desirable structural information in the text x indicated by the schema s. generally, there are two main challenges here. firstly, due to the diversity of ie tasks, there are many different target structures to extract, e.g., entity, relation, event, etc. secondly, ie tasks are often demand-specific which are defined using different schemas, therefore we need to adaptively control the extraction process. in this section, we describe how to jointly formulate, learn, and conduct various ie tasks in a unified text-to-structure generation architecture, named uie. specifically, we first design structured extraction language (sel) to uniformly encode heterogeneous extraction structures, i.e., encode entity, relation, event into a unified representation. then we describe structural schema instructor (ssi), a schema-based prompt mechanism that controls the uie model which to spot, which to associate, and which to generate for different extraction settings. the details are as follows. ## pre-training and fine-tuning for uie in this section, we describe: 1) how to pre-train a large-scale uie model which captures common ie abilities for different ie tasks; 2) how to adapt uie to different ie tasks in different settings via quick fine-tuning. specifically, we first collect several large-scale datasets from the web, including structured (e.g., knowledge bases), unstructured (e.g., raw texts), and parallel (e.g., wikipedia-wikidata links) data, then we uniformly pre-train our uie model on these heterogeneous datasets. finally, we adapt the pre-trained uie model to the specific downstream ie tasks via on-demand fine-tuning. we found that the pre-trained uie model provides a solid foundation for capturing, sharing, and transferring knowledge between different ie tasks, and new ie tasks can be effectively solved because uie learns general ie ability. ## experiments to verify the effectiveness of uie, we conducted experiments on different ie tasks and settings. ## related work building and pre-training universal models of nlp tasks has attracted a lot of attention in recent years, e.g., contextualized representation @xcite ), text generation @xcite , multi-modal @xcite , and multi-lingual @xcite . this paper proposes and pre-trains the first universal model for information extraction. ie is a long-researched area and many classical neural architectures have been proposed, such as sequence tagging @xcite @xcite @xcite , and mrc @xcite @xcite . and several task-specific pre-training techniques are proposed on these architectures @xcite @xcite . more relevant to our work are generation-based ie methods, which generate text spans via tagging @xcite , index pointer @xcite or copy mechanism @xcite , and these methods usually employ specific classifiers to represent labels. the generation can be enhanced using label templates @xcite @xcite , schema @xcite , and augmented language methods @xcite . compared with previous ie studies which focus on developing more effective task-specialized mod-els, this paper aims to universally model various ie tasks in an unified text-to-structure framework, which can greatly benefit the rapid development, effective knowledge sharing, and quick adaptation of ie systems. ## conclusion in this paper, we propose a unified text-to-structure generation framework -uie, which can universally model different ie tasks, adaptively generate targeted structures, and unfiedly learn general ie abilities from different knowledge sources. experimental results show that uie achieves very competitive performance in both supervised and low-resource settings, which verified its universality, effectiveness, and transferability. a large-scale pre-trained text-to-structure model is also released, which will benefit future studies. for future work, we want to extend uie to kb-aware ie tasks such as entity linking @xcite , and documentaware ie tasks such as co-reference @xcite .