Easy-to-Hard Learning for Information Extraction (2023)

Abstract: Information extraction (IE) systems aim to automatically extract structured information, such as named entities, relations between entities, and events, from unstructured texts. While most existing work addresses a particular IE task, universally modeling various IE tasks with one model has achieved great success recently. Despite their success, they employ a one-stage learning strategy, i.e., directly learning to extract the target structure given the input text, which contradicts the human learning process. In this paper, we propose a unified easy-to-hard learning framework consisting of three stages, i.e., the easy stage, the hard stage, and the main stage, for IE by mimicking the human learning process. By breaking down the learning process into multiple stages, our framework facilitates the model to acquire general IE task knowledge and improve its generalization ability. Extensive experiments across four IE tasks demonstrate the effectiveness of our framework. We achieve new state-of-the-art results on 13 out of 17 datasets.

## introduction information extraction (ie) is a crucial task in natural language processing (nlp) that involves extracting structured knowledge from unstructured text data @xcite , enabling various applications such as information retrieval @xcite , knowledge graph construction @xcite , and question answering @xcite . depending on what kind of information is to be extracted, ie consists of a wide range of tasks, including named entity recognition (ner) @xcite , joint entity and relation extraction (re) @xcite , event extraction (ee) @xcite , and aspect-based sentiment analysis (absa) @xcite . traditionally, ie has been approached with specialized models that are designed to handle specific ie tasks. for example, ner is often formulated as a sequence labeling @xcite or span-based classification @xcite problem. the more complex re or ee task is usually solved with pipeline approaches that split the original task into several sequential subtasks and design specific models for each subtask @xcite @xcite . these models often require extensive task-specific knowledge to design dedicated model architectures and thus suffer from poor generalization. recently, motivated by pre-trained generative models such as t5 @xcite that handle multiple tasks with the unified text-to-text format, there has been a shift towards the use of unified models for ie as well, which can tackle all ie tasks with a single model structure. for example, tanl @xcite tackles various ie tasks with a text-to-text generative model by framing them as translation between augmented natural languages. uie @xcite models heterogeneous ie structures into a uniform representation via a structural extraction language. despite the success of existing unified models on various ie tasks, they typically adopt a one-stage learning paradigm, i.e., directly learning to predict the target structure given the input text. in contrast, humans often learn to tackle a task in an easy-tohard manner. they learn basic concepts or skills before solving more complex problems and often tackle harder examples to gain a better understanding of the problem. taking the re task as an example, it aims to extract relational triplets, where each triplet consists of a head entity, a relation, and a tail entity. to tackle it, humans first learn some basic skills, such as identifying entities, recognizing relations, and associating entities and relations, before extracting complex relational triplets. this process facilitates humans to learn meaningful substructures and the dependencies among them. moreover, in practical scenarios, humans usually encounter harder cases, i.e., long input context of multiple sentences containing more entities and relations. by solving hard cases, humans improve their understanding of the task and problem-solving skills. by comparison, models are only trained with the provided training data. the gap between the model and human learning strategies hinders ie models from further development. to bridge the gap, we propose an easy-to-hard (e2h) learning framework for ie tasks in this paper. e2h mimics the human learning procedure to learn each ie task in stages, i.e., the easy stage, the hard stage, and the main stage. the easy stage aims to help the model acquire basic skills of the task, and the hard stage aims to assist the model in handling broad-range variations of the task via training the model with diverse and harder data. finally, the main stage focuses on the main task at hand for training. thus an immediate question is how to prepare the data with different levels of difficulty for the easy and hard stages. it is labor-intensive and challenging to construct such data manually. in this work, we attempt only to leverage the existing data of the main task for constructing the data. specifically, for the easy stage, we observe that the target ie structure often has meaningful substructures. therefore, we identify several basic skills for each task according to the substructures of its target structure. returning to the re example, the skills can be recognizing the entities, relations, and dependencies between them. we can automatically construct training data for learning these skills by modifying the input prompt and decomposing the target structure of the main task. for the hard stage, we combine two training instances of the main task to build a harder training instance by concatenating their input texts to form the new text and their targets to build the new target. the new instance contains more entities, relations, and complicated contexts, making it harder than the original instances. through these two novel construction strategies, we can reduce much human effort to obtain the data for different stages. to summarize, our contributions are three-fold: ## task definition this paper investigates four common ie tasks, i.e., ner, re, ee, and absa. in this section, we provide formal definitions of these tasks. detailed examples of these tasks are in appendix a.3. ## our e2h framework our proposed easy-to-hard (e2h) framework consists of three sequential stages: the easy stage, the hard stage, and the main stage. in this section, we first introduce our text-to-structure formulation for facilitating three-stage learning in a unified framework. next, we will describe how to realize the easy and hard stages. finally, we will discuss the main stage as well as the detailed training and inference process of our framework. ## more analysis analysis on different learning strategies in the main result table, we report the results of e2h trained with the @xmath0, i.e., training the model in the easy, hard, and main stages sequentially. in this section, we investigate alternative learning strategies. is each skill necessary in the easy stage? to quantify the contribution of each skill, we examine the performance of e2h-base after removing a basic skill for training in the easy stage. ablation results on four datasets across four tasks are shown in ## conclusion this paper proposes an easy-to-hard learning framework consisting of the easy stage, the hard stage, and the main stage for ie. two novel strategies are proposed to build the easy and hard parts of the framework to enable the learning process. experimental results in both full and low-resource scenarios demonstrate the effectiveness of our framework and its superiority over one-stage learning methods.