Generative Entity Typing with Curriculum Learning (2022)

Abstract: Entity typing aims to assign types to the entity mentions in given texts. The traditional classification-based entity typing paradigm has two unignorable drawbacks: 1) it fails to assign an entity to the types beyond the predefined type set, and 2) it can hardly handle few-shot and zero-shot situations where many long-tail types only have few or even no training instances. To overcome these drawbacks, we propose a novel generative entity typing (GET) paradigm: given a text with an entity mention, the multiple types for the role that the entity plays in the text are generated with a pre-trained language model (PLM). However, PLMs tend to generate coarse-grained types after fine-tuning upon the entity typing dataset. In addition, only the heterogeneous training data consisting of a small portion of human-annotated data and a large portion of auto-generated but low-quality data are provided for model training. To tackle these problems, we employ curriculum learning (CL) to train our GET model on heterogeneous data, where the curriculum could be self-adjusted with the self-paced learning according to its comprehension of the type granularity and data heterogeneity. Our extensive experiments upon the datasets of different languages and downstream tasks justify the superiority of our GET model over the state-of-the-art entity typing models. The code has been released on https://github.com/siyuyuan/GET.

## introduction entity typing aims to assign types to mentions of entities from a predefined type set, which enables machines to better understand natural languages and benefit many downstream tasks, such as entity linking @xcite ) and text classification @xcite . traditional entity typing approaches follow the classification paradigm to classify (assign) the entity into a predefined set of types, which have the following two unignorable drawbacks. in the early 1980s , p & g tried to launch here a concentrated detergent under the ariel brand name that it markets in europe. input text predefined type set substance product continent … company entity: p & g generation classification detergent company* detergent manufacturer* company company (*) means the generated types are out of predefined type set 1) closed type set: the classification-based approaches fail to assign the entity to the types out of the predefined set. 2) few-shot dilemma for long-tail types: although fine-grained entity typing (fet) and ultra-fine entity typing approaches can classify entities into fine-grained types, they can hardly handle few-shot and zero-shot issues. in fact, there are many long-tail types only having few or even no training instances in the datasets. for example, more than 80% types have less than 5 instances and 25% types even never appear in the training data from the ultra-fine dataset @xcite . to address these drawbacks, in this paper, we propose a novel generative entity typing (get) paradigm: given a text with an entity mention, the multiple types for the role that the entity plays in the text are generated by a pre-trained language model (plm). compared to traditional classification-based entity typing methods, plmbased get has two advantages. first, instead of a predefined closed type set, plms can generate more open types for entity mentions due to their strong generation capabilities. for example, in figure 1 , fine-grained types such as "large detergent company" and "large detergent manufacturer" can be generated by plms for entity p&g, which contain richer semantics but are seldom included by a predefined type set. second, plms are capable of conceptual reasoning and handling the few-shot and zero-shot dilemma @xcite , since massive knowledge has been learned during their pre-training. however, it is nontrivial to realize plm-based get due to the following challenges: 1) entity typing usually requires generating fine-grained types with more semantics, which are more beneficial to downstream tasks. however, plms are biased to generate high-frequency vocabulary in the corpus due to their primary learning principle based on statistical associations. as a result, a typical plm tends to generate high-frequent but coarse-grained types even if we carefully finetune the plm on the fine-grained entity typing dataset (refer to figure 5 in section 4). therefore, how to guide a plm to generate high-quality and fine-grained types for entities is crucial. 2) it is costly for humans to annotate a great number of samples with fine-grained types. therefore, most existing works adopt heterogeneous data consisting of a small portion (less than 10%) of human-annotated data and a large portion (more than 90%) of auto-generated lowquality data (e.g., by distant supervision), which greatly hurts the performance of entity typing models @xcite . how to train a plm to generate desirable types on these low-quality heterogeneous data is also challenging. the difficulty of using plms to generate highquality fine-grained types based on the low-quality heterogeneous training data motivates us to leverage the idea from curriculum learning (cl) @xcite , which better learns heterogeneous data by ordering the training samples based on their quality and difficulty @xcite . in this paper, we propose a cl-based strategy to train our get model. specifically, we first define a fixed curriculum instruction and partition the training data into several subsets according to the granularity and heterogeneity of samples for model training. based on the curriculum instruction, cl can control the order of using these training subsets from coarse-grained and lower-quality ones to finegrained and higher-quality ones. however, a fixed curriculum ignores the feedback from the training process. thus, we combine the predetermined curriculum with self-paced learning (spl) @xcite , which can enforce the model dynamically self-adjusting to the actual learning order according to the training loss. in this way, our cl-based get model can make the learning process move towards a better global optimum upon the heterogeneous data to generate high-quality and fine-grained types. our contributions in this paper are summarized as follows: ## related work classification-based entity typing the traditional classification-based entity typing methods can be categorized into three classes. 1) coarsegrained entity typing methods @xcite assign mentions to a small set of coarse types; 2) finegrained entity typing (fet) methods @xcite classify mentions into more diverse and semantically richer ontologies; 3) ultra-fine entity typing methods @xcite @xcite use a large open type vocabulary to predict a set of natural-language phrases as entity types based on texts. however, fet and ultra-fine entity typing methods hardly perform satisfactorily due to the huge predefined type set. they also hardly handle few-shot and zero-shot issues. comparatively, our get model can generate high-quality multigranularity types even beyond the predefined set for the given entity mentions. concept acquisition concept acquisition is very related to entity typing which also aims to obtain the types for the given entities, since entity types are often recognized as concepts. concept acquisition can be categorized into the extraction-based or generation-based scheme. the extraction scheme cannot acquire concepts not existing in the given text @xcite . the existing approaches of concept generation @xcite focus on utilizing the existing concept taxonomy or knowledge bases to generate concepts but neglect to utilize the large corpus. our get model can also achieve text-based concept generation. curriculum learning according to the curriculum learning (cl) paradigm, a model is first trained with the easier subsets or subtasks, and then the training difficulty is gradually increased @xcite to improve model performance in difficult target tasks, such as domain adaption @xcite and training generalization @xcite . the existing cl methods can be divided into predefined cl (pcl) @xcite and automatic cl (acl) @xcite . pcl divides the training data by the difficulty level with prior knowledge, while acl, such as self-paced learning (spl), measures the difficulty according to its losses or other models. ## methodology in this section, we first formalize our task in this paper and overview the framework of our get model. then, we introduce the details of model implementation. ## experiments in this section, we verify the advantages of our get model over the classification-based entity typing models through our experiments. we also explore the role of cl in guiding plm-based type generation with different language data. we further display the effectiveness of our generated entity types in two downstream tasks. ## conclusion in this paper, we propose a novel generative paradigm for entity typing, which employs a generative plm trained with curriculum learning. compared with the traditional classification-based entity typing methods, our generative entity typing (get) model can generate new types beyond the predefined type set for given entity mentions. our extensive experiments on several benchmark datasets justify that the curriculum learning with spl and the prior knowledge of type length and subset order help our model generate more high-quality finegrained types.