Prompt-learning for Fine-grained Entity Typing (2022)

Abstract: As an effective approach to adapting pre-trained language models (PLMs) for specific tasks, prompt-learning has recently attracted much attention from researchers. By using cloze-style language prompts to stimulate the versatile knowledge of PLMs, prompt-learning can achieve promising results on a series of NLP tasks, such as natural language inference, sentiment classification, and knowledge probing. In this work, we investigate the application of prompt-learning on fine-grained entity typing in fully supervised, few-shot, and zero-shot scenarios. We first develop a simple and effective prompt-learning pipeline by constructing entity-oriented verbalizers and templates and conducting masked language modeling. Further, to tackle the zero-shot regime, we propose a self-supervised strategy that carries out distribution-level optimization in prompt-learning to automatically summarize the information of entity types. Extensive experiments on four fine-grained entity typing benchmarks under fully supervised, few-shot, and zero-shot settings show the effectiveness of the prompt-learning paradigm and further make a powerful alternative to vanilla fine-tuning.

## introduction in recent years, pre-trained language models (plms) have been widely explored and become a key instrument for natural language understanding @xcite and generation @xcite . by applying self-supervised learning on large-scale unlabeled corpora, plms can capture rich lexical @xcite , syntactic @xcite , and factual knowledge @xcite that well benefits downstream nlp tasks. considering the versatile knowledge contained in plms, many efforts of researchers have been devoted to stimulating taskspecific knowledge in plms and adapting such knowledge to downstream nlp tasks. and finetuning with extra classifiers has been one typical solution for adapting plms to specific tasks in nlp tasks @xcite . some recent efforts on probing knowledge of plms show that, by writing some natural language prompts, we can induce plms to complete factual knowledge @xcite . gpt-3 further utilizes the information provided by prompts to conduct few-shot learning and achieves awesome results @xcite . inspired by this, promptlearning has been introduced. as shown in figure 1 , in prompt-learning, downstream tasks are formalized as equivalent cloze-style tasks, and plms are asked to handle these tasks instead of original downstream tasks. compared with vanilla fine-tuning methods, prompt-learning does not require extra neural layers and intuitively bridges the objective form gap between pre-training and fine-tuning. sufficient empirical analysis shows that, either for manually picking hand-crafted prompts @xcite or automatically building auto-generated prompts @xcite , taking prompts for tuning models is surprisingly effective for the knowledge stimulation and model adaptation of plms. intuitively, prompt-learning is applicable to finegrained entity typing, which aims at classifying marked entities from input sequences into specific types in a pre-defined label set. we discuss this topic with a motivating example, "he is from new york". by adding a prompt with a masking token @xcite , the sentence becomes "he is from new york. in this sentence, new york is [mask]". due to the wealth of knowledge acquired during pretraining, plms can compute a probability distribution over the vocabulary at the masked position, and a relatively higher probability with the word "city" than the word "person". in other words, with simple prompts, the abstract entity attributes contained in plms can be efficiently exploited, which is meaningful for downstream entity-related tasks. we comprehensively explore the application of prompt-learning to fine-grained entity typing in fully supervised, few-shot and zero-shot settings. particularly, we first introduce a naive pipeline, where we construct entity-oriented prompts and formalize fine-grained entity typing as a cloze-style task. this simple pipeline yields promising results in our experiments, especially when supervision is insufficient. it is worth noting that fewshot and zero-shot fine-grained entity typing is then, to tackle the zero-shot scenario where no explicit supervision exists in training, we develop a self-supervised strategy under our prompt-learning pipeline. our self-supervised strategy attempts to automatically summarize entity types by optimizing the similarity of the predicted probability distributions of paired examples in prompt-learning. four popular benchmarks are used for our experiments, including few-nerd @xcite , ontonotes @xcite , bbn @xcite , and open entity @xcite . all these datasets have a complex type hierarchy consisting of rich entity types, requiring models to have good capabilities of entity attribute detection. empirically, our method yields significant improvements on these benchmark datasets, especially under the zero-shot and few-shot settings. we also make an analysis and point out both the superiority and bottleneck of prompt-learning in fine-grained entity typing, which may advance further efforts to extract entity attributes using plms foot_0 . ## related work after a series of effective plms like gpt @xcite and bert @xcite , finetuned plms have demonstrated their effectiveness on various important nlp tasks @xcite @xcite . despite the success of fine-tuning plms, the huge objective form gap between pre-training and fine-tuning still hinders the full use of per-trained knowledge for downstream tasks @xcite @xcite . to this end, prompt-learning has been proposed. the seminal work that stimulates the development of prompt-learning is the birth of @xcite , which uses hand-crafted prompts for tuning and achieves impressive performance on various tasks. a series of hand-crafted prompts have been widely explored in knowledge probing @xcite , relation classification @xcite , sentiment classification and natural language inference @xcite . to avoid labor-intensive prompt design, automatic prompt search has also been extensively explored @xcite ; @xcite ; @xcite ; @xcite to generate language phrases for prompts. recently, some continuous prompts have also been proposed @xcite , which directly use a series of learnable continuous embeddings as prompts rather than discrete language phrases. this paper aims to stimulate plms with promptlearning to capture the attribute information of entities. we take fine-grained entity typing, a crucial task in knowledge extraction to assign entity types to entity mentions @xcite , as the foothold to develop prompt-learning strategies. in fact, @xcite use hypernym extraction patterns to enhance the context and apply masked language modeling to tackle the ultra-fine entity typing problem @xcite with free-form labels, which shares a similar intuition with promptlearning. in our work, we mainly emphasize using prompt-learning to extract entity types that have been pre-defined in low-data scenarios. ## background in this section, we first give a problem definition of the entity typing task ( § 3.1), followed by an introduction of conventional vanilla fine-tuning and prompt-based tuning ( § 3.2) with plms. ## self-supervised prompt-learning with prompt-learning, the answer is yes, because in the pre-training stage, the contexts of entities have already implied the corresponding type information, which provides an advantageous initialization point for the prompt-learning paradigm. for example, in the input sentence with the t 3 (•) template: "steve jobs found apple. in this sentence, steve jobs is a [mask] ". in our observations, the probability of plms predicting person at the masked position will be significantly higher than the probability of location. and if we make reasonable use of this superior initialization point, it is possible for plms to automatically summarize the type information, and finally extract the correct entity type. ## experiments we comprehensively assess the effectiveness of our methods, we use ft and plet to denote the standard fine-tuning method and method in § 4 and use plet (s) to denote the self-supervised method in § 5. plet(t • ) denotes the prompt-based method with corresponding template. we evaluate our methods on four widely used entity typing datasets, including few-nerd @xcite , ontonotes @xcite , bbn @xcite , and ope-nentity @xcite . our experimental set-tings contain fully supervised, few-shot, and zeroshot scenarios. the backbone plms used in our experiments include bert @xcite , t5 @xcite and gpt-2 @xcite . ## conclusion this work investigates the application of promptlearning on fine-grained entity typing in in fully supervised, few-shot and zero-shot scenarios. we first investigate a simple and effective promptlearning pipeline that could be used to extract entity types with both sufficient and insufficient supervision. furthermore, to handle the zero-shot setting, we propose a self-supervised prompt-learning approach that automatically learns and summarizes entity types based on unlabeled corpora and a predefined label schema, which utilizes prompts to take advantage of prior knowledge distributed in plms, and could learn pre-defined type information without overfitting by performing distributionlevel optimization. the experimental results verify that prompt-learning is a strong alternative to fine-tuning for fine-grained entity typing, we comprehensively explore and analyze various attributes under this scenario. • labelgcn @xcite is a graphbased approach for ultra-fine grained entity typing. a gcn layer is used to encode the global label co-occurrence and statistics and word-level similarities. note that the test data and the evaluation is identical to our method, but some methods use the weakly supervised data provided by openentity and ours does not use any other data rather than the 1,998 examples in the training set. the results are shown in