Type-enriched Hierarchical Contrastive Strategy for Fine-Grained Entity Typing (2022)

Abstract: Fine-grained entity typing (FET) aims to deduce specific semantic types of the entity mentions in the text. Modern methods for FET mainly focus on learning what a certain type looks like. And few works directly model the type differences, that is, let models know the extent that which one type is different from others. To alleviate this problem, we propose a type-enriched hierarchical contrastive strategy for FET. Our method can directly model the differences between hierarchical types and improve the ability to distinguish multi-grained similar types. On the one hand, we embed type into entity contexts to make type information directly perceptible. On the other hand, we design a constrained contrastive strategy on the hierarchical structure to directly model the type differences, which can simultaneously perceive the distinguishability between types at different granularity. Experimental results on three benchmarks, BBN, OntoNotes, and FIGER show that our method achieves significant performance on FET by effectively modeling type differences.

## introduction entity typing is a fundamental research problem in natural language processing (nlp), which aims to deduce the semantic types of the entity mentions in text. with the deepening of text understanding, the type sets of entities become more refined and ranging in from dozens @xcite to hundreds @xcite or thousands @xcite . therefore, fine-grained entity typing (fet) has gained more attention, which focuses on assigning more specific types to entities. for sentence in figure 1 , a fet system needs to assign a coarse-grained type "/person" and a fine-grained type "/person/actor" to the entity "vivien leigh". the inferred finegrained types could provide more specific prior knowledge for downstream nlp tasks, such as question answering @xcite and entity linking @xcite . considering the partial ontology of the figer dataset @xcite in figure 1 , finegrained entity types are often linked together in a hierarchical taxonomy, which makes the type boundaries increasingly blurred, especially for subtypes under the same coarse type. as shown in figure 1 , the fine-grained types "coach", "athlete" and "actor", all of which fall into the coarse-grained type "person", are less differentiated. in order to identify fine-grained types, prior work has concentrated on excavating more informative representations of types or entities, which benefiting from hand-crafted features @xcite , external resources @xcite or external pre-trained task @xcite . most of them focus on learning what a certain type looks like, but few works have gone further to directly model the differences between types, that is, let models know the extent that one type is different from others, which is more effective to distinguish among similar fine-grained types. how to directly model the type differences? we argue that there are two key points, entity type awareness, which refers to directly perceiving what type of entity is in the sentence, and type differences measure, which refers to modeling how different the perceived type is from other types. for the first point, the intuitive idea is to expose the types directly, leading to a direct focus on what type the context represents. for the second point, heuristically, direct is effective, i.e., directly modeling which contexts represent the same types and which are different is the most efficient way to measure differences. to this end, we propose a type-enriched hierarchical contrastive stratey (picot) for fine-grained entity typing. specifically, for entity type awareness with limited annotated data, inspired by prompt learning in entity typing @xcite , picot embeds the entity types in contexts via prompts to build type-rich expressions that guide the learning of correct types. additionally, for type differences measure, picot takes a constrained contrastive strategy on hierarchical taxonomy to directly model the type differences from type-rich expressions. concretely, as shown in figure 1 , picot is only concerned with the finegrained types under the same coarse-grained type to learn the differences between fine-grained types. similarly, picot is not concerned with what the fine-grained types are when distinguishing dissimilarities between coarse-grained types. methodologically, picot learns the type differences at different granularity through type-rich expressions by limiting the scope of attention to types. moreover, to further show models what a particular type is, we introduce a small number of type descriptions that directly expose richer type knowledge. in experiments, we evaluate our model on three benchmarks. first, we concern with the standard evaluations and show that our model achieves the state-of-the-art performance on fet. then we estimate the main components of picot. finally, we do a visual analysis of the effectiveness of the type differentiation of picot. in summary, the contributions are as follows: ## related work entity typing named entity recognition @xcite and entity typing @xcite are fundamental research problems in nlp. recently researchers pay more attention to fine-grained entity typing (fet) and ultra-fine entity typing (ufet) @xcite , which predict specific fine or ultra-fine types for given entities. to do so, obtaining more labeled data is the first research perspective, represented by distant supervision @xcite . with these, some researchers had focused on how to reduce noises in automatically labeled data @xcite @xcite @xcite @xcite . additionally, another key challenge is how to deal with hierarchical ontology. most prior works regarded the hierarchical typing problem as a multi-label classification task and incorporated the hierarchical structure in different ways @xcite @xcite @xcite . some works attempted to mine more label information or better label representation. @xcite enhanced the label representation by sharing parameters; lópez and strube (2020) embed types into a high-dimension; @xcite introduced associated labels to enhance the label representation; @xcite ; @xcite exploited co-occurrence structures and latent label representation; additionally, several novel textual representations were applied to obtain richer entity contextual information, such as prompt based architecture @xcite and box embeddings framework @xcite . moreover, fet and ufet suffer from an obvious issue of the unseen types due to the lack of annotated data. therefore, a variety of paradigms were be studied to alleviate this issue @xcite @xcite @xcite . moreover, some works further drew on different large-scale external data or knowledge to understand entity types @xcite @xcite . in summary, few prior works focus on directly modeling type differences. therefore, this paper tries to let models know that one type is different from others without large-scale external resources. see appendix b for more details. contrastive learning contrastive learning aims to further improve the model's ability to distinguish positive and negative examples, and has been a popular method for representation learning on computer vision tasks @xcite @xcite . recently, some researches have applied contrastive learning to natural language understanding tasks, aiming to obtain better text representations or to distinguish similar labels, such as the event causality identifier @xcite , the contrastive self-supervised encoder (fang and xie, 2020), the supporting clustering framework @xcite , the abstractive summarization framework @xcite , the contrastive fine-tuning paradigm of pre-trained language for fine-grained text classification @xcite , and so on. in this paper, we propose a constrained contrastive framework to directly model the hierarchical type differences. prompt learning prompt learning aims to leverage language prompts as contexts, and downstream tasks can be expressed as some cloze-style objectives similar to those pre-training objectives. recently, a series of hand-crafted prompts have been widely used in natural language understanding @xcite @xcite @xcite . moreover, to avoid expensive prompt design, automatic prompt has also been explored @xcite @xcite , and some continuous prompts have also been proposed @xcite . in this paper, we embed prompts to directly expose types in the entity contexts. bertenc sca-θ bertenc ric-θ predictor δ [cls] [ent] sca [ent] ric original expression & labels type-scarce expression type-rich expression sample constrained contrastive strategy annotated data gradient update gradient update gradient update coarse-grained contrastive space fine-grained contrastive space labels figure 2: the framework of picot for fet (sec. 4). ## problem formulation the input of fine-grained entity typing (fet) is a dataset @xmath0, x 2 , ..., x n } with n sentences, a pre-defined hierarchical type ontology y, and each sentence x contains a marked entity e. a fet system is required to assign corresponding types to the given marked entity. methodologically, for each input sequence w @xmath1, w 2 n , ..., w t n } of the sentence x n , fet aims to predict the correct multigrained types y @xmath2, y 2 n , ..., y m n } ∈ y of the marked entity e @xmath3, ..., w r n }. for example in figure 1 , the correct type set of "vivien leigh" is {/person, /person/actor}, which contains a coarse-grained type "/person" and a fine-grained type "/person/actor". ## methodology as shown in figure 2 , there are two key stages of picot for fine-grained entity typing. fine-grained label：/person/actor he described his portrait of actress vivien leigh as lit by a top spotlight diffused by tracing paper. [cls] ...vivien leigh...[ent] vivien leigh is a [mask] [sep] [cls] ...vivien leigh...[ent] vivien leigh is an actor [sep] coarse-grained label：/person original expression and labels type-scarce expression type-rich expression [cls] e vir person who can perform [ent] e vir is an actor [sep] actor's description: person who can perform conceptnet original expression & labels labels impart the type knowledge from type-rich expressions to predictor. ## conclusion we propose a type-enriched hierarchical contrastive strategy for fine-grained entity typing. our method can directly model the differences between hierarchical types and improve the ability to distinguish multi-grained similar types. first, we embed types into entity contexts to make type information directly perceptible. then we design a constrained contrastive strategy on hierarchical taxonomy to directly model type differences at different granularities simultaneously. experimental results on three benchmarks show that picot can achieve state-of-the-art performance on fet with limited annotated data. performance. from the results, we can see that placing "[ent]" between the entity context and the type prompt allows for more efficient migration and reception of type knowledge.