Modeling Label Correlations for Ultra-Fine Entity Typing with Neural Pairwise Conditional Random Field (2022)

Abstract: Ultra-fine entity typing (UFET) aims to predict a wide range of type phrases that correctly describe the categories of a given entity mention in a sentence. Most recent works infer each entity type independently, ignoring the correlations between types, e.g., when an entity is inferred as a president , it should also be a politician and a leader . To this end, we use an undirected graphical model called pairwise conditional random field (PCRF) to formulate the UFET problem, in which the type variables are not only unarily influenced by the input but also pairwisely relate to all the other type variables. We use various modern backbones for entity typing to compute unary potentials, and derive pairwise potentials from type phrase representations that both capture prior semantic information and facilitate accelerated inference. We use mean-field variational inference for efficient type inference on very large type sets and unfold it as a neural network module to enable end-to-end training. Experiments on UFET show that the Neural-PCRF consistently outperforms its backbones with little cost and results in a competitive performance against cross-encoder based SOTA while being thousands of times faster. We also find Neural-PCRF effective on a widely used fine-grained entity typing dataset with a smaller type set. We pack Neural-PCRF as a network module that can be plugged onto multi-label type classifiers with ease and release it in .

## introduction entity typing assigns semantic types to entities mentioned in the text. the extracted type information has a wide range of applications. it acts as a primitive for information extraction @xcite and spoken language understanding @xcite , and assists in more complicated tasks such as machine reading comprehension @xcite and semantic parsing @xcite . during its long history of development, the granularity of the type set for entity typing and recognition changes from coarse (sized less than 20) @xcite , to fine (sized around 100) @xcite @xcite , to ultra-fine and free-form (sized 10k) @xcite . the expansion of the type set reveals the diversity of real-world entity categories and the importance of a finer-grained understanding of entities in applications @xcite . the increasing number of types results in difficulties in predicting correct types, so a better understanding of the entity types and their correlations is needed. most previous works solve an n -type entity typing problem as n independent binary classifications @xcite @xcite @xcite @xcite . however, types are highly correlated and hence they should be predicted in a joint manner. as an example, when 'joe biden' is inferred as a president, it should also be inferred as a politician, but not a science fiction. type correlation is partially specified in fine-grained entity typing (fet) datasets by a two or three-level type hierarchy, and is commonly utilized by a hierarchy-aware objective function @xcite @xcite . however, type hierarchies cannot encode type relationships beyond strict containment, such as similarity and mutual exclusion @xcite , and could be noisily defined @xcite or even unavailable (in ultra-fine entity typing (ufet)). many recent works handle these problems by embedding types and mentions into special spaces such as the hyperbolic space @xcite or box space @xcite that can be trained to latently encode type correlations without a hierarchy. although these methods are expressive for modeling type correlations, they are constrained by these special spaces and thus incapable of being combined with modern entity typing backbones such as prompt learning @xcite and cross-encoder @xcite , and cannot integrate prior type semantics. in this paper, we present an efficient method that expressively models type correlations while being backbone-agnostic. we formulate the ufet and fet problems under a classical undirected graphical model (ugm) @xcite called pairwise conditional random field (pcrf) @xcite . in pcrf, types are binary variables that are not only unarily influenced by the input but also pairwisely relate to all the other type variables. we formulate the unary potentials using the type logits provided by any modern backbone such as prompt learning. to compose the pairwise potentials sized op4n 2 q (n is the number of types, 10k for ufet), we use matrix decomposition, which is efficient and able to utilize prior type semantic knowledge from word embeddings @xcite . exact inference on such a large and dense ugm is intractable, and therefore we apply mean-field variational inference (mfvi) for approximate inference. inspired by @xcite , we unfold the mfvi as a recurrent neural network module and connect it with the unary potential backbone to enable end-to-end training and inference. we call our method neural-pcrf (npcrf). experiments on ufet show that npcrf consistently outperforms the backbones with negligible additional cost, and results in a strong performance against cross-encoder based sota models while being thousands of times faster. we also found npcrf effective on a widely used fine-grained entity typing dataset with a smaller type set. we pack npcrf as a network module that can be plugged onto multi-label type classifiers with ease and release it in github.com/ modelscope/adaseq/examples/npcrf. ## conclusion and future work we propose npcrf, a method that efficiently models type correlation for ultra-fine and finegrained entity typing, and is applicable to various entity typing backbones. in npcrf, the unary potential is formulated as the type logits of modern ufet backbones, the pairwise potentials are derived from type phrase representations that both capture prior semantic information and facilitate accelerated inference. we unfold mean-field variational inference of npcrf as a neural network for end-to-end training and inference. we find our method consistently outperforms its backbone, and reach competitive performance against very recent baselines on ufet and fet. npcrf is efficient and require low additional computation costs. for future work, modeling higher-order label correlations, injecting prior knowledge into pairwise potentials, and extending npcrf to other tasks are worth exploring.