Ultra-Fine Entity Typing with Prior Knowledge about Labels: A Simple Clustering Based Strategy (2023)

Abstract: Ultra-fine entity typing (UFET) is the task of inferring the semantic types from a large set of fine-grained candidates that apply to a given entity mention. This task is especially challenging because we only have a small number of training examples for many types, even with distant supervision strategies. State-of-the-art models, therefore, have to rely on prior knowledge about the type labels in some way. In this paper, we show that the performance of existing methods can be improved using a simple technique: we use pre-trained label embeddings to cluster the labels into semantic domains and then treat these domains as additional types. We show that this strategy consistently leads to improved results as long as high-quality label embeddings are used. Furthermore, we use the label clusters as part of a simple post-processing technique, which results in further performance gains. Both strategies treat the UFET model as a black box and can thus straightforwardly be used to improve a wide range of existing models.

## introduction entity typing is the task of inferring the semantic type(s) of an entity that is mentioned in some sentence. while only a few coarse types were traditionally considered, such as person and organisation, the focus has shifted to increasingly finergrained types. consider the following example: foot_1 "the company said its production upgrades would also have a "short-term impact" on the delivery of vaccines to the uk". from this sentence, we cannot only infer that "the company" refers to an organisation but also that it refers to a pharmaceutical company. to allow for more informative predictions, @xcite proposed the task of fine-grained entity typing. however, the 122 entity types that were considered in their work are still too coarse for many applications. for instance, the most specific type from their taxonomy that applies to the example above is company. a further refinement of entity types was proposed by @xcite , who considered a total of 10, 331 semantic types and introduced the term ultra-fine entity typing (ufet). for instance, the ultra-fine type pharmaceutical is available for the above example. ufet is essentially a multi-label text classification problem. the main challenge stems from the fact that we only have a few labelled training examples for most of the semantic types, and sometimes even none at all. the solution proposed by @xcite is based on two distant supervision strategies. their first strategy is to use mentions that link to a wikipedia page and use the definition on that page to infer semantic types. their second strategy relies on nominal expressions which include their semantic type as a head word (e.g. president joe biden). more recent work has improved on these strategies using denoising techniques @xcite . even with distant supervision, the number of training examples remains prohibitively small for many types. state-of-the-art methods, therefore rely on prior knowledge about the labels themselves. for instance, @xcite use the representation of the labels in the decoder of bert to initialise the weights of the label classifiers, while @xcite use a fine-tuned bert @xcite model to map type labels onto prototypes. however, the token embeddings from bert are known to be sub-optimal as pre-trained representations of word meaning @xcite , whereas fine-tuning a bert encoder may lead to overfitting. in this paper, we therefore pursue a different strategy to inject prior knowledge about the labels into ufet models. we use pretrained embeddings to cluster the labels into groups, which we call domains. rather than changing the ufet model itself, we simply treat these domains as additional semantic types. in other words, given a training example with labels l 1 , ..., l n , we now also add the synthetic labels c 1 , ..., c n , where c i is the cluster to which l i belongs. this strategy has the advantage that arbitrary label embeddings can be used and that the ufet model itself does not need to be modified. despite its simplicity, we find this strategy to perform remarkably well, provided that high-quality label embeddings are used. the best results are obtained with recent bertbased strategies for learning word vectors @xcite . in contrast, the representations from classical word embedding models @xcite are too noisy to have a positive impact, with conceptnet numberbatch @xcite being a notable exception. apart from using pre-trained label embeddings, another line of work has focused on modelling label correlations @xcite . essentially, the idea is to train a probabilistic model on top of the predictions of a base ufet classifier. while meaningful improvements are possible in this way, the extent to which label correlations can be learned is inherently limited due to the sparsity of the training data. as an alternative, we focus on two simple strategies to improve the predictions of the base ufet model. both strategies directly rely on prior knowledge about the labels and are thus well-suited for labels that are rare in the training set. the first strategy is aimed at inferring missing labels: if the domain label c i is predicted but none of the labels that belong to that domain, we add the most likely label from c i to the predicted set. the second strategy is aimed at removing conflicting labels: if an entity is predicted to have multiple labels from the same cluster, then we use a pre-trained classifier to check whether these labels are mutually inconsistent, and if so, we remove the labels with the lowest confidence degree. the contributions of this paper can be summarised as follows. we use pre-trained label embeddings to group the given set of type labels into semantic domains. these domains are then used to improve a base ufet model in different ways: label dependencies in fine-grained entity typing, there are usually various dependencies between the labels. for instance, in the case of figer @xcite and ontonotes @xcite , the labels are organised into a tree, which models can exploit @xcite @xcite @xcite . in the case of ufet, however, the label set is essentially flat @xcite , which means that label dependencies have to be learned from the training data and/or external sources. @xcite use a graph neural network (gnn) @xcite to take label correlations into account. the edges of the graph, which indicate potential label interactions, are selected based on (i) label co-occurrence in the training data and (ii) the similarity of the glove embeddings of the label names. the model thus relies on the assumption that labels with similar embeddings are likely to apply to the same entities, which is too strong: the embeddings of mutually exclusive labels, such as teacher and student, can also be similar. @xcite rely on two strate-gies for modelling label correlation. one strategy is based on directly observed label correlations. for the other strategy, they replace the entity by the [mask] token and obtain predictions from bert for this token. then they model correlations between these predictions and the actual labels. @xcite also use a gnn to model label interactions. the nodes of their graphs correspond to both entity types and keywords from the given sentences. @xcite model label correlations using a probabilistic graphical model. different from the aforementioned works, we exploit prior knowledge about label dependencies in a way which is decoupled from the entity typing model, by adding synthetic labels to training examples and by post-processing the predictions. this makes our method more transparent (i.e. we know which dependencies are used and why) and modular (i.e. our method can directly be applied to existing entity typing models). while the lack of integration with the entity typing model may seem overly simplistic, note that for most entity types we only have a few training examples. it is difficult to learn meaningful label dependencies from such data without also learning spurious correlations. label embeddings recent ufet models typically rely on some kind of prior knowledge about the labels. as already mentioned, @xcite rely on glove embeddings to construct a label graph. @xcite initialise the scoring function for each label based on that label's representation in the decoder of the language model (see section 3). rather than using pre-trained embeddings, @xcite encode the labels using a language model, which is fine-tuned together with the entity mention encoder. a similar approach was used by @xcite in the context of few-shot entity typing. in these approaches, the label embeddings are directly used as label prototypes for the classification model, which could make it harder to distinguish between similar but mutually exclusive labels. in a broader context, label embeddings have also been used for few-shot text @xcite @xcite and image @xcite @xcite classification. clusters of label embeddings have been used in the context of extreme multi-label text classification @xcite @xcite . however, in this setting, clusters are used to make the problem tractable, rather than being a strategy for injecting prior knowledge. the representation of each label is typically derived from the sparse features of the documents that label, and the focus is on learning balanced clusters. ## problem setting in ultra-fine entity typing, the input consists of a sentence in which a particular entity mention is highlighted. the aim is to assign all the semantic types that apply to that entity, given a large set l of around 10,000 possible types @xcite . this task can be treated as a standard multilabel classification problem. in particular, let ϕ be an encoder which maps an input sentence s with entity mention m to a corresponding embedding ϕ(s, m) ∈ r n . together with the encoder ϕ, we also learn a scoring function f l : r @xmath0, 1] for each label @xmath1. the probability that label l applies to the entity mentioned in the input is then estimated as f l (ϕ(s, m)). as a representative recent approach, we consider the denoisefet model from @xcite , which uses a bert encoder with the following input: here s is the input sentence, m is the span corresponding to the entity mention, and [p 1 ], [p 2 ], [p 3 ] are trainable tokens. these trainable tokens intuitively allow the model to learn a soft prompt, and were found to improve performance by @xcite . the encoding ϕ(s, m) is then taken to be the embedding of @xcite in the final layer of the bert model. the scoring functions f l take the form f l (x) = σ(x • y l + b l ), where σ is the sigmoid function, y @xmath2. the parameters y l and b l are initialised according to the corresponding weights in the decoder of the masked language model (where the average across all tokens is used if l consists of several tokens). ## adding domain labels we now describe our main strategy for improving ufet models based on pre-trained label embeddings. broadly speaking, the label embeddings provide us with prior knowledge about which labels are similar. crucially, however, the fact that two labels have similar embeddings does not necessarily mean that the occurrence of these labels is positively correlated. labels can be similar because they denote categories with similar meanings (e.g. student and learner), but similar labels can also denote mutually exclusive categories from the same domain (e.g. student and teacher, or ambulance and fire truck). for this reason, we do not require that the scoring functions (or prototypes) for similar labels should always be similar. instead, we propose a strategy based on clusters of labels. we use an off-the-shelf clustering method to cluster labels based on their pre-trained embedding. each cluster intuitively represents a domain. we associate a synthetic label with each of the domains and add these synthetic labels to the training examples. for instance, consider the following cluster: l @xmath3, fire engine, air ambulance, ambulance, police car} (1) whenever a training example has any of these labels, we add "cluster i" as an additional label. this tells the model that examples about different types of emergency vehicles all have something in common. 3 if a linear scoring function is used, as in denoisefet, examples involving emergency vehicles are thus encouraged to be linearly separated from other examples. this helps the model to uncover features that are common to emergency vehicles, while not requiring that prototypes of different types of emergency vehicles be similar. ## post-processing predictions domains can also be used for improving the predictions of an ufet model post hoc. we focus on two simple strategies that exclusively rely on prior knowledge about the label dependencies. in particular, we do not attempt to learn about label dependencies from the training data. we assume that dependencies between frequent labels are already modelled by the base ufet model, whereas for rare labels, relying on the training data may lead to overfitting. this also has the advantage that we can continue to treat the base ufet model as a black box. we only require that the base ufet model can provide us with a confidence score conf(l; s, m) for any label @xmath4, m). ## experimental results we evaluate our approach on the ultra-fine entity typing (ufet) benchmark from @xcite . although we expect our strategies to be most effective for ultra-fine entity types, we also carry out an evaluation on two standard fine-grained entity typing benchmarks: ontonotes foot_4 @xcite and figer foot_5 @xcite . ## conclusions we have analysed the effectiveness of two types of external knowledge for ultra-fine entity typing: pretrained label embeddings and a conceptual neighbourhood classifier. the strategies that we considered are straightforward to implement and can be added to most existing entity typing models. one of the most effective strategies simply requires adding synthetic labels, corresponding to clusters of semantic types, to the training examples. we also discussed two post-processing strategies which can further improve the results. despite their simplicity, the proposed strategies were found to consistently improve the performance of the base ufet model. the nature of the pre-trained label embeddings played an important role in the performance of the overall method. an interesting avenue for future work would be to design strategies for identifying label domains which are specifically designed for the ufet setting.