E n C ore: Fine-Grained Entity Typing by Pre-Training Entity Encoders on Coreference Chains (2024)

Abstract: Entity typing is the task of assigning semantic types to the entities that are mentioned in a text. In the case of fine-grained entity typing (FET), a large set of candidate type labels is considered. Since obtaining sufficient amounts of manual annotations is then prohibitively expensive, FET models are typically trained using distant supervision. In this paper, we propose to improve on this process by pre-training an entity encoder such that embeddings of coreferring entities are more similar to each other than to the embeddings of other entities. The main problem with this strategy, which helps to explain why it has not previously been considered, is that predicted coreference links are often too noisy. We show that this problem can be addressed by using a simple trick: we only consider coreference links that are predicted by two different off-the-shelf systems. With this prudent use of coreference links, our pre-training strategy allows us to improve the state-of-the-art in benchmarks on fine-grained entity typing, as well as traditional entity extraction.

## introduction entity typing is a fundamental task in natural language processing (nlp), with important applications to entity linking @xcite and relation extraction @xcite , among others. in recent years, the main focus has been on fine-grained entity typing @xcite , where around 100 different entity types are considered, or even ultra-fine entity typing @xcite , where around 10000 types are considered. a key challenge then consists in compiling enough training data. this is particularly problematic because the distribution of entity types is highly skewed, with many types occurring only rarely in text. the main strategy thus far has been to create automatically labelled training sets. for instance, @xcite relied on the fact that entity mentions in wikipedia are linked to the article of the corresponding entity, which is in turn linked to freebase @xcite . entity mentions in wikipedia can thus be linked to their freebase types without any manual effort. however, these distantly supervised training sets are still highly skewed. as a result, models trained on such datasets may concentrate more on learning to recognise the most prevalent entity types than on deriving meaningful entity representations (i.e. embeddings which accurately capture semantic types of entities). for this reason, we propose to first train a general-purpose entity encoder, which maps entity mentions to meaningful embeddings, independent of a particular label set. we can then train an entity type classifier in the usual way, using the embeddings from our encoder as input. our approach relies on a supervision signal that has thus far remained largely unexplored for entity typing: coreference chains. in particular, we train an entity encoder with contrastive learning to represent coreferring entity mentions close to each other in the embedding space. while conceptually straightforward, this training signal forces the entity encoder to identify subtle cues in the context of an entity mention, to characterise the entity at a level which is sufficiently fine-grained to distinguish it from other entities. our strategy only need access to an off-the-shelf coreference resolution system. this means that we can train the entity encoder on different genres of text and generate as much training data as is needed. figure 1 illustrates the three main steps of our approach. in the first step, an off-the-shelf coreference resolution system is applied to a large collection of stories. second, we use contrastive learning to train an entity encoder, which maps mentions from the same coreference chain to similar vectors, while mentions from different chains are mapped to dissimilar vectors. in the third step, to learn a fine-grained entity typing model, we simply train a linear classifier in the resulting embedding space for each considered entity type. an important challenge in implementing the proposed strategy is that coreference resolution systems are still far from perfect. whenever two mentions are erroneously assumed to be referring to the same entity, the entity encoder is trained on a noisy signal, which has a detrimental impact on the overall performance of the method. in our experiments, we found that the success of our strategy indeed strongly depends on the quality of the coreference resolution system that is used. in fact, our best results are obtained by using two different systems, and only keeping coreference links that are predicted by both. when adopting this strategy, our model outperforms the current state-of-the-art in three entity typing benchmarks. fine-grained entity typing most work on finegrained entity typing uses distant supervision of some kind. as already mentioned in the introduction, one strategy is to rely on wikipedia links in combination with an external knowledge base @xcite . a common problem with distantly supervised datasets is that they can be noisy: the fact that an entity has a particular type does not necessarily imply that this information is expressed in a given sentence mentioning that entity. to address this issue, several authors have proposed strategies for denoising distantly supervised datasets for entity typing @xcite @xcite . given that two sentences referring to the same entity may emphasise different elements, a similar problem can also arise in our case. for example, we might have a sentence referring to ben affleck as an actor and another referring to him as a director. as the embedding of an entity mention should capture the semantic type that is represented in the relevant sentence context, using such sentence pairs will confuse the model. we may anticipate that such instances will be rare, however, as we only take into account co-referring entity mentions that originate from the same story. another possible source of noise comes from mistakes that are made by the coreference resolution system. this effect will be analysed in section 4. pre-training entity encoders previous work has already explored a number of pre-training strategies for learning entity representations. first, methods such as spanbert @xcite focus on learning better representations of text spans. within this class of methods, strategies that rely on infonce have also been considered @xcite . while our method also uses in-fonce, the training signal is fundamentally different: the aforementioned methods focus on learning span representations, using tasks such as reconstructing the correct order of tokens in shuffled text spans. such models have not proven superior to the standard bert model for entity typing. in our experiments, we also found that modelling text spans is not essential for entity typing, as our best configuration simply uses the embedding of the head token of an entity span (see section 4.2). another line of work, which includes models such as ernie @xcite , knowbert @xcite , luke @xcite , ke-pler @xcite and k-adapter @xcite , improve lms by modelling entities as separate tokens and leveraging information from knowledge graphs. the main focus of these models is to improve the amount of factual knowledge that is captured, rather than on learning the representations of (possibly) previously unseen entities. our approach also has some similarities with the matching-the-blanks model for relation extraction @xcite . the idea of this model is to learn a label-independent relation encoder, similar to how we are learning a labelindependent entity encoder. in their case, the supervision signal comes from the idea that sentences mentioning the same pair of entities are likely to express the same relationship, hence the relation embeddings obtained from such sentences should be similar. building on this approach, a number of authors have recently used infonce to encode similar ideas @xcite @xcite . @xcite use a contrastive loss to pre-train a mention encoder for coreference resolution based on two heuristics: leveraging coreference chains to the best of our knowledge, the idea of pre-training an entity encoder based on coreference chains has not yet been considered. however, a number of authors have proposed multi-task learning frameworks in which coreference resolution and entity typing are jointly learned, along with other tasks such as relation and event extraction @xcite . surprisingly, perhaps, such approaches have failed to outperform simpler entity typing (and relation extraction) models @xcite . ## our approach in section 3.1, we first discuss the basic entity typing model that we rely on in this paper. section 3.2 subsequently describes our proposed pre-training strategy based on coreference chains. ## experimental analysis in this section, we evaluate the performance of our proposed strategy on (fine-grained) entity typing. 2 experimental setup in all our experiments, we initialise the entity encoder with a pre-trained language model. we consider bert-base-uncased foot_2 , albert-xxlarge-v1 foot_3 and roberta-large foot_4 for this purpose, as these are commonly used for entity typing. we use the gigaword corpus foot_5 as the collection of stories. this corpus consists of around 4 million news stories from four different sources. we use two state-of-the-art coreference resolution systems: the explosion ai system coreferee v1.3.1 foot_6 and the allennlp coreference model foot_7 . as explained in section 3.2, we only keep coreference links that are predicted by both of these systems. once the encoder has been pre-trained, we train an entity type classifier on the standard training set for each benchmark. we report results for two different variants of this process: one where the entity encoder is fine-tuned while training the entity type classifiers and one where the encoder is frozen. we will refer to these variants as encore and encorefrozen, respectively. we train all of the models for 25 epochs with the adamw optimizer @xcite and save the checkpoint with the best result on the validation set. the temperature τ in the contrastive loss was set to 0.05. benchmarks our central hypothesis is that the proposed pre-training task makes it possible to learn finer-grained entity representations. as such, we focus on fine-grained entity typing as our main evaluation task. we use the ontonotes @xcite and figer @xcite benchmarks. ontonotes is based on the news stories from the ontonotes 5.0 corpus foot_8 . we use the entity annotations that were introduced by @xcite , considering a total of 89 different entity types (i.e. 88 types + other). they also introduced a distantly supervised training set, consisting of 133k automatically labelled news stories. figer considers a total of 113 types (i.e. 112 types + other). the test set consists of sentences from a student newspaper from the university of washington, two local newspapers, and two specialised magazines (on photography and veterinary). along with this test set, they also provided automatically labelled wikipedia articles for training. for finegrained entity typing, we report the results in terms of macro and micro-averaged f1, following the convention for these benchmarks. we also experiment on standard entity typing, using the ace 2005 corpus foot_9 , which covers the following text genres: broadcast conversation, broadcast news, newsgroups, telephone conversations and weblogs. it differentiates between 7 entity types. for this benchmark, the entity spans are not provided. we thus need to identify entity mentions in addition to predicting the corresponding types. we treat the problem of identifying entity span as a sequence labelling problem. we follow the strategy from @xcite , but start from our pre-trained entity encoder rather than a standard lm. we summarise this strategy in appendix a. we use the standard training/development/test splits that were introduced by @xcite . following standard practice, we report the results in terms of micro-averaged f1. we take individual sentences as input. existing work on this benchmark jointly evaluates span detection and entity typing, i.e. a prediction is only correct if both the span and the predicted type are correct. we will refer to this as the strict evaluation setting, following @xcite . we also consider the lenient setting from, where a prediction is scored as correct as soon as the type is correct and the predicted span overlaps with the gold span. baselines we report results for a number of simplified variants of our main model. first, we consider a variant which uses the same strategy for training the entity type classifier as our full model, but without pre-training the entity encoder on the gigagword corpus. this variant is referred to as the base model. second, we investigate a setup in which the entity encoder is pre-trained on gigaword, but only using the masked language modelling (mlm) objective. this setting, which we refer to as mlm-only, allows us to analyse to what extent improvements over the base model are due to the continued training of the language model. for reference, we also compare our models with the published results of state-of-the-art models. for fine-grained entity typing, we consider the following baselines: dsam @xcite is an lstmbased model, which we include as a competitive baseline; box4types @xcite uses hyperboxes to represent mentions and types, to take advantage of the hierarchical structure of the label space; @xcite uses a contrastive learning strategy based on the given type hierarchy; relational inductive bias (rib) @xcite uses a graph neural network to model correlations between the different labels. entity mentions are encoded using a transformer layer on top of pretrained elmo @xcite embeddings; lite @xcite assigns entity types by finetuning a pre-trained natural language inference model; seprem @xcite improves on the standard roberta model by exploiting syntax during both pre-training and fine-tuning, and then using a standard entity typing model on top of their pre-trained model; mlmet @xcite extends the standard distantly supervised training data, using the bert masked language model for generating weak labels; denoisefet @xcite uses a denoising strategy to improve the quality of the standard distantly supervised training set, and furthermore exploits prior knowledge about the labels, which is extracted from the parameters of the decoder of the pre-trained bert model; pkl @xcite improves on denoisefet by incorporating pre-trained label embeddings. for ace 2005, we consider the following baselines: dygie++ @xcite uses multitask learning to jointly train their system for coreference resolution, entity typing, relation extraction and event extraction; tableseq (wang and lu, 2020) jointly trains a sequence encoder for entity extraction and a table encoder for relation extraction; unire @xcite also uses a table based representation, which is shared for entity and relation extraction; pure (zhong and chen, 2021) uses bert-based models to get contextualised representations of mention spans, which are fed through a feedforward network to predict entity types; pl-marker @xcite builds on pure by introducing a novel span representation. ## conclusion we have proposed a strategy which uses coreference chains to pre-train an entity encoder. our strategy relies on the natural idea that coreferring entity mentions should be represented using similar vectors. using a contrastive loss for implementing this intuition, we found that the resulting encoders are highly suitable for (fine-grained) entity typing. in our analysis, we found that restricting our strategy to high-quality coreference links was important for its success. we also found that focusing on the head of the entity span, rather than the span itself, was beneficial, both when it comes to representing the entity span and when it comes to masking entities during training (where only masking the head was found to be most helpful).