A Study on Entity Linking Across Domains: Which Data is Best for Fine-Tuning? (2022)

Abstract: Entity linking disambiguates mentions by mapping them to entities in a knowledge graph (KG). One important question in today’s research is how to extend neural entity linking systems to new domains. In this paper, we aim at a system that enables linking mentions to entities from a general-domain KG and a domain-specific KG at the same time. In particular, we represent the entities of different KGs in a joint vector space and address the questions of which data is best suited for creating and fine-tuning that space, and whether fine-tuning harms performance on the general domain. We find that a combination of data from both the general and the special domain is most helpful. The first is especially necessary for avoiding performance loss on the general domain. While additional supervision on entities that appear in both KGs performs best in an intrinsic evaluation of the vector space, it has less impact on the downstream task of entity linking.

## introduction entity linking, i.e., the task of disambiguating mentions in text by linking them to entities of a knowledge graph @xcite , is key to many semantic applications, such as kg population, question answering or information retrieval @xcite . in the context of kgs, a domain is characterized, i.a., by the set and distribution of entities @xcite . for kgs from special domains, the availability of annotated data for training entity linking is limited. thus, there is a need for methods that work across domains in low-resource settings, such as transfer or few-shot learning techniques @xcite . given a kg from a special domain, it is useful for many applications to not treat this kg in isolation but still be able to link mentions to the general domain as well. figure 1 illustrates this. without combining the kgs wikipedia and doctor who, it would not be possible to link all mentions of the jaime holloway was an american high school senior example sentence to their respective entities. early works prior to neural entity linking @xcite allow linking to multiple kgs by combining the kgs before applying the methods. in the context of neural networks, a more elegant way is to combine different kgs via a joint vector space @xcite . this also enables us to learn similar embeddings for overlapping entities, i.e., entities that appear in more than one kg, which would arguably be more difficult when only uniting triple sets. in figure 1 , the entities "clara oswald" and "clara oswald (immortals)" are an example of overlapping entities. in this paper, we aim at methods for adding a kg from a specific target domain into an existing vector space from a source-domain kg and fine-tuning the joint vector space to improve the entity linking results. while some recent work has considered zeroshot entity linking @xcite , a systematic investigation on which data sources are most useful for fine-tuning entity linking systems, is still missing. thus, the first research question we address is: which data is best suited for fine-tuning joint vector spaces of kgs? furthermore, it is unclear how fine-tuning on the target domain affects the vector space of the sourcedomain kg. thus, the second research question we pose is: does fine-tuning harm performance of entity linking on the general domain? to answer these research questions, we present a systematic investigation of the impact of different information sources on the vector space (intrinsic evaluation) as well as on the entity linking performance (extrinsic evaluation). further, we will publish the list of overlapping entities that we created along with this paper to ensure reproducibility. ## related work as in other fields of natural language processing, deep learning became the predominant approach in entity linking @xcite @xcite @xcite . in today's research, the usage of pre-trained language models, such as bert @xcite , is particularly popular @xcite @xcite . in this paper, we aim at a neural entity linking system which allows linking mentions to more than one kg by creating a joint vector space for entity representations from different kgs. related to this, @xcite propose a method to create a joint vector space for entities from different sources that are represented by different means, such as descriptions, contexts or fine-grained types. in the context of entity linking across domains, onoe and durrett (2020) build a domain-independent system that relies on fine-grained entity types. in contrast, @xcite and @xcite utilize descriptions of entities from kgs to obtain entity representations. in particular, @xcite propose domain-adaptive pre-training to apply entity linking to unseen entities from a kg of a new domain. @xcite build on that work but train their model only on labeled data from a general domain (wikipedia). vyas and ballesteros (2021) generalize those models and allow them to handle arbitrary kgs with entities represented by an arbitrary set of attribute-value pairs. in contrast to those works, we also take into account overlapping entities between the two kgs and study which impact fine-tuning on different data sources has on the joint vector space as well as on entity linking performance. ## linking model and extension method in this section, we detail the entity linking model and describe how the model can be extended to a new domain. ## experiments and results in this section, we present our study and report the effects of fine-tuning on the latent representation as well as on the downstream task of entity linking. baseline. we use the neural entity linking model blink @xcite ) as our baseline model. 1 fine-tuning configurations. we experiment with the following combinations of the information sources presented in section 3.2: s, t, to, ts, tos. 2 hyperparameters are provided in section a of the appendix. ## conclusion in this paper, we presented a systematic investigation of extending an entity linking system to a new domain by creating a joint vector space. our results showed that it is helpful to add data from both the source domain and the target domain. while an additional supervision on entities that appear in both knowledge graphs improves the quality of the vector space, it has less impact on the downstream task of entity linking. additional data linked to the source-domain kg avoids performance loss on the general domain and is, thus, especially useful to achieve a system that can link mentions to both source and target-domain kgs at the same time.