Human-In-The- L oop E ntity Linking for Low Resource Domains (2021)

Abstract: Entity linking (EL) is concerned with disambiguating entity mentions in a text against knowledge bases (KB). To quickly annotate texts with EL even in low-resource domains and noisy text, we present a novel Human-In-The-Loop EL approach. We show that it greatly outperforms a strong baseline in simulation. In a user study, annotation time is reduced by 35 % compared to annotating without interactive support; users report that they strongly prefer our system over ones without. An open-source and ready-to-use implementation based on the text annotation platform is made available.

## introduction entity linking (el) describes the task of disambiguating entity mentions in a text by linking them to a knowledge base (kb), e.g. the text span earl of orrery can be linked to the kb entry john boyle, 5 th earl of cork, thereby disambiguating it. el is highly relevant in many fields like digital humanities, classics, technical writing or biomedical sciences for applications like search @xcite , semantic enrichment @xcite or information extraction @xcite . in these scenarios, the first crucial step is typically to annotate data. manual annotation is laborious and often prohibitively expensive. to improve annotation speed and quality, we have developed a novel human-in-the-loop (hitl) entity linking approach. it helps annotators finding entity mentions in the text and linking them to the correct knowledge base entries. the more mentions get linked over time, the better the annotation support will be. we demonstrate the effectiveness of our approach with extensive simulation as well as a user study on different, challenging datasets. we have implemented our approach based on the opensource annotation platform inception @xcite and publish all datasets and code. ## implementation entity linking describes the task of disambiguating mentions in a text against a knowledge base. manual annotation of el consists of three steps @xcite . first, the annotator selects a span that contains an entity. then, they search for the correct entity in a kb. these search results are reranked to rank more suitable candidates higher. each candidate from the knowledge base is assumed to have a label and a description. to speed up this annotation process, we support users twofold: to find suitable spans, we provide recommenders that suggest potential entity spans. they can also classify these entity spans (e.g. as person, location, etc.). these recommenders learn from new annotations and are retrained in the background. for candidate ranking, we follow @xcite and model it as a learning-to-rank problem: given a marked span, search query and a list of candidates, sort the candidates so that the most relevant candidate is at the top. by selecting an entity label from the candidate list, users express that the selected one was preferred over all other candidates. these preferences are used to train state-of-the-art pairwise learning-to-rank models from the literature: the gradient boosted trees variant lightgbm @xcite and ranksvm @xcite . the continuously updated models improve over time with an increasing number of annotations. as input features, we use different similarity measures between the marked span and the candidate label, between the spans' context and the candidate description as well as dense word and sentence embeddings of the descriptions. datasets we use the following three datasets for validating our approach: 1) the aida-yago stateof-the art dataset introduced by @xcite . ## experiments to validate our approach, we simulate a user annotating with our hitl ranker. then, we conduct a user study to test it in a real-life setting. similar to other work on el, our main metric for ranking is accuracy. we also measure accuracy@5, as our experiments showed that users can quickly scan and select the right entity from a list of five elements. simulation fig. 1 depicts the simulation results. all models outperform a majority baseline over most of the annotation process. it can be seen that both of our used models achieve high performance even if trained on very few annotations. the ranksvm handles low data better than lightgbm, but quickly reaches its peak performance due to it being a linear model. this potentially allows to first use a ranksvm for the cold start and when enough annotations are made, lightgbm, thereby combining the best of both. ## conclusion we presented a domain-agnostic annotation approach for annotating entity linking for lowresource domains. it consists of two main components: recommenders that are algorithms that suggest potential annotations to users and a ranker that, given a mention span, ranks potential entity candidates so that they show up higher in the candidate list, making it easier to find for users. both systems are retrained whenever new annotations are made, forming the human-in-the-loop. in a user study, results show that users prefer our approach compared to the typical annotation process; annotation speed improves by around 35% when using our system relative to using no reranking support.