F rom Z ero to H ero: H uman- I n- T he- L oop E ntity L inking in L ow R esource D omains (2020)

Abstract: Entity linking (EL) is concerned with disambiguating entity mentions in a text against knowledge bases (KB). It is crucial in a considerable number of fields like humanities, technical writing and biomedical sciences to enrich texts with semantics and discover more knowledge. The use of EL in such domains requires handling noisy texts, low resource settings and domain-specific KBs. Existing approaches are mostly inappropriate for this, as they depend on training data. However, in the above scenario, there exists hardly annotated data, and it needs to be created from scratch. We therefore present a novel domain-agnostic Human-In-The-Loop annotation approach: we use recommenders that suggest potential concepts and adaptive candidate ranking, thereby speeding up the overall annotation process and making it less tedious for users. We evaluate our ranking approach in a simulation on difficult texts and show that it greatly outperforms a strong baseline in ranking accuracy. In a user study, the annotation speed improves by 35% compared to annotating without interactive support; users report that they strongly prefer our system. An open-source and ready-to-use implementation based on the text annotation platform INCEpTION ( https://inception-project.github.io ) is made available.

## introduction entity linking (el) describes the task of disambiguating entity mentions in a text by linking them to a knowledge base (kb), e.g. the text span earl of orrery can be linked to the kb entry john boyle, 5. earl of cork, thereby disambiguating it. el is highly beneficial in many fields like digital humanities, classics, technical writing or biomedical sciences for applications like search (meij et al., 1 https://inception-project.github.io 2 https://github.com/ukplab/ acl2020-interactive-entity-linking figure 1 : difficult entity mentions with their linked entities: 1) name variations, 2) spelling variation, 3) ambiguity 2014), semantic enrichment @xcite or information extraction @xcite . these are overwhelmingly low-resource settings: often, no data annotated exists; coverage of open-domain knowledge bases like wikipedia or dbpedia is low. therefore, entity linking is frequently performed against domainspecific knowledge bases @xcite . in these scenarios, the first crucial step is to obtain annotated data. this data can then be either directly used by researchers for their downstream task or to train machine learning models for automatic annotation. for this initial data creation step, we developed a novel human-in-the-loop (hitl) annotation approach. manual annotation is laborious and often prohibitively expensive. to improve annotation speed and quality, we therefore add interactive machine learning annotation support that helps the user find entities in the text and select the correct knowledge base entries for them. the more entities are annotated, the better the annotation support will be. throughout this work, we focus on texts from digital humanities, to be more precise, texts written in early modern english texts, including poems, biographies, novels as well as legal documents. in this domain, texts are noisy as they were written in times where orthography was rather incidental or due to ocr and transcription errors (see fig. 1 ). tools like named entity recognizers are unavailable or perform poorly @xcite . we demonstrate the effectiveness of our approach with extensive simulation as well as a user study on different, challenging datasets. we implement our approach based on the open-source annotation platform inception @xcite and publish all datasets and code. our contributions are the following: ## related work in the following, we give a broad overview of existing el approaches, annotation support and human-in-the-loop annotation. entity linking describes the task of disambiguating mentions in a text against a knowledge base. it is typically approached in three steps: 1) mention detection, 2) candidate generation and 3) candidate ranking @xcite (fig. 2 ). mention detection most often relies either on gazetteers or pretrained named entity recognizers. candidate generation either uses precompiled candidate lists derived from labeled data or uses full-text search. candidate ranking assigns each candidate a score, then the candidate with the highest score is returned as the final prediction. existing systems rely on the availability of certain resources like a large wikipedia as well as software tools and often are restricted in the knowledge base they can link to. off-the-shelf systems like dexter @xcite , dbpedia spotlight @xcite and tagme @xcite most often can only link against wikipedia or a related knowledge base like wikidata or dbpedia. they require good wikipedia coverage for computing frequency statistics like popularity, view count or pagerank @xcite . these features work very well for standard datasets due to their zipfian distribution of entities, leading to high reported scores on stateof-the art datasets @xcite . however, these systems are rarely applied out-of-domain such as in digital humanities or classical studies. compared to state-of-the-art approaches, only a limited amount of research has been performed on entity linking against domainspecific knowledge bases. agdistis @xcite developed a knowledge-base-agnostic approach based on the hits algorithm. the mention detection relies on gazetteers compiled from resources like wikipedia and thereby performs string matching. @xcite propose reden, an approach based on graph centrality to link french authors to literary criticism texts. it requires additional linked data that is aligned with the custom knowledge base-they use dbpedia. as we work in a domain-specific low resource setting, access to large corpora which can be used to compute popularity priors is limited. we do not have suitable named entity linking tools, gazetteers or a sufficient amount of labeled training data. therefore, it is challenging to use state of the art systems. human-in-the-loop annotation hitl machine learning describes an interactive scenario where a machine learning (ml) system and a human work together to improve their performance. the ml system gives predictions, and the human corrects if they are wrong and helps to spot things that have been overlooked by the machine. the system uses this feedback to improve, leading to better predictions and thereby reducing the effort of the human. in natural language processing, it has been applied in scenarios like interactive text summarization @xcite , parsing @xcite or data generation @xcite . regarding machine-learning assisted annotation, @xcite propose an annotation editor that during annotation, interactively trains a model using annotations made by the user. they use string matching and mira @xcite as recommenders, evaluate on pos and ner annotation and show improvement in annotation speed. tasty @xcite is a system that is able to perform el against wikipedia on the fly while typing a document. a pretrained neural sequence tagger is being used that performs mention detection. candidates are precomputed and the candidate is chosen that has the highest text sim-figure 2 : entity linking pipeline: first, mentions of entities in the text need to be found. then, given a mention, candidate entities are generated. finally, entities are ranked and the top entity is chosen. ilarity. the system updates its suggestions after interactions such as writing, rephrasing, removing or correcting suggested entity links. corrections are used as training data for the neural model. however, due to the following reasons, it is not yet suitable for our scenario. in order to overcome the cold start problem, it needs annotated training data in addition to a precomputed index for candidate generation. it also only links against wikipedia. ## architecture the following section describes the three components of our annotation framework, following the standard entity linking pipeline (see fig. 2 ). throughout this work, we will mainly focus on the candidate ranking step. we call the text span which contains an entity the mention and the sentence the mention is in the context. each candidate from the knowledge base is assumed to have a label and a description. for instance, in fig. 2 , one mention is dublin, the context is dublin is the capital of ireland, the label of the the first candidate is trinity college and its description is constituent college of the university of dublin in ireland. ## datasets there are very few datasets available that can be used for el against domain-specific knowledge bases, further stressing our point that we need more of these, thereby requiring approaches like ours to create them. we use three datasets: aida-yago, women writers online (wwo) and 1641 depositions. aida consists of reuters news stories. to the best of our knowledge, wwo has not been considered for automatic el so far. the 1641 depositions have been used in automatic el, but only when linking against dbpedia which has a very low entity coverage @xcite . we preprocess the data, split it in sentences, tokenize and reduce noise. for wwo, we derive a rdf kb from their personography, for 1641 we derive a knowledge base from the annotations. the exact processing steps as well as example texts are described in the appendix. the resulting data sets for wwo and 1641 depositions are also made available in the accompanying code repository. aida-yago: for validating our approach, we evaluate on the aida-yago state-of-the art dataset introduced by @xcite . originally, this dataset is linked against yago and wikipedia. we map the wikipedia urls to wikidata and link against this kb, as wikidata is available in rdf and the official wikidata sparql endpoint offers full text search: it does not offer fuzzy search though. women writers online: women writers online foot_0 is a collection of texts by pre-victorian women writers. it includes texts on a wide range of topics and from various genres including poems, plays, and novels. they represent different states of the english language between 1400 and 1850. a subset of documents has been annotated with named entities (persons, works, places) @xcite . persons have also been linked to create a personography, a structured representation of persons' biographies containing names, titles, time and place of birth and death. the texts are challenging to disambiguate due to spelling variance, ciphering of names and a lack of standardized orthography. sometimes, people are not referred to by name but by rank or function, e.g. the king. this dataset is interesting, as it contains documents with heterogeneous topics and text genres, causing low redundancy. ## experiments to validate our approach, we first evaluate recommender performance. then, non-interactive ranking performance is evaluated similarly to state-ofthe-art el. afterwards, we simulate a user annotating corpora with our human-in-the-loop ranker. finally, we conduct a user study to test it in a realistic setting. similar to other work on el, our main metric for ranking is accuracy. we also measure accuracy@5, as our experiments showed that users can quickly scan and select the right entity from a list of five elements. in our annotation editor, the candidate list shows the first five elements without scrolling. as a baseline, we use the most-frequently linked entity baseline (mfleb). it assigns, given a mention, the entity that was most often linked to it in the training data. ## conclusion we presented a domain-agnostic annotation approach for annotating entity linking for lowresource domains. it consists of two main components: recommenders that are algorithms that suggest potential annotations to users and a ranker that, given a mention span, ranks potential entity candidates so that they show up higher in the candidate list, making it easier to find for users. both systems are retrained whenever new annotations are made, forming the human-in-the-loop. our approach does not require the existence of external resources like labeled data, tools like named entity recognizers or large-scale resources like wikipedia. it can be applied to any domain, only requiring a knowledge base whose entities have a label and a description. in this paper, we evaluate on three datasets: aida, which is often used to validate state-of-the-art entity linking systems as well as wwo and 1641 from the humanities. we show that in simulation, only a very small subset needs to be annotated (fewer than 100) for the ranker to reach high accuracy. in a user study, results show that users prefer our approach compared to the typical annotation process; annotation speed improves by around 35% when using our system relative to using no reranking support. in the future, we want to investigate more powerful recommenders, combine interactive entity linking with knowledge base completion and use online learning to leverage deep models, despite their long training time.