Resources and Evaluations for D anish Entity Resolution (2021)

Abstract: Automatic coreference resolution is understudied in Danish even though most of the Danish Dependency Treebank (Buch-Kromann, 2003) is annotated with coreference relations. This paper describes a conversion of its partial, yet well-documented, coreference relations into coreference clusters and the training and evaluation of coreference models on this data. To the best of our knowledge, these are the first publicly available, neural coreference models for Danish. We also present a new entity linking annotation on the dataset using WikiData identifiers, a named entity disambiguation (NED) dataset, and a larger automatically created NED dataset enabling wikily supervised NED models. The entity linking annotation is benchmarked using a state-of-the-art neural entity disambiguation model.

## introduction the danish dependency treebank (ddt) @xcite is a beneficial resource for danish nlp that contains several annotation layers. most of the layers were annotated as part of the copenhagen dependency treebank project, but a conversion of the dependency syntax annotation into universal dependencies @xcite ) and the addition of named entities annotation layers @xcite @xcite ) are newer additions. the partial coreference annotation has received no attention for nlp purposes despite being very well documented. this paper describes converting the coreference relations into coreference clusters and a new entity linking annotation with unique wikidata item identification codes (qids) @xcite on the same data. entity linking is the task of detecting mentions and matching the mentions to a knowledge base. the two annotation layers-coreference and entity linking-complement each other as two types of entity resolution. in practice, entity linking can be reduced to a binary classification task, called named entity disambiguation (ned), thereby simplifying the task and omitting mention detection. the coreference-annotated data is benchmarked using a strong neural model. the entity linking annotation is turned into a ned dataset and benchmarked using a state-of-the-art ned model. we automatically create a larger ned train set from wikipedia articles for train set augmentation, and we observe an improvement for the wikily supervised models. all data and models are publicly available. foot_0 2 related work danish anaphors have received some attention: navarretta @xcite shows that danish deictics are used in more contexts than the english ones and @xcite specifically discuss the use of verb phrase pronominalization in danish. danish has gendered possessive pronouns, but nongendered reflexive pronouns. this has made it useful as an unambiguous testbed for gender bias in natural language inference models, machine translation models, and language models @xcite . but automatic coreference resolution for danish has received no attention, and there was no established evaluation set for this task. linked resources such as wikipedia enable multi-lingual entity linking/ned models and datasets, and danish is often among the evaluation languages @xcite . dbpedia spotlight foot_1 is the most recent entity linking system that also supports danish. but due to this being a different task from ned, we can not compare our model to dbpedia spotlight. annes bil ramte et trae . køretøjet blev totalskadet i ulykken . den skete utilsigtet . bilen blev kørt vaek . annes car hit a tree . the vehicle was totalled in the accident . it happended unintentionally . the car was driven away . coref-var coref-res coref coref-iden figure 1: a constructed example of a coreference-annotated paragraph with binary relations as graphs and three clusters marked with background colour or borders around the word spans. the english word-by-word translation is below. translation: "anne's car crashed into a tree. the vehicle was damaged beyond repair in the accident. it happened unintentionally. the car was towed." n la ua rel coref 238 61 64 89 coref-var 146 59 72 73 ref 70 90 91 94 coref-iden 62 73 81 77 coref-res 36 62 67 75 coref-evol 1 0 100 0 the coreference-annotated part spans 2/3 of the ddt and is documented in @xcite . this subset encompasses 341 documents/3,403 sentences/64,076 tokens. the source is the parole corpus @xcite . the domain is mostly newswire (299 documents) but also a small fraction of magazine text (41 documents) and news broadcast (1 document). we refer to this dataset as dacoref. ## coreference annotation and conversion the ddt documentation 3 is extensive and this section's details concerning the annotation come from this resource. we refer to the ddt documentation for more details. key observations about the annotation scheme: like the ontonotes @xcite coreference annotation, singletons are not annotated. the annotation does not label attributive noun phrases connected through copula verbs such as "to be" ("at vaere"). verb phrases can only be linked if they are coreferent with a noun phrase. the dataset is annotated with binary relations between core nodes. we consider nine different coreference labels that we merge heuristically into clusters. we omit the associative anaphors. coreference clusters are sets of text spans that all refer to the same entity and labels are omitted. each cluster is uniquely numbered. we merge the following relations if they span the same target words: coref (coreferential personal pronoun), coref-ell (elliptic anaphor demonstrative pronoun), coref-evol (evolving anaphor), coref-iden (coreferential noun phrase with complete lexical identity), coref-iden.sb (coreferential noun phrase with lexical identity in the noun but lexical variety in some other (typically attributive) component), coref-var (coreferential noun phrase with lexical variety in the noun), and ref (syntactically determined coreference, e.g., relative pronouns). coref-res (resumptive anaphor clause or predicate) and coref-res.prg (pragmatic resumptive anaphor) are also used for clustering but not merged with any other label, nor each other. figure 1 shows a constructed example where coref-var, coref-iden and coref are merged when on the same target word(s), but coref-res is not, thereby making ulykken part of two clusters. seven documents are doubly annotated, and they are assumed to form the basis of the reported annotation quality in the documentation, which we reprint in the coreference annotation only contains a label on the core node. for dacoref, we propagate the label to the relevant span. due to several instances of the label not being on the core node, this is done manually by one annotator. the spans encompass the relevant noun phrases. when linking verb phrases, the span covers the entire verb phrase. this is different from ontonotes @xcite , where the span only covers the head verb. even though the ddt has been split into train, development and test sets as part of the universal dependencies (ud) conversion @xcite , a new split is created for document-level coreference resolution since the ud dataset scrambled the sentences across documents. the train split contains 290 documents, the development set 23, and the test set 28 documents. coreference models we provide benchmark scores for two strong neural models using three different transformer representations. @xcite (lee2017) presented the first neural endto-end coreference model in which the spans are learned in the same training pass as the pairwise clustering. the pairwise clustering may produce globally inconsistent clusters, and @xcite (lee2018) presented a higher-order upgrade that did not only consider pairs of spans but a matrix of all spans to counter global inconsistencies. @xcite showed that the model from @xcite performed even better on english data when using transformer-based models instead of the word embeddings of the original implementations. we follow their approach and try representations from three different pre-trained models: danish bert (dabert) foot_3 and two multilingual, cased models: multilingual bert (mbert) @xcite and the base model of xlm-roberta (xlm-r) @xcite . instead of the tensorflow implementations released by @xcite , we use the pytorch-based implementation from allennlp 1.3.0. foot_4 with pytorch version 1.7.1. a description of the tuning process is in appendix a. after model selection, we retrain the models for a maximum of 1200 epochs with early stopping and a patience of 10. ## entity linking entity linking annotation the resource was annotated with qids in the spring of 2020 with the wikidata entries available at that time in a semimanual process: first, all tokens in named entities were used to search the mediawiki api. given the entire list of wikidata matches for each token, one annotator decided which qid match was correct for each. we did not search for entire mentions since mentions were not always equal to the wiki-data label and thus returned too many wrong qid suggestions leaving this method inefficient. in the case of multi-word entity names, we searched for possible qid matches for each token and manually selected the correct one. one annotator decided in each case whether "margrethe" referred to the danish queen, a foundation in her name or another person/ship etc. this was checked by another annotator, who also manually added the qid to the correct span in the text. both were native speakers of danish. the average accuracy of the set of qids per document of annotator 1 compared to annotator 2 is 0.98 (std. 0.05). furthermore, the second annotation process also included adding a generic qid for words that matched one of 47 categories (ship q11446, award q618779, automobile model q3231690, hospital q16917 etc.) for which no specific wikidata entry existed. the list of all generic qids can be seen in appendix b. in total, 7,173 tokens were annotated with a qid. 2,193 unique qids were used. construction of the daned dataset entity disambiguation is a binary classification variant of entity linking. given a sentence, the entity name as printed in the sentence, and a qid; the model classifies whether this qid is the mentioned entity in the sentence. the task requires creating a new classification dataset also containing negative examples. only sentences from dacoref that have at least one qid annotation are part of the datasets. we create the negative examples in the following manner: for each token annotated with a qid in the train, development, and test set, we search wikidata using sparql for other matches than the correct qid but with the same label name. if possible, up to two negative examples per qid are included. the same sentence occurs several times in each split due to negative examples and possibly multiple qids in the same sentence, but the same sentence does not occur across splits. we used the mediawiki api to obtain all wiki-data knowledge graph (kg) contexts in danish for both the positive and negative examples. due to model constraints, we cut off the properties after 512 characters. an example of the representation of an instance is presented in the train-development-test split is determined by the dacoref splits and ended in a 78-9-13% distribution. class balance and split size after the construction of negative samples and dataset size are shown in figure 2 . this dataset is referred to as daned. train set augmentation we automatically construct a large danish ned dataset from wikipedia, dawikined. we use a list of 391,102 danish articles names from the latest wiki dump 6 . initially, we further gather 82,150 person names that have danish articles from the sparql endpoint. we fetch the wikipedia text using the sparql api for each of the article names and split it into sentences using the nltk library. we then pick the longest sentence that also contains the article title 7 . if possible, we fetch the corresponding qid, the corresponding kg context and the description for this entity and add it to the dataset. we fetch up to two negative examples per qid. final dawikined dataset size is in figure 2 . ## conclusion we have presented a semi-manual conversion of coreference relations into coreference clusters and a novel entity linking annotation for danish. the latter annotation is transformed into a named entity disambiguation dataset, and we further described the automatic construction of a danish wikipedia named entity disambiguation dataset. we have reported the first benchmarks for danish coreference resolution and an evaluation of our named entity disambiguation dataset with train set augmentation.