The DA - ELEXIS Corpus - a Sense-Annotated Corpus for D anish with Parallel Annotations for Nine E uropean Languages (2023)

Abstract: In this paper, we present the newly compiled DA-ELEXIS Corpus, which is one of the largest sense-annotated corpora available for Danish, and the first one to be annotated with the Danish wordnet, DanNet. The corpus is part of a European initiative, the ELEXIS project, and has corresponding parallel annotations in nine other European languages. As such it functions as a cross-lingual evaluative benchmark for a series of low and medium resourced European language. We focus here on the Danish annotation process, i.e. on the annotation scheme including annotation guidelines and a primary sense inventory constituted by DanNet as well as the fall-back sense inventory namely The Danish Dictionary (DDO). We analyse and discuss issues such as out of vocabulary (OOV) problems, problems with sense granularity and missing senses (in particular for verbs), and how to semantically tag multiword expressions (MWE), which prove to occur very frequently in the Danish corpus. Finally, we calculate the inter-annotator agreement (IAA) and show how IAA has improved during the annotation process. The openly available corpus contains 32,524 tokens of which sense annotations are given for all content words, amounting to 7,322 nouns, 3,099 verbs, 2,626 adjectives, and 1,677 adverbs.

## introduction even today in the era of neural language models, high-quality, sense-annotated corpora that are openly accessible prove to be highly requested for the training and evaluation of semantically related nlp tasks, in particular tasks such as word sense disambiguation (wsd) and natural language understanding (nlu). in spite of numerous initiatives in the field during the last decades, such corpora are still in short supply for many lower-resourced languages, including to some extent the nordic languages. two main factors lie at the root of this scarcity: the former factor plays a particularly important role, since most curated dictionaries are not open for such use in practice, and since those that are available, may not be well-suited for several reasons. we present here the da-elexis corpus, which is one of the largest sense-annotated corpora available for danish, and the first one to be annotated with the danish wordnet, dannet @xcite . the corpus is compiled as part of a larger european initiative, the elexis project, @xcite and corresponding parallel annotations have taken place in nine other european languages. as such it functions as a crosslingual evaluative benchmark for a series of low and medium resourced european language. for a preliminary presentation of the design of the joint initiative, cf. @xcite . the initiative was led by the artificial intelligence laboratory, jozef stefan institute in ljubljana and the department of computer science, sapienza university of rome; each language group was, however, responsible for their own annotation procedures and sense inventories. in this paper, we focus mainly on the danish annotation process, including the danish annotation scheme and the issues that arose during annotation with regards to calibration and agreement among annotators etc. the corpus is freely available and can be downloaded from clarin, www.clarin.si under a cc-by-sa 4.0 license. the paper is structured as follows: in section 2 we give an account on related work, and in 3 we present the corpus and its annotation layers previous to the semantic annotation and provide examples where specific danish adjustments were required. section 4 discusses the danish sense inventories applied for the task, and in section 5 we describe the annotation process in more detail, whereas section 6 discusses issues on interannotator agreement of the annotations. finally, in section 7, we conclude and discuss potential future investigations and development. ## related work since the early days of semcor @xcite , which is one of the first sense-annotated corpora for english based on the princeton word-net sense inventory @xcite , there has been a continuous request in the nlp community for sense-annotated corpora for the world's languages. hence, semantic annotation projects have been carried out for a variety of languages; some are based on purely monolingual grounds, while others have adopted different kinds of multilingual approaches. @xcite provides an overview of semcor corpora and other corpora annotated with wordnets for different languages, and @xcite provides more detail on the multilingual semcor approaches. the ontonotes corpus @xcite which comprises english, chinese and arabic, also uses wordnet as a starting point for its sense annotations of the english part whereas the chinese and arab parts base the sense annotations on various lexical sources. newer initiatives experiment with semiautomatic approaches to sense annotation in order to overcome the scarcity of such data sets, among others the onesec corpora created by @xcite in five languages, namely english, french, german, italian, and spanish. these corpora consist of wikipedia texts with between 1.2 and 8.8 m sense annotations of nouns per language. if we look into the scandinavian languages, the swedish eukalyptus corpus @xcite is a sense annotated corpus of 100,000 tokens annotated with the senses from the saldo lexicon @xcite , a swedish lexicalsemantic resource based on a concept of 'centrality' instead of being based on the hyponymy relation, which is a central organisational relation in wordnets. for norwegian, sense tagging has mostly focused on named entity tags, see e.g. @xcite whereas a semcor-like resource does not exist to the best of our knowledge. in the case of danish, there have been but a few previous initiatives concerned with sense annotation. @xcite presents the sem-dax corpus, which comprises 86,786 tokens, of which the 34,421 content words are sense annotated. it is important to notice, however, that the sense inventory applied in semdax refers to the so-called supersense inventory, which is a very coarse-grained, multilingual sense inventory derived from the list of wordnet's first beginners or lexicographical files -corresponding roughly to top-ontological types. in contrast, the da-elexis corpus applies a fully-fledged sense inventory derived from monolingual sources, as we will describe in more detail below. ## the corpus and its annotation layers da-elexis consist of 2024 sentences that were extracted from wikimatrix3 @xcite . wikimatrix3 is an immense open-access collection of parallel sentences derived from wikipedia covering a diverse set of technical domains from this resource. the wikimatrix corpus overall covers wikipedia articles in 96 languages, resulting in 1620 language pairs. the elexis sense-annotated parallel corpora have been extracted from this collection for 10 european languages, namely bulgarian, danish, dutch, english, estonian, hungarian, italian, portuguese, slovene, and spanish @xcite . only sentences in english with a counterpart in one of the other nine languages were extracted from wikimatrix, and missing translations were either retrieved automatically and validated manually or translated manually. to ensure that the sense annotation (also for multiword entities (mwes)) can be performed in a flexible and consistent way, the elexis corpus contains a total of five annotation layers, four of which are completed prior to the sense annotation: a challenge for danish was how to deal with compounds, which, as for most germanic languages, are quite common and relatively dynamically generated, and more importantly: they are written as a single word. we adopted the approach that conventionalized compounds found either in dannet or in the danish dictionary (henceforth ddo) @xcite should be kept as such, while compounds not found in any of these resources should be split into lemmas included in the resources, in order to enable them to be semantically tagged. when splitting compounds with a binding element, e.g. 's' in forsøgsperiode ('trial period'), it was decided to keep the binding element during the subtokenisation and pos-tagging phase and to finally remove it in the lemmatization phase. in several other cases, decisions were required at the pos-tagging level in order to facilitate the semantic tagging. for example in cases when participles are used as adjectives. participles with adjective entries in the dictionary were lemmatised as such, e.g. udstrakt ('outstretched', fig: 'extensive'), while those that had only verb entries in the dictionary were lemmatised as verbs, e.g. samlede ('assembled', fig: 'total'). ## sense inventories for the danish annotation the sense inventory applied for the annotation, is mainly constituted by the dannet resource foot_0 @xcite , which is an open-source wordnet compiled semi-automatically on the basis of ddo. dannet covers 70,000 danish lemmas and includes approximately half of the ddo senses from the first, printed edition of the dictionary, mainly from nouns and verbs. it contains a slightly simplified sense inventory, where some ddo senses are collapsed into one when they have been considered very close in meaning. senses are organised in synonym sets, each one called a synset, which constitute the basic building blocks in a standard wordnet, cf. fellbaum et al. (eds) 1998. dannet has taken over the sense definitions and usage examples from ddo, but due to copyright retrictions, definitions are given in an abbreviated form where only the first 50 characters are represented. in contrast to ddo, dannet is open-source (cc by-sa 4.0) allowing the sense-annotated corpus to be freely used and integrated in all kinds of pipelines and applications. this was a prerequisite defined by the elexis project for participating in the annotation task. since not all senses are covered in dannet, a current online version of ddo was used as fallback and new senses from this resource were established via the annotation tool when required. this version of the dictionary covers approximately 100,000 lemmas and 150,000 senses and is continuously updated and published online since 2009 at ordnet.dk/ddo. ## agreement among annotators it is one thing whether a sense is actually described in the sense inventory at hand, another, however, is to what extent annotators agree on which sense tags to use for a given example. to study this aspect, we have, in accordance with consensus for semantic annotations tasks, triple annotated a little over 5% of the corpus, amounting to 108 sentences. this triple annotation has enabled us to calculate inter-annotator agreement and examine differences among annotators. inter-coder agreement reveals interesting things about several aspects of the annotation task. a number of issues come into play, such as: we calculate an average cohen's kappa agreement for the triple annotated data of 0.68 between all three annotators as seen in figure 4 . we would have liked also to employ the krippendorph alpha measure, which takes into account the fact that it is generally easier to agree on few labels than on many, but this measure proved impossible to calculate in practice since it requires a full list of all possible senses (including mwes) that every word can occur in, and such mwe lists are not provided in dannet. however, since the first annotation of the 108 sentences was performed in the first phase of the annotation project (corresponding to annotator 1 in figure 5 ), several issues regarding delimitations of mwe, diverging pos tags, and problems with the encoding of compounds etc. had not yet been clarified. the two other annotations (corresponding to annotators 2 and 3 in figure 5 and 6 ) were performed at the end of the project when more or less all pending issues had been clarified. here we achieve an inter-coder agreement of 0.78. this can be considered a quite substantial level of agreement for this kind of task. as revealed by the bullet points above, most issues come into play in the diverging annotations. as already touched upon, preprocessing proves to have caused some divergences, in particular in re-lation to mwe, and since these are relatively frequent, in particular for verbs (phrasal verbs), we positively know that this has caused divergences in a number of cases. with regards to coverage, it can be assessed to be relatively good if we see dannet and ddo all together even if a small percentage of the examples could not be tagged with existing senses. the granularity of our sense inventory is quite high as is the case for most dictionaries, in fact we have an average of 4.2 senses per lemma in the ddo overall. we can see that the fine granularity causes agreement problems in some cases, in particular when distinguishing between a high number of verbal senses. in spite of problems with fine-grained senses, differences in agreement early and late in the project show that the guidelines improved substantially after the first round. finally, the annotators involved in the task were professional lexicographers or computational linguists with overall high proficiency in annotating, and no annotators proved to protrude in the overall quality of their annotations compared to others. ## conclusions and future work in this paper, we have presented the da-elexis corpus, which is one of the largest senseannotated corpora available for danish and the first one to be annotated with the danish wordnet, dannet. we have described the careful preprocessing and preparation necessary to ensure a high quality of the resulting resource, and we have presented a series of difficulties in relation to sense coverage and in achieving a high inter-annotator agreement. in particular, the limitation and semantic description of mwe have proven to cause divergences among annotators even if these were reduced during the development of the annotation guidelines. even if we know that the granularity of our sense inventory is also crucial to the interannotator agreement, we have chosen not to go too deeply into the theoretical discussion of this issue here. this is in spite of the fact that sense coverage and granularity is highly discussed in lexical semantic literature @xcite , fillmore and atkins (1992) and many others) and that some even claim that word senses don't exist, or at least only relative to specific tasks @xcite . along the same lines, fine-grained sense invento-ries (like the ones we have applied here) have been deemed somewhat unsuitable for nlp tasks such as wsd and nlu. in this context, we are currently examining ways to achieve a high-quality, coarsegrained sense inventory for danish, since we are building a new lexical resource, the central word register of danish, cor, particularly for nlp, @xcite . still based on -and linked to -the same sources, namely dannet and the ddo, we are developing principled and semi-automatic procedures for reducing the inventory with more than 40 % @xcite . thus, we expect to have a more suitable inventory for sense annotation available in future projects, probably by the end of 2023. we also plan to experiment with automatically added semantic and thematic information to the corpus, based on the available information in different semantic lexicons linked at sense level to the ddo and dannet, including thesaurus information on topics and themes. finally, another aspect that deserves further attention in future work is the potential of the multilingual setup of the elexis corpus. having parallel sense annotations in nine aligned languages, several of them being generally low-resourced, provides valuable information not only for each individual language (danish, in our case), but also for cross-lingual studies in nlp and lexicography. this provides us with a valuable crosslingual evaluation benchmark to be applied for future wsd and nlu tasks.