Modelling and Annotating Interlinear Glossed Text from 280 Different Endangered Languages as Linked Data with LIGT (2020)

Abstract: This paper reports on the harvesting, analysis, and enrichment of 20k documents from 4 different endangered language archives in 300 different low-resource languages. The documents are heterogeneous as to their provenance (holding archive, language, geographical area, creator) and internal structure (annotation types, metalanguages), but they have the ELAN-XML format in common. Typical annotations include sentence-level translations, morpheme-segmentation, morpheme-level translations, and parts-of-speech. The ELAN-format gives a lot of freedom to document creators, and hence the data set is very heterogeneous. We use regularities in the ELAN format to arrive at a common internal representation of sentences, words, and morphemes, with translations into one or more additional languages. Building upon the paradigm of Linguistic Linked Open Data (LLOD, Chiarcos, Nordhoff, et al. 2012), the document elements receive unique identifiers and are linked to other resources such as Glottolog for languages, Wikidata for semantic concepts, and the Leipzig Glossing Rules list for category abbreviations. We provide an RDF export in the LIGT format (Chiarcos & Ionov 2019), enabling uniform and interoperable access with some semantic enrichments to a formerly disparate resource type difficult to access. Two use cases (semantic search and colexification) are presented to show the viability of the approach.

## introduction we see that, next to english, there are only 6 further languages for which the resources can be considered satisfying (class 5). as we relax requirements for labeled and unlabeled data, we arrive at this work is licensed under a creative commons attribution 4.0 international license. licence details: http:// creativecommons.org/licenses/by/4.0/ . classes 4, 3, and 2, with each one numbering languages in the low two-digit range. in class 1, only some unlabeled data are available, which joshi et al find to be true for 222 languages. for 2 191, they find no noteworthy data whatsoever. class 0 alone holds 7 times more languages than all of the other classes combined. of course, there is more data available than joshi et al include in their study. the glottlog glottoscope foot_0 lists 7 794 languages, of which 2,088 have only a short word list or less, and 5 706 languages have better data available than "word list". this being said, virtually all those additional languages mentioned in glottolog will be in joshi et al's class 0: the resources listed are in the range of 10 to 10 4 tokens, whereas nlp applications typically require at least 10 6 datapoints to work properly, possibly much more. but the human language faculty is not restricted to the 7 languages where we have a good data situation. it is due to historical accidents that these languages are predominant today, and they are actually not very protoypical representatives of the class "natural language". foot_1 by focusing our attention, and our research, on these 7 larger languages, we are making a commercially sensible choice, but we might be missing important insights into the nature of the human mind. but not all hope is lost, since for the last 25 years, data from many class-0 languages have been collected, largely unnoticed by the nlp communities. these data reside tucked away in endangered language archives, waiting to be discovered. ## data in the context of this study, we restrict ourselves to documents in the elan-xml format (*.eaf). the reason for this is that this is a well-structured format used for many documents. the competing shoebox/toolbox foot_10 format is idiosyncratic and undocumented, while the flex format 12 is not very well represented in the archives under discussion. ## data enrichment for the vernacular languages, there are no nlp tools available, as explained above, but an explanation on how to carve a canoe with an axe should yield the relevant concepts 'canoe' and 'axe' via the english translation just as well. in order to tackle the task of discoverability, we ran the grobid-ner named entity recognition service foot_14 on the english translation. for this, we collated all translations of all the sentences in a given document. ## integration our export as ligt-rdf has, among other things, the advantage of an easy integration into the linguistic linked open data cloud (llodc, @xcite ). next to metadata about the languages from glottolog, igt data from endangered language archives can be integrated with various other resources with the linked data approach used here. @xcite used dbnary @xcite ), based on wiktionary, to provide glosses in 15 additional languages, beyond the glosses supplied in the original document. @xcite expanded on this and model igt based on dublincore, nif @xcite ) and webannotation @xcite ) vocabularies. this allows them to integrate igt from toolbox, flex, and the xigt format into ligt. to this set, we can now add the elan data from the endangered language archives. nordhoff (2020) enriched the documents with semantic concept annotations linked to wikidata. he used this for an analysis of the most frequent semantic domains found in documents in endangered language archives (to wit, the caucasus and agriculture). we have now added the transitive closure of the parent relations of these concepts. another integration is the generation of candidates for colexification @xcite ). concepts do not map alike on lexemes in the languages of the world. some languages use different words for the concepts hand and arm for instance, while others use the same word. those languages which use the same word are said to "colexify" the two concepts. the study of colexification is complicated by homonymous words which do not share any semantics, like arm upper limb and arm weapon . ## sharing of resources the legal status of the underlying documents is shaky. the legal domains concerned are privacy law on the one hand and intellectual property law on the other. depending on where the data were collected, different legal frameworks might apply. some data might not enjoy any protection, like someone counting from 1 to 10, while other documents have clearly copyrightable content, e.g. a narrative. the same holds true for translations of said content. things are different for linguistic analysis. the fact that in a given document, we find a word which has been glossed as "give-1pl.pres" can probably be shared without too much of a legal risk. the same is true for lists of such annotations, i.e. morpheme-by-morpheme glosses of whole sentences. these collections eaf files transcribed duration words transcribed export triples can be made available for further inspection. even less problematic are the semantic enrichments. the fact that a given document is a about a certain kind of woodpecker is unproblematic and can be shared. the linked data approach used here allows us to elegantly circumvent the problem: we use the landing page of a document in a given archive as our uri variable for linked data purposes. we can now predicate over this variable, and say things like that it contains 34 sentences, that it contains information about woodpeckers, and that the 2nd morpheme of the 4th word in the 3rd sentence is glossed as acc. interested users can look up the page on the archive and request access to the original document if they so wish. one remaining issue is that the semantic structuring of the archive web sites is slightly different. we have the logical levels of "collection", consisting of "bundles/sessions", in turn containing a number of "files", but not all of these have landing pages for each archive or are cross-linked in an obvious manner with a transparent url. we are currently working on a resolver service which will allow the uris used in rdf predicates to be resolved to the urls in the endangered language archives. ## evaluation collections typically contain one language, but some projects working in multilingual settings have more than one language. on the other hand, some languages are found in more than one archive. still, we can say that the number of collections is a good approximation of the number of different languages we know have structured and annotated data for. for 4 100 files, at least one concept could be retrieved. in total, 34 336 concepts were retrieved from the texts, of which 8 457 are distinct. as for lgr abbreviations, there are 438 874 instances of 80 distinct types (there are 84 abbreviations listed in the lgr). 9 outlook @xcite describe progress in the database of cross-linguistic colexifications (clics ). they set up a data format following cross-linguistic data formats (cldf, @xcite ) and detail how other projects can add to their colexifcation data in a well-defined workflow. colexification candidates generated from endangered language documentation archives would be a very good candidate for such a workflow. ## conclusion around 90% of all human languages fall into @xcite 's 'group-0' languages with no noteworthy data capable of providing a starting point for nlp application development. in this paper, we have shown that structured data are indeed available from endangered language archives, a resource by and large ignored by the nlp community up to now. to the extent that legal and technical barriers can be overcome, analyzable and annotated data from almost 300 different languages could be downloaded, 20 the current way of retrieving tla metadata does not allow a grouping by collection, but grouping by sessions is possible.