Theoretical and Practical Issues in the Semantic Annotation of Four Indigenous Languages (2021)

Abstract: Computational resources such as semantically annotated corpora can play an important role in enabling speakers of indigenous minority languages to participate in government, education, and other domains of public life in their own language. However, many languages – mainly those with small native speaker populations and without written traditions – have little to no digital support. One hurdle in creating such resources is that for many languages, few speakers would be capable of annotating texts – a task which requires literacy and some linguistic training – and that these experts’ time is typically in high demand for language planning work. This paper assesses whether typologically trained non-speakers of an indigenous language can feasibly perform semantic annotation using Uniform Meaning Representations, thus allowing for the creation of computational materials without putting further strain on community resources.

## introduction over the last few decades, there has been a call to enable speakers of indigenous minority languages to participate in government, education, and other domains of public life in their own language. computational resources can play an important role in such efforts @xcite . for example, semantically annotated corpora for minority languages can be used for information extraction to obtain situational awareness in disaster situations @xcite , to link unstructured text in various languages to structured knowledge bases @xcite , and as scaffolding for machine translation into these languages . as of 2019, only 1705 out of the 7795 languages in @xcite , 22%, had any digital support. even for languages with large native speaker populations and considerable political standing such as farsi, computational resources are often limited @xcite . the limited availability of (digital) data in such minority languages is only one hurdle to the creation of computational resources. wherever data are available, they need to be provided with semantic annotations in order to be made maximally useful for the purposes described above. semantic annotation allows unstructured text to be linked to representations such as abstract meaning representations (amr, @xcite or discourse representation structures (drs, @xcite . such annotation schemes have become more crosslinguistically informed over the years. the darpa low resource languages for emerging incidents project (lorelei), for one, has conducted shared annotation tasks with languages such as tagalog, yoruba and somali @xcite . the uniform meaning representation project, on the other hand, aims to make english-based amr crosslinguistically applicable @xcite . in practice, however, current annotation workflows have little chance of being applied to truly "no-resource" languages. semantic annotation is typically done by speakers of the target language, as it is assumed that (native) speaker intuitions are necessary to make judgments required for semantic annotation. this may be feasible for "low-resource" languages with millions of speakers such as oromo, tigrinya, uyghur, and ukrainian -the "incident languages" in @xcite . for many others, including most of the 1500 languages with fewer than 1000 speakers @xcite , such annotators are unlikely to be available for several reasons (see section 2). this paper therefore has two main goals. firstly, it assesses whether the structure of umr indeed makes it scalable to languages with a different typological profile than traditionally well-represented languages in nlp such as english and mandarin. secondly, it assesses whether non-speakers of an indigenous language trained in typological linguistics can successfully perform umr semantic annotation of such languages based on morpheme-level glosses, utterance-level free translations, grammars, and dictionaries. specifically, we present quantitative results of two annotation experiments using umr to annotate texts in kukama (tupían, peru), and arapaho (algonquian, us), and qualitative results of initial annotation efforts with sanapaná (enlhet-enenlhet, paraguay) and navajo (athabaskan, us). these four languages were chosen because (1) they represent a range of resource availability from noresource (sanapaná) to low-resource (arapaho, see sections 2.1-2.4), (2) they are typologically diverse, representing more isolating (kukama), agglutinating (sanapaná), and polysynthetic (arapaho, navajo) types, and (3) co-authors of this paper have significant expertise in them. in section 2, the advantages of a workflow using linguistically trained non-speakers as annotators are laid out, and its necessity is illustrated through sociolinguistic sketches of the four languages at hand. section 3 introduces umr. section 4 presents an overview of theoretical issues relating to the umr guidelines encountered during the annotation of these four languages. sections 5-6 present the inter-annotator agreement and adjudication results of the kukama and arapaho annotation experiments. section 7 presents an overview of difficulties with the annotation workflow used in these two experiments. ## limitations on current semantic annotation procedures regardless of how large a corpus is available and how typologically informed an annotation scheme is, trained and qualified annotators are vital to successful annotation. these are often hard to come by in low-resource language contexts. linguists working on such languages have the training required to familiarize themselves with semantic values and concepts used by annotation schemes. since they are often still in the process of studying the language, however, their semantic judgments may be less than reliable. native speakers can provide the most accurate interpretations of the meanings of forms in their language. from this point of view, they make "better" annotators. however, they vary with respect to various "literacies". according to @xcite , 3982 of the world's languages -slightly over half -have a writing system. however, only 15-20% of languages have an actual written tradition @xcite . therefore, native speaker consultants often have limited literacy in their own language, although they may have literacy in regional or national linguae francae. many low-resource languages are spoken in remote areas where people may not yet have easy access to or familiarity with digital technology. they are also likely to have more limited access to formal linguistics training than the national societies surrounding them. in contexts of advanced language shift (which many indigenous language-speaking communities face), these factors may form correlated continua, and may be related to age. there may be a continuum of multilingualism so that younger speakers are more likely to be dominant in a lingua franca, while older generations are near-monolingual in the local language. similarly, the speakers most fluent in an endangered language are likely to have gone through less formal education, be less literate, and be less comfortable with digital tools. these factors conspire to create a situation where few speakers of low-resource indigenous languages are currently well-equipped to conduct semantic annotation projects on their language. furthermore, wherever there are qualified local language experts, their time is typically in high demand for language planning projects such as curriculum development and teaching, which are typically given higher priority in language maintenance efforts. the following brief sociolinguistic sketches of kukama, arapaho, sanapaná, and navajo illustrate these points. ## uniform meaning representation (umr) uniform meaning representation (umr) is a semantic annotation scheme designed to meet the needs of a wide range of nlp applications which require intermediate meaning representations. umr is based on abstract meaning representation @xcite . amr captures the meanings of natural language sentences as single-rooted, directed, node-and edge-labelled graphs, and focuses on predicate-argument structure, named entities, and word sense disambiguation. it was mainly designed for the annotation of english. the usefulness of amr and its amenability to machine learning have been repeatedly proven @xcite . ## umr applied to four indigenous languages the umr annotation scheme and its guidelines were designed with cross-lingual variation in mind. we tested umr on narrative texts in two indigenous languages, navajo and sanapaná. 107 lines of sanapaná text were annotated by the first author, amounting to 332 sanapaná words and 600 words in the free spanish translation. 261 lines of navajo text were annotated by the third author, amounting to 2044 navajo words and 5020 words in the free english translation. in addition, issues that arose in the kukama and arapaho experiments (sections 5-6) also tested the umr annotation scheme. umr adds semantic structure to amr annotation, specifically aspect, general participant roles, quantifi-cation, scope, and temporal and modal structure (including polarity). we focus on the annotation of those categories here. ## discussion: implications for annotation workflows this paper set out to assess whether a typologically trained linguist can semantically annotate languages they do not speak or otherwise have expertise in, in order to facilitate the creation of computational resources for indigenous languages without having to further burden field linguists or communities. the iaa scores for the experiments discussed in sections 5-6 suggest that this is relatively possible: we were able to reach fairly high agreement between annotators, and between each annotator and an expert in the target language, with a mini-mal amount of practice. in other words, kukama and arapaho morpheme-by-morpheme glosses and free translations, together with reference materials such as a dictionary and grammar, allowed annotators to interpret the meanings of sentences in these languages fairly accurately. however, the types of disagreements between annotators, and between annotators and language experts, point towards a number of ways in which there is still room for improvement regarding the workflow described here. on the one hand, a fair number of disagreements between annotators, and inaccuracies on their part, stemmed from inconsistencies and/or gaps in the available materials. for example, in the kukama sentence in figure 1 , the spanish and english free translations are not internally consistent -the english translation does not contain the word 'almost', and in the spanish version, the word for 'wanted' is between brackets to indicate the wanting is implied. even though @xcite details the use of iyara as both an adverb meaning 'almost' and a desiderative marker, these inconsistent translations would lead annotators to make different decisions. issues like these are likely to be quite frequent when working from materials collected in language documentation projects. these materials are often collected over multiple years and by multiple people, and analyses often change throughout these periods and are not always automatically updated in materials collected earlier. other disagreements and inaccuracies stemmed from notational and content-based common practice in language description and grammar writing. for instance, meanings of derived stems or compounds are often not transparently clear from the sum of the glosses of the component parts: the kukama stem 1ts1-kaka is glossed in the annotated text as 'be scared-rec', but is listed in @xcite ) as one of a set of verbs with which the reciprocal marker takes on an inchoative function ('become scared'). since the transcriber/translator of the text opted for 'idiomatic' translations on the utterance-level rather than a more 'literal' translation ('my daughters call scared,' rather than something like 'my daughters call, having become scared'), such nuances of meaning slip between the cracks of the morphemelevel gloss and the utterance-level translation. this lost information can have important ramifications for annotation choices, in this case for the aspect annotation of this predicate. regarding content, many arapaho noun incorporation and associated motion constructions were difficult to annotate, since umr guidelines for event and participant identification rest on syntactic tests that are not always discussed in detail in reference grammars, such as the ability of a verb with incorporated object to co-occur with an independent np co-referential with this object, @xcite . even though it is unrealistic to expect field linguists to gear their grammar-writing practices specifically towards creating materials with maximal usefulness for semantic annotation, there may be ways in which the impact of these issues can be limited. for example, the issue of meanings slipping between the cracks of morpheme-level glosses and utterance-level translations could partially be remedied by providing an intermediate level of granularity in translations -translations at the word level. many field linguists provide the texts they collect with word-level translations in the early stages of their analysis (before they have a thorough understanding of the morphological structure of the language). for certain types of languages (e.g. languages with complex and irregular morphophonemics, or languages that make extensive use of derivation and compounding with idiosyncratic semantics for word-formation), linguists may continue producing these word-level translations in their databases even at more advanced stages of analysis. these word-level translations are usually not exported when texts are published or cited as examples in linguistics publications. however, providing them to semantic annotators would be likely to significantly improve inter-annotator agreement and overall accuracy. for those aspects of descriptive linguistic practice which are harder to influence, it may be easier to compromise on the side of the annotation workflow. a workflow where annotators are given time to read a grammar of the language and familiarize themselves with its structure before annotating, rather than using the grammar only as a reference work to look up constructions and their functions in an ad-hoc way during annotation is likely to result in higher agreement and accuracy scores. and, rather than hoping that annotators can reach a high enough accuracy to establish gold standard annotations by themselves, allowing for consultation and adjudication with a language expert is likely to remain necessary. ## summary and conclusions this paper has aimed to raise awareness among computational linguists that the languages currently treated in nlp research and shared tasks as lowresource languages, and especially no-resource languages lacking any digital presence to speak of, do not just present problems of scale in comparison to major world languages. they also present qualitatively different challenges, especially regarding the recruitment of qualified annotators. the numerous languages with small speaker numbers and limited resources, which are often spoken in more remote areas, pose unique challenges for creating digitally robust and cross-linguistically comparable semantic annotation projects. nevertheless, this paper showed that workflows can be designed which allow for semantic annotation without placing an undue burden on the limited resources of native speaker communities. semantically annotated corpora developed in this way, and computational resources derived from them, have the potential to both provide benefits to indigenous communities, and to allow us to progress in our understanding of lexical and grammatical semantics.