A Word-and-Paradigm Workflow for Fieldwork Annotation (2022)

Abstract: There are many challenges in morphological fieldwork annotation, it heavily relies on segmentation and feature labeling (which have both practical and theoretical drawbacks), it’s time-intensive, and the annotator needs to be linguistically trained and may still annotate things inconsistently. We propose a workflow that relies on unsupervised and active learning grounded in Word-and-Paradigm morphology (WP). Machine learning has the potential to greatly accelerate the annotation process and allow a human annotator to focus on problematic cases, while the WP approach makes for an annotation system that is word-based and relational, removing the need to make decisions about feature labeling and segmentation early in the process and allowing speakers of the language of interest to participate more actively, since linguistic training is not necessary. We present a proof-of-concept for the first step of the workflow, in a realistic fieldwork setting, annotators can process hundreds of forms per hour.

## introduction a major component of current workflows for linguistic fieldwork is the creation and curation of interlinear glossed texts (igt), in which morphological forms are segmented into meaning-bearing units. these are expensive and time-consuming to produce, but constitute important training data for computational fieldwork methods. while igt are a valuable resource for the study of endangered and under-described languages @xcite , annotations that directly segment and label morphemes may have both practical and theoretical shortcomings. segmentation-based analyses may not always straightforwardly account for the diversity of phenomena attested in the world's languages, making them especially problematic in the early stages of understanding a morphological system. an alternative is provided by analyses that characterize morphological relations at the word level, such as those associated with word-and-paradigm approaches (wp; named by @xcite ; see @xcite for a general overview), which do not require segmentation and may allow for more efficient and informative morphological annotation in a low-resource fieldwork setting. wp theories classify word forms in terms of the shared relationships they exhibit within a connected lexicon. these relationships may be conceptualised as tabular paradigms in which one axis groups items sharing a lexeme and the other groups items sharing a morphosyntactic cell. note that paradigmatic tables would still be informative about the identities of morphosyntactic cells even without cell labels. such an unlabeled table can be assembled first and then serve as an aid for post-hoc decisions about how to label the contrasts. wp-style analyses inform recent work on unsupervised paradigm discovery @xcite @xcite as well as neural inflection and reinflection models without internal segmentation @xcite @xcite . since wp theories have proven to be such a good fit for "big data" morphology @xcite , this paper asks whether they can also benefit the "small data" fieldwork setting. we see several potential advantages: modern computational tools can be used to provide initial analyses or suggestions for the annotator; grouping forms together as belonging to the same cells or lexemes may be faster and easier than segmenting on a first pass; finally, segment-free annotations of some morphological phenomena may be preferable on theoretical grounds. we conduct pilot experiments in three languages, including a true underresourced language, to show that trained human annotators can rapidly improve on the results of an unsupervised morphological analyzer @xcite . the workflow we propose takes these corrections as input to bootstrap iterated active learning. we collaborate with non-linguist native speakers of wao terero, a language isolate spoken in ecuador, to evaluate the potential of the proposed methodology to increase community engagement in the annotation process. finally, we discuss possible next steps in the design of an interactive annotation environment for word-and-paradigm morphology. ## workflow and experiments we propose an annotation workflow (figure 1 ) that begins with the collaborative collection of primary data within a language community. for morphological analysis, the model makes use of both naturalistic transcriptions or texts as well as a list of target lemmas for analysis. these files are used as input to an unsupervised model that identifies potential instances of lexemes and cells and outputs a sample of occurrences for each category to be annotated. the annotation process involves both excluding occurrences that don't belong in their assigned group and seeking out new occurrences to add to the group. the annotated output could then be used along with additional primary data as input to a supervised classifier within an active learning framework. since the workflow is modular, each individual component can be updated over time in line with future advances in state of the art machine learning. as a proof of concept that wp annotation is viable, we implement the first stage of our proposed workflow in which fieldworkers begin with the results of transcribed language data target lemma list unsupervised wp model grouped cell and lemma occurrences annotator corrections output lexeme and cell clusters wp active learning an unsupervised analysis. we conduct a series of pilot experiments demonstrating that annotators can rapidly improve the results of such a system. while our full proposed workflow would integrate active learning within an interactive annotation environment, in the present work, we do not currently use these annotations as additional input to the model. ## results ## discussion a more relational, word-based approach to morphological annotation for language documentation is desirable for both theoretical and practical reasons. theoretically, word-and-paradigm annotation allows fieldworkers to avoid, or at least defer, difficult decisions about both morphological form and function. our proposal separates the identification of a morphological cell from the application of a morphosyntactic label or set of features, a difficult analytical task often involving comparison of many related examples. for instance, distinguishing past tense from perfect aspect is a sensitive task @xcite ) which might best be done once the forms in question can be reliably separated from the rest of the verbal paradigm. thus, even if the desired end goal of annotation is igt, we believe that our proposed annotation methodology can speed up the early stages of annotation and prevent the researcher from having to commit to an analysis too early. from a practical standpoint, by starting with an automatically generated concordance set for each proposed cell and lexeme, the annotator can focus on making direct comparisons -is the proposed grouping coherent in terms of its surface form and distributional context? annotating at the level of unsegmented words in context makes it possible for a native speaker who has no knowledge of technical or grammatical terms to easily identify the patterns represented by the concordances and contribute their expert knowledge. the proposed workflow is therefore designed to facilitate greater community involvement in the development of language resources and technologies in line with community needs. our experiment results show that even in an unfamiliar language for which the annotator does not yet understand the functions of different morphological markers, there is still some capacity to weed out forms that do not belong. the tight integration of this workflow with unsupervised learning technology also means that any future improvements in unsupervised paradigm discovery can immediately benefit fieldworkers. while our pilot experiment focuses on the initial steps of a computationally-aided field documentation project, our full proposed workflow envisions both an interactive environment and active learning. an interactive environment would allow annotators to view proposed paradigm tables alongside the text, helping to see where forms might be missing. annotators could also search more effectively for forms that could fill in these gaps, for instance by viewing a word cloud of similar forms. this could make it easier to fix recall errors in system-proposed paradigms. even a few labeled instances can vastly improve the performance of part of speech taggers @xcite , and the same is true in a relational setting @xcite . in our environment, we envision active learning directing the annotator's attention to the least certain distinctions, eliminating repetitive annotation of "easy" instances. our proposal shows promise as a faster and more theoretically grounded alternative to existing tools. its modular structure makes it easy to integrate advances in the field of computational linguistics, and it allows the fieldworker to quickly and easily involve the intuitions of native speakers with little linguistic training, boosting community engagement.