Resisting the Lure of the Skyline: Grounding Practices in Active Learning for Morphological Inflection (2024)

Abstract: Active learning (AL) aims to lower the demand of annotation by selecting informative unannotated samples for the model building. In this paper, we explore the importance of conscious experimental design in the language documentation and description setting, particularly the distribution of the unannotated sample pool. We focus on the task of morphological inflection using a Transformer model. We propose context motivated benchmarks: a baseline and skyline. The baseline describes the frequency weighted distribution encountered in natural speech. We simulate this using Wikipedia texts. The skyline defines the more common approach, uniform sampling from a large, balanced corpus (UniMorph, in our case), which often yields mixed results. We note the unrealistic nature of this unannotated pool. When these factors are considered, our results show a clear benefit to targeted sampling.

## introduction active learning (al) @xcite is a data annotation approach, where the aim is to direct annotation effort at examples that are maximally helpful for model performance. most active learning work in nlp involves pool-based active learning @xcite where a small seed training set is used to create an initial model, and additional examples are selected and annotated from a large pool of unannotated data. several selection strategies exist, including confidencebased @xcite @xcite , diversity-based @xcite @xcite and committee-based approaches @xcite ; these approaches aim to outperform a uniform random selection baseline. al is often advocated as a method to rapidly improve model performance in low-resource settings @xcite ; the seed, lmc(lemma,msd), lmc(wordform, msd) and tmc experiments. the maroon lines mark the frequency stratified (baseline) accuracy, and the blue lines mark the uniform sampling (skyline) accuracy. @xcite , where limited annotation capacity needs to be directed intelligently. nevertheless, al performance is inconsistent in practice and both success-stories and failures are reported in the literature @xcite @xcite , demonstrating that it is non-trivial to beat a uniform random selection baseline. language documentation is a natural application for active learning. approximately half the world's languages face the grim forecast of extinction, with around 35-42% of these still substantially undocumented @xcite @xcite @xcite . however, data for training automated systems is often limited, and additional annotation bears a high opportunity cost, limited not only by resources but also native speaker availability. ## data we conduct experiments on the unimorph database of inflection tables @xcite foot_0 on a typologically diverse set of eight languages: arapaho (arp), finnish (fin), georgian (kat), quechua (que), sakha (sah), turkish (tur), tuvan (tyv) and zulu (zul). our choice of languages is motivated by a balance between morphological complexity, data availability (both unimorph and wikipedia) and endangerment classification according to unesco atlas of the world's languages in danger (moseley, 2010). where possible, we have attempted to maximise the diversity of our subject languages. across 8 languages, 6 language families foot_1 . further, three of the languages considered (sah, tyv and arp) are considered endangered. we exclusively include adjectives and nouns in our experiments. 3 this simplifies analysis while still representing substantial morphological diversity as nouns make up a sizable portion of text cross-linguistically @xcite . to model word frequencies, we extract the wikipedias for each language and form the intersection of word types present in unimorph (u) and wikipedia (w): u ∩ w. we also retain the much larger part of the unimorph database u \ w, representing types not found in the wikipedia. data sampling is visualized in figure forms, respectively, from u ∩ w (with their uni-morph lemmata and msds). for each language, we additionally supplement the seed training set with four complete inflection paradigms extracted from u \ w to ensure that all inflections are covered by the seed training data foot_4 . from the remaining types in u \ w, we then sample 1,000 for testing. thus, we ensure that there is no overlap between the data splits. using each of the different active learning selection strategies presented below in section 3, we sample an additional 500 training examples. for our baseline method, those are sampled from u ∩ w, while for all the other methods, additional data comes from u \ w. ## experimental setup we perform experiments on the word inflection task @xcite with datasets consisting of triplets ⟨lexeme, msd, inflected form⟩, e.g. ⟨smile, v;pst, smiled⟩. models are trained to predict the correct inflected form based on the lemma and msd. we train transformer @xcite inflection models using fairseq @xcite . 6 in all experiments, we apply data augmentation using the lemma-copy mechanism @xcite . we initially train models on the seed training set and use various sampling strategies to select 500 additional examples from the unused pool, evaluating the change in inflection performance when training on the augmented set. the test and development sets, disjoint with all training data, remain unchanged through this process. we experiment with the following strategies: frequency stratified (baseline) we use word frequencies from wikipedia to perform weighted random sampling from the pool u ∩ w. this method serves as a linguistically motivated, realistic baseline, accounting for the zipfian nature of language, and approximating realistic lexical diversity and the naturalistic distribution of inflected forms. wiki uniform we additionally report results on a baseline which samples from u ∩ w without frequency weighting. uniform sampling (skyline) our second baseline (which we call skyline, as it is near-unbeatable) uses uniform sampling without word frequency information from u \ w. this setting is unrealistic in a language documentation setting-due to the lexical diversity and balanced class distribution of the samples, rare paradigm slots are over-represented. oracle inspired by muradoglu and hulden (2022), we sample forms which the model fails to inflect correctly. since this requires knowledge of gold standard forms, the method can only be used for comparison. this strategy mimics feedback from a linguist or language expert. in many cases, there are more than 500 incorrectly inflected forms to choose from. when this happens, we select maximally erroneous examples, that is, the examples with the greatest levenshtein distance to the gold standard form. foot_6 in contrast, when there are fewer than 500 incorrectly inflected forms, we augment the set using correctly inflected forms with the lowest confidence. transformer model confidence (tmc) again following @xcite , we train an initial inflection model on the seed training set. we use this model to make predictions and select the examples with the lowest confidence scores. ## results and discussion it is surprising that the oracle, intended to mimic a language expert, is outperformed by the either the tmc or lmc(wordform, msd) strategies for six of the languages considered. for almost all of the languages examined, the levenshtein distance is the primary weighing factor foot_9 . the edit distance fails to consider the diversity of vocabulary or msd. compound words can also skew the levenshtein distance significantly. for example, for the turkish compound otomatik bilet makinası ("automatic ticket machine"), if the model does not capture the space between otomatik and bilet, though characters are merely shifted to the left, the levenshtein distance is artificially high. we seek to understand the effects of the various sampling strategies by estimating the relative edit diversity for each sample. for each dataset, we enumerate the edits (insertion, deletion, or replacement of subwords) needed to transform each lexeme to the inflected word. we collect edits of the same type and subword to give an edit distribution. using this distribution, we compute entropy, which is higher for a distribution with a more diverse set of edits, and lower when the dataset is dominated by a few frequent edits. we provide the entropy, relative to the base training set, in figure 3 . ## conclusion computational methods can aid language documentation and description projects by processing and analyzing recorded data. active learning approaches can greatly aid in the rapid development of robust automated systems by focusing annotation on highly beneficial samples, but existing research on simulated al often makes unrealistic assumptions. we compare a standard approach (skyline), where data is sampled from unrealistic linguistic resources, an approach based on naturalistic word frequencies (baseline), and a number of strategies motivated by encouraging lexical diversity. our skyline and baseline approaches serve as analogs to elicitation and naturalistic recording. we find that the skyline approach is difficult to beat, but as few languages have sufficient corpora with complete, diverse paradigms, we argue this approach is an unrealistic baseline for al. meanwhile, we find clear benefits from targeted sampling strategies, with inflection model confidence (tmc) and character lm scores (lmc(wordform, msd)) yielding the greatest improvements.