Eeny, meeny, miny, moe. How to choose data for morphological inflection. (2022)

Abstract: Data scarcity is a widespread problem for numerous natural language processing (NLP) tasks within low-resource languages. Within morphology, the labour-intensive task of tagging/glossing data is a serious bottleneck for both NLP and fieldwork. Active learning (AL) aims to reduce the cost of data annotation by selecting data that is most informative for the model. In this paper, we explore four sampling strategies for the task of morphological inflection using a Transformer model: a pair of oracle experiments where data is chosen based on correct/incorrect predictions by the model, model confidence, entropy, and random selection. We investigate the robustness of each sampling strategy across 30 typologically diverse languages, as well as a 10-cycle iteration using Natügu as a case study. Our results show a clear benefit to selecting data based on model confidence. Unsurprisingly, the oracle experiment, which is presented as a proxy for linguist/language informer feedback, shows the most improvement. This is followed closely by low-confidence and high-entropy forms. We also show that despite the conventional wisdom of larger data sets yielding better accuracy, introducing more instances of high-confidence, low-entropy, or forms that the model can already inflect correctly, can reduce model performance.

## introduction the need for linguistically annotated data sets is a drive that unites many fields within linguistics. computational linguists often use labelled data sets for developing nlp systems. theoretical linguists may utilise corpora for constructing statistical argumentation to support hypotheses about language or phenomena. documentary linguists create interlinear glossed texts (igts) to preserve linguistic and cultural examples, which typically aids in generating a grammatical description. with the renewed focus on low-resource languages and diversity in nlp and the urgency propelled by language extinction, there is widespread interest in addressing this bottleneck. one method for reducing annotation costs is active learning (al). al is an iterative process to optimise model performance by choosing the most critical examples to label. it has been successfully employed for various applications through nlp tasks including deep pre-trained models (bert) @xcite , semantic role labelling (myers and palmer, 2021), named entity recognition @xcite , word sense disambiguation @xcite , sentiment classification @xcite and machine translation @xcite . the iterative nature of al aligns nicely with the language documentation process. it can be tied into the workflow of a field linguist who consults with a language informant or visits a field site in a periodic manner. prior to a field trip, a linguist typically prepares material/questions (such as elicitation's or picture tasks foot_0 ) for language consultants which may focus on elements of the language they are working to describe or for material creation (e.g., pedagogical). we propose al as a method which can provide a supplementary line of insight into the data collection process, particularly for communities that wish to develop and engage with language technology and/or resource building. previous work by palmer (2009) details the efficiency gains from al in the context of language documentation for the task of morpheme labelling. with deep learning models leading performance for the task of morphological analysis @xcite the reported error bars are calculated across 3 separate runs. see ## data we use data from the unimorph project @xcite , interlinear glossed texts (igt) from @xcite and @xcite . in addition to the data availability, we consider typological diversity when selecting languages to include. broadly, we attempt to include types of languages that exhibit varying degrees of complexity for inflection. we also consider morphological characteristics coded in wals; prefixing vs. suffixing @xcite , inflectional synthesis of the verb @xcite and exponence @xcite ). an additional consideration is the paradigm size for the morphological system modelled. we note data source type to account for the variation in standard across wikipedia, igt field data, glossed examples from grammars and data generated from computational grammars. ## experimental setup we train the model as if we were addressing an 'inflection' task @xcite . the data is in the form of triplets: lexeme, morphosyntactic tags and the desired output inflected form (e.g. ⟨walk, v;pst, walked⟩) foot_1 . each model is trained with the fairseq transformer @xcite and our hyperparameters follow @xcite . a baseline model is trained, after which more examples are resampled from the baseline test file using the methods detailed below. the initial baseline model is trained with 3,500 instances, 1,000 test and 500 for development. we resample 250 instances. ## results and discussion to simulate a documentation process, we have chosen natügu as a case study. the inflection data is from @xcite and is derived from igts-a form that is commonly utilised by field linguists. our choice of language is further motivated by the morphological complexity exhibited by natügu. by all accounts natügu showcases complex morphology @xcite , particularly on the verb. historically, this observed complexity led to the language family named as papuan instead of austronesian. additionally, we observe a positive correlation between prediction correctness and model confidence (0.605). in fact, 4 out of the top 8 correlations (as shown in figure 2 ) are languages with igts as a data source. for these reasons, we have chosen to examine iterative sampling over 10 cycles. figure 1 summarises our results for natügu. the re-sampling process is iterated over 10 cycles. the first cycle is the baseline/seed run and consists of a 600 instance training set. to account for the impact of random factors affecting the initial training data selection, we have conducted 3 independent seed runs-differing solely on the initial training set. the average accuracy and corresponding standard deviation is reported with the error bars. 3 the small starting size is motivated by the parallels with language documentation efforts, which are typically a low-resource setting. in each cycle, 250 forms are sampled via the corresponding sampling strategy. by the last cycle the training data consists of 2,850 instances. aside from the 3rd and 10th cycle, the lowest log-likelihood sampling consistently provides the greatest improvement. for these two cycles sampling based on incorrect forms outperforms selection based on low confidence. in general, the top 3 selection methods are ranked as follows: low loglikelihood, incorrect and highest entropy forms. we note the possible interplay between paradigm size (907 unique tag combinations) and training size set 3 individual values can be found in given the strong correlation between prediction accuracy and model confidence for natügu, we expect similarity in trajectory across cycle number and accuracy for the oracle and model-confidence based sampling strategies. figure 1 verifies these forecasts; we see that the sampling based on prediction correctness (in light blue) and the sampling based on the highest log-likelihood (in light green) almost look identical. the same is observable for low log-likelihood (in red) and sampling based on incorrect prediction (green). the lowest log-likelihood sampling method can be seen as an approximation for the highest entropy selection method, and by extension, the highest log-likelihood as an approximation for the lowest entropy selection. our results for iterative al for natügu show that choosing by approximation is a higher risk endevaour. the choice either works really well or not at all. when we contrast low entropy and high model confidence as a selection strategy we can see that low entropy limits the impact of high model confidence since it accounts for a distribution rather than the single value approximation. we observe similar behaviour between the the high entropy and low confidence selection strategies. random sampling shows gradual improvement. work by @xcite highlight the issues with uncertainty sampling for deep learning models; noting that neural networks are poorly calibrated @xcite , and that the correlation between high confidence and correctness is not well established. we explore this correlation for our models in figure 2 . we observe a similar uncertainty with an overall slight positive correlation across the 30 languages examined. despite this, our results show that data selection based on low model confidence yields significant improvement of model accuracy. the work presented here is intended as a preliminary baseline; we leave it to future work to consider calibration methods such as temperature scaling. interestingly, despite an increase in training data size, introducing new data that the model already can inflect correctly, or low-entropy or highconfidence forms actually reduces model performance despite the widely-held notion that more data is better. another recent study by samir and silfverberg (2022) reports similar behaviour, where data hallucination reduces prediction accuracy for words that exhibited reduplication. we extend the same sampling strategies to 30 different languages for one round of re-training. the results are summarised in figure 3 . within the 30 languages we ensure to include languages with large inflection table sizes (ranging from 12 to 700+), different scripts (latin, cyrillic, arabic, hangul, ge'ez and gujarati) and morphological typology (agglutinating, fusional, polysynthetic). we code for the source of the data, and see no particular deviation from the overall observed behaviour. the reported error bars for random sampling correspond to the standard deviation across three independent runs of random sampling. it is clear that in general, the sampling strategies can be ordered for prediction accuracy improvement in the following manner: incorrect, lowest log-likelihood, highest entropy, random, highest log-likelihood, lowest entropy and finally correct form sampling. while a handful of languages deviate from this pattern (e.g. swahili or dido), 4 it 4 see table .3 in appendix for more detail. ## conclusion in this paper we examine four different sampling strategies within an al framework for modelling morphological inflection using a transformer model. we consider correct/incorrect prediction, model confidence, entropy and random selection as sampling strategies. our results clearly show that al can significantly improve learning rates for morphological inflection. unsurprisingly, adding oracle-indicated incorrect forms for training yields the greatest model improvement. in the absence of a language expert, model confidence can be used to prioritise data annotation. this holds true across 30 different languages. we also show that larger datasets do not always yield better results; the diversity of the training set matters. future research should extend the analysis to incorporate language-specific factors-such as model performance for each morphosyntactic slot within the morphological paradigm.