Data-driven Model Generalizability in Crosslinguistic Low-resource Morphological Segmentation (2022)

Abstract: Common designs of model evaluation typically focus on monolingual settings, where different models are compared according to their performance on a single data set that is assumed to be representative of all possible data for the task at hand. While this may be reasonable for a large data set, this assumption is difficult to maintain in low-resource scenarios, where artifacts of the data collection can yield data sets that are outliers, potentially making conclusions about model performance coincidental. To address these concerns, we investigate model generalizability in crosslinguistic low-resource scenarios. Using morphological segmentation as the test case, we compare three broad classes of models with different parameterizations, taking data from 11 languages across 6 language families. In each experimental setting, we evaluate all models on a first data set, then examine their performance consistency when introducing new randomly sampled data sets with the same size and when applying the trained models to unseen test sets of varying sizes. The results demonstrate that the extent of model generalization depends on the characteristics of the data set, and does not necessarily rely heavily on the data set size. Among the characteristics that we studied, the ratio of morpheme overlap and that of the average number of morphemes per word between the training and test sets are the two most prominent factors. Our findings suggest that future work should adopt random sampling to construct data sets with different sizes in order to make more responsible claims about model evaluation.

## introduction in various natural language processing (nlp) studies, when evaluating or comparing the performance of several models for a specific task, the current common practice tends to examine these models with one particular data set that is deemed representative. the models are usually trained and evaluated using a predefined split of training/test tests (with or without a development set), or using cross-validation. given the results from the test set, authors typically conclude that certain models are better than others @xcite , report that one model is generally suitable for particular tasks @xcite @xcite , or state that large pretrained models have ''mastered'' linguistic knowledge at human levels @xcite . consequently, the ''best'' models, while not always surpassing other models by a large margin, are subsequently cited as the new baseline or adopted for state-of-the-art comparisons in follow-up experiments. these new experiments might not repeat the comparisons that were carried out in the work that identified the best model(s). this model might be extended to similar tasks in different domains or languages, regardless of whether these new tasks have characteristics comparable to those where the model was demonstrated to be effective. that being said, just because a self-proclaimed authentic chinese restaurant is very good at cooking orange chicken, does not mean that its orange chicken will be delicious on every visit to the restaurant. similarly, common implementations of model comparisons have been called into question. recent work by @xcite (see also @xcite and @xcite ) analyzed a set of the best part-of-speech (pos) taggers using the the wall street journal (wsj) from the penn treebank @xcite . they demonstrated inconsistent model performance with randomly generated splits of the wsj data in contrast to the predefined splits as used in previous experiments @xcite . others (see @xcite for a review) have argued that the majority of model evaluations are conducted with data drawn from the same distribution-that is, the training and the test sets bear considerable resemblance in terms of their statistical characteristics-and therefore these models will not generalize well to new data sets from different domains or languages @xcite @xcite . in other words, to evaluate the authenticity of a chinese restaurant, one would need to not only try the orange chicken, but also order other traditional dishes. previous work on model evaluation, however, has focused on high-resource scenarios, in which there is abundant training data. very little effort has been devoted to exploring whether the performance of the same model will generalize in low-resource scenarios @xcite ). concerns about model evaluation and generalizability are likely to be even more fraught when there is a limited amount of data available for training and testing. imagine a task in a specific domain or language that typically utilizes just one specific data set, which is common in low-resource cases. we will refer to the data set as the first data set. when this first data set is of a reasonable size, it may well be the case (though it might not be true) that the data is representative of the population for the domain or language, and that the training and the test sets are drawn more or less from the same distribution. the performance of individual models and the rankings of different models can be expected to hold, at the very least, for new data from the same domain. now unfold that supposition to a task in a specific domain or language situated in a low-resource scenario. say this task also resorts to just one particular data set, which we again call the first data set. considering the limitation that the data set size is relatively small, one might be skeptical of the claim that this first data set could represent the population distribution of the domain or language. accordingly, within the low-resource scenario, model evaluations obtained from just the first data set or any one data set could well be coincidental or random. these results could be specific to the characteristics of the data set, which might not necessarily reflect the characteristics of the domain or language more broadly given the small size of the data. this means that even within the same domain or language, the same model might not perform well when applied to additional data sets of the same sizes beyond the first data set, and the best models based on the first data set might not outperform other candidates when facing new data. continuing the chinese restaurant analogy, the same orange chicken recipe might not turn out well with a new shipment of oranges. as an empirical illustration of our concerns raised above, this paper studies model generalizability in crosslinguistic low-resource scenarios. for our test case, we use morphological segmentation, the task of decomposing a given word into individual morphemes (e.g., @xmath0. in particular, we focus on surface segmentation, where the concatenation of the segmented sub-strings stays true to the orthography of the word @xcite . 1 this also involves words that stand alone as free morphemes (e.g., @xmath1. leveraging data from 11 languages across six language families, we compare the performance consistency of three broad classes of models (with varying parameterizations) that have gained in popularity in the literature of morphological segmentation. in each of the low-resource experimental settings (section 6.1), we ask: ## prior work: model generalizability a number of studies have noted inconsistent performance of the same models across a range of generalization and evaluation tasks. for example, @xcite demonstrated that the same pretrained language models, when paired with classifiers that have different initial weights or are fine-tuned with different random seeds, can lead to drastically unstable results @xcite . this pattern seems to hold for both synthetic @xcite and naturalistic language data @xcite . using approximately 15,000 artificial training examples, lake and baroni (2018) showed that sequence-to-sequence models do not generalize well with small data sets (see also @xcite . through a series of case studies, @xcite demonstrated that model performance showed a different picture depending on the amount of computation associated with each model (e.g., number of iterations, computing power). they suggested that providing detailed documentation of results on held-out data is necessary every time changes in the hyperparameters are applied. another line of research has focused on testing the generalizability of large language models with controlled psycholinguistic stimuli such as subject-verb agreement @xcite , filler-gap dependencies @xcite , and garden-path sentences @xcite , where the stimuli data share statistical similarities with the training data of the language models to different degrees @xcite @xcite . this targeted evaluation paradigm @xcite tends to compare model performance to behaviors of language users. it has been claimed that this approach is able to probe the linguistic capabilities of neural models, as well as shed light on humans' ability to perform grammatical generalizations. thus far, related studies have mainly investigated english, with notable exceptions of crosslinguistic extensions to other typologically diverse languages, such as hebrew @xcite , finnish @xcite , and mandarin chinese @xcite . as fruitful as the prior studies are, they are mostly targeted towards evaluating models that have access to a large or at least reasonable amount of training data. in contrast, there is, as yet, no concrete evidence regarding whether and to what extent models generalize when facing limited data. additionally, although recent work has proposed different significance testing methods in order to add statistical rigor to model evaluations @xcite @xcite , most tests make assumptions on the distributional properties or the sampling process of the data (e.g., data points are sampled independently @xcite ), and a number of these tests such as bootstrapping are not suitable when data set size is extremely small due to lack of power (see also @xcite . ## morphological surface segmentation why morphological surface segmentation? first, previous work has demonstrated that morphological supervision is useful for a variety of nlp tasks, including but not limited to machine translation @xcite , dependency parsing @xcite , bilingual word alignment @xcite , and language modeling @xcite . for languages with minimal training resources, surface or subword-based segmentation is able to effectively mitigate data sparsity issues @xcite . for truly low-resource cases, such as endangered and indigenous languages, morpheme-level knowledge has the potential to advance development of language technologies such as automatic speech recognition @xcite in order to facilitate community language documentation. second, information about morphological structures is promising for language learning. especially for indigenous groups, prior studies have proposed incorporating morphological annotation into the creation of online dictionaries as well as preparing teaching materials, with the goal of helping the community's language immersion programs @xcite . third, despite its utility in different tasks, semi-linguistically informed subword units are not as widely used as they might be, since acquiring labeled data for morphological segmentation in general, including the case of surface segmentation, requires extensive linguistic knowledge from native or advanced speakers of the language. linguistic expertise can be much more difficult to find for critically endangered languages, which, by definition @xcite , are spoken by very few people, many of whom are not native speakers of the language. these aforementioned limitations necessitate better understanding of the performance for different segmentation models and how to reliably estimate their effectiveness. ## meet the languages a total of eleven languages from six language families were invited to participate in our experiments. following recently proposed scientific practices for computational linguistics and nlp @xcite , we would like to introduce these languages and their data sets explored in our study ( first we have three indigenous languages from the yuto-aztecan language family (baker, 1997), including yorem nokki (southern dialect) @xcite spoken in the mexican states of sinaloa and sonora, nahuatl (oriental branch) (de @xcite spoken in northern puebla, and wixarika @xcite spoken in central west mexico. these three mexican languages are highly polysynthetic with agglutinative morphology. the data for these languages was originally digitized from the book collections of archive of indigenous language. data collections were carried out and made publicly available by the authors of @xcite given the narratives in their work. next are four members from the indo-european language family: english, german, persian, and russian, all with fusional morphological characteristics. the data for english was provided by the 2010 morpho challenge @xcite , a shared task targeted towards unsupervised learning of morphological segmentation. the german data came from the celex lexical database @xcite and was made available by @xcite . the persian data from ansari et al. (2019a) (see also ansari et al., 2019b ) contains crowd-sourced annotations, while the data for russian was extracted from an online dictionary @xcite . the remaining languages each serve as a representative of a different language family. the data sets for turkish from the turkic language family and finnish from the uralic language family were again provided by the 2010 morpho challenge @xcite . the data collection for both zulu and indonesian was carried out by @xcite . the zulu words were taken from the ukwabelana corpus @xcite , and the segmented indonesian words were derived via application of a rule-based morphological analyzer to an indonesian-english bilingual corpus. 2 all four of these languages have agglutinative morphology. ## meet the models we compared three categories of models: sequence-to-sequence (seq2seq) @xcite , conditional random field (crf) @xcite , and morfessor @xcite . ## discussion and conclusion using morphological segmentation as the test case, we compared three broad classes of models with different parameterizations in crosslinguistic low-resource scenarios. leveraging data from 11 languages across six language families, our results demonstrate that the best models and model rankings for the first data sets do not generalize well to other data sets of the same size, though the numerical differences in the results of the better-performing model alternatives are small. in particular, within the same experimental setting, there are noticeable discrepancies in model predictions for the first data set compared to the averages across all data sets; and the performance of each model alternative presents different yet significant score ranges and variances. when examining trained models on new test sets, considerable variability exists in the results. the patterns described above speak to our concerns raised at the beginning, namely, that when facing a limited amount of data, model evaluation gathered from the first data set-or essentially any one data set-could fail to hold in light of new or unseen data. to remedy the observed inconsistencies in model performance, we propose that future work should consider utilizing random sampling from initial data for more realistic estimates. we draw support for this proposal from two patterns in our study. first, in each experimental setting, the difference between the average f1 score of the first data set and that across all 50 data sets is in general smaller than the range of the scores. second, the average metric scores across unseen test sets of varying sizes in every setting are comparable to each other; this holds for models trained from random splits and those trained using adversarial splits. therefore, depending on the initial amount of data, it is important to construct data sets and new test sets of different sizes, then evaluate models accordingly. even if the size of the initial data set is small, as is the case with most endangered languages, it is worthwhile to sample with replacement in order to better understand model performance. based on these observations, it would be interesting to analyze the potential factors that lead to the different degrees of generalization, which could in turn provide guidance on what should be included or sampled in the training sets in the first place. thorough compar-isons of different models should be encouraged even as the amount of data grows, particularly if the experimental settings are still situated within low-resources scenarios for the task at hand @xcite . lastly, while we did not perform adversarial splits for all experimental settings here, for the cases that we have investigated, model performance from adversarial splits also yields high variability both across data sets of the same size and in new test sets. though our criteria for heuristic splits were not applicable with the data explored, we would like to point out that for future endeavors, it would be necessary to at least check the applicability of different data splits, as we did in section 7.4. as long as the data set possesses the properties that allow it to be divided by different split methods, it is worthwhile to further explore the influence of these splits, coupled with random sampling.