The Effect of Data Partitioning Strategy on Model Generalizability: A Case Study of Morphological Segmentation (2024)

Abstract: Recent work to enhance data partitioning strategies for more realistic model evaluation face challenges in providing a clear optimal choice. This study addresses these challenges, focusing on morphological segmentation and synthesizing limitations related to language diversity, adoption of multiple datasets and splits, and detailed model comparisons. Our study leverages data from 19 languages, including ten indigenous or endangered languages across 10 language families with diverse morphological systems (polysynthetic, fusional, and agglutinative) and different degrees of data availability. We conduct large-scale experimentation with varying sized combinations of training and evaluation sets as well as new test data. Our results show that, when faced with new test data: (1) models trained from random splits are able to achieve higher numerical scores; (2) model rankings derived from random splits tend to generalize more consistently.

## introduction evaluations of computational models in natural language processing typically rely on a single dataset, though there are exceptions, such as highresource languages like english, which have multiple datasets available for specific tasks @xcite @xcite . such datasets are typically provided with one partitioning, including at least a training and test set, with an optional validation set depending on data availability @xcite . the single data split is typically determined by shared task organizers @xcite , benchmark designers @xcite , and ground-breaking paper authors @xcite . the rationale behind the particular data partition, however, is not always clear or mentioned at all (cf. de @xcite ). recent work has called into question the adoption of one data split @xcite @xcite or one dataset @xcite for model evaluation. these previous studies point out that individual model performance as well as model rankings derived from just one single split of training-(validation)-test sets may fail to generalize when applied to an alternative split of the same dataset, or even to new unseen data from the same domain. thus, drawing conclusions based on a single data partition has the potential for being unreliable. this study investigates the effect of different data partitioning methods on model generalizability in cross-linguistic scenarios with varying data availability. we use morphological segmentation as the testbed, i.e., the task of decomposing a word into its component morphemes (@xmath0. while a range of studies have been undertaken to explore the impact of data partitioning strategies on generalizability of model performance (see section 2 for some examples), it is safe to say that no consensus has been reached regarding the specific choices of data partition strategies. this is largely due to the fact that these studies face several important limitations, which we describe below. lack of language diversity first, a considerable portion of prior work @xcite predominantly focuses on english (cf. @xcite ; @xcite ). it is possible, however, that an optimal data partitioning strategy, if one exists, is dependent on the languages (and tasks) under investigation. this is because the typological traits of the languages can have an impact on the distributions of the resulting training/test sets and new unseen data. for example, if languages exhibit greater morphological regularity, alternative data partitioning approaches might yield comparable model performance. lack of multiple datasets and data splits building on the first point, the tasks investigated in previous literature often enjoy ample data availability. for high-resource languages with abundant data for a given task, it is often assumed (implicitly or explicitly) that the selected dataset or data split adequately represents the task or language. therefore, a data partitioning strategy that fares well on the same dataset or data split is expected to yield models that generalize reasonably to new unseen data, particularly from the same domain. however, in scenarios with constraints on data availability, the representativeness of the chosen dataset or data split becomes questionable. what kind of data partitioning strategy is appropriate to apply when facing different extents of data availability thus remains an open question. liu et al. (2023 ) address the aforementioned issues to some extent, but lack evaluation of trained models on new test samples. although søgaard et al. recommend inclusion of multiple test sets, they do not consistently experiment accordingly for each task. lack of model comparisons finally, while gorman and bedrick compare a number of pos taggers, søgaard et al. and liu et al. apply one model for each task. thus, they fail to provide a detailed analysis of model rankings and how these rankings may be affected by different partitioning strategies. it remains unclear if these rankings would still hold when considering new test samples. taking a data-driven approach, this work transcends that of prior approaches in several respects: • we attend to a typologically diverse set of 19 languages from ten language families, covering polysynthetic, fusional, and agglutinative morphological systems. these languages have different amounts of data available pertaining to morphological segmentation. in addition, ten of these languages are indigenous or endangered languages, painting a typologically rich set of language samples for our study. ## experiments this section introduces our experimentation to investigate the impact of different data partitioning strategies on model generalizability for morphological segmentation. we first present the details regarding the data used in our experiments, including the languages/families contained therein. we then explore the dataset construction process and describe the model architectures applied. ## analysis our analysis seeks to address two questions: ## conclusions this study investigates the impact of data partitioning strategies on model generalizability, using morphological segmentation as a test case, drawing data from 19 typologically diverse languages, including ten indigenous/endangered languages. our results demonstrate that, independent of the morphological properties of the languages, random splits, in contrast to adversarial splits, yield: (1) better model performance, and (2) more reliable model rankings on new test data. these patterns hold across varying sized combinations of training and eval sets, as well as new test samples, as evidenced by the minimal variation in model performance across datasets for the languages examined. the findings are also supported by our statistical regression analysis, where random splits are shown to have a pronounced positive impact on model performance; this pattern holds for most languages, in spite of the fact that each individual language has a different dataset size. it is worth noting that the average f 1 score differences between random and adversarial splits on the new test samples are much larger across model architectures, when the new test samples are generated adversarially, in comparison to when they are derived randomly (section 4.1). what's more, the trend that random splits yield more consistent model rankings is stronger when facing adversarially constructed new test samples as well (section 4.2). recall that adversarial samples are posed to be as distant from the training data as possible. these tendencies suggest that when facing more challenging new test data in the wild (challenging relative to the training data), there is potentially more benefit in applying random splits, at least in the case of morphological segmentation. while random splits seem to outperform adversarial splits in our study, we do not wish to draw the same conclusions for other tasks. with the methodologies outlined here, for future work, we would like to expand to different tasks from a crosslinguistic angle. in particular, we are interested in settings where the languages have a spectrum of data availability, in order to probe what data partitioning strategy will be preferred given different extents of data limitation. in addition, while heuristic splits are not plausible in our experiments, in future cases where applicable (e.g., text classification where the dataset can possibly be split heuristically based on the number of tokens or token types in the sentences), there is potential value in including such splits for more thorough comparisons.