C luster D ata S plit: Exploring Challenging Clustering-Based Data Splits for Model Performance Evaluation (2020)

Abstract: This paper adds to the ongoing discussion in the natural language processing community on how to choose a good development set. Motivated by the real-life necessity of applying machine learning models to different data distributions, we propose a clustering-based data splitting algorithm. It creates development (or test) sets which are lexically different from the training data while ensuring similar label distributions. Hence, we are able to create challenging cross-validation evaluation setups while abstracting away from performance differences resulting from label distribution shifts between training and test data. In addition, we present a Python-based tool for analyzing and visualizing data split characteristics and model performance. We illustrate the workings and results of our approach using a sentiment analysis and a patent classification task.

## introduction in natural language processing (nlp), the standard approach for tuning and selecting machine learning models is by means of using a held-out development set. however, recent work has pointed out that evaluation scores on a development set are often not indicative of the model performance on an unseen test set @xcite . in addition, it is an open research question how to choose a good development set. while @xcite suggest to use random splits instead of a given benchmark development set, @xcite argue that randomly selecting a development set is not the best option either. this currently ongoing discussion in the nlp community highlights the need for more extensive research on model development using a variety of data splits. in this paper, we directly add to this discussion by proposing a strategy in which models are evalu-ated in a setup that is challenging given the available dataset. for machine learning models, it is of utmost importance that they are applicable to different data distributions, possibly even coming from different domains. we argue that models should also be tested under data splits reflecting such real-world settings. therefore, we propose a clustering-based data splitting approach that creates data splits where the development or test data differ from the training set. our clustering algorithm ensures a similar label distribution across the produced cross-validation folds in order to abstract away from challenges due to label distribution shifts. in addition, we present clusterdatasplit, a suite of jupyter notebooks implementing several possibilities for splitting data into training and development sets, or into folds for cross-validation. in addition, our tool provides functionalities for visualizing different data splits and thus may help clarifying their influence on model performance. furthermore, it offers several ways to inspect the data, such as visualization of dataset key figures, scatter plots, label distributions and sentence length distributions. the tool is publicly available. 1 in sum, our contributions are as follows: (i) we propose a clustering-based data splitting algorithm that creates a challenging evaluation setup and has the potential to reveal difficulties when the model is applied on data that deviates from the training data (section 3). (ii) we present clusterdatasplit, a tool that allows to split data into training and development sets and provides different visualizations for analyzing the data splits as well as model performance (section 4). finally, we demonstrate a worked example of using our data inspection tool as well as results for our clustering-based data splitting methods for two sequence classification tasks, sentiment analysis and patent classification (section 5). ## related work in this section, we give an overview of related work on evaluation and data splitting techniques as well as analysis tools in the nlp community. @xcite show that model rankings on standard splits @xcite can often not be reproduced using randomly generated splits. in a follow-up study, @xcite find that evaluation results on random splits are often too optimistic, even for in-domain test samples. in order to make the data splits more challenging, they introduce heuristic splits based on sentence length and adversarial splits based on wasserstein distance. the clustering-based data split we propose in this paper follows the same idea of creating a challenging evaluation setup. in contrast to the adversarial splits proposed by @xcite , our splitting strategy controls for label distribution, allowing to abstract away from the effect of different label distributions on the evaluation score @xcite . ## clustering-based data splitting we here propose a novel algorithm dubbed size and distribution sensitive k-means (sds kmeans) which has two important properties relevant to generating challenging clustering-based cross-validation folds. the sds k-means algorithm produces clusters that (1) each have approximately the same size, i.e., a similar number of data points, and that (2) are controlled for label distribution. in the default case, all clusters have a similar label distribution. by clustering the data points, we ensure that training and development data are different (for now in a lexical sense), hence creating a challenging evaluation setup. the sds k-means algorithm thereby overcomes the following two difficulties: (1) varying cluster sizes: if clusters had different sizes, performance differences could simply be attributed to varying amounts of training data. (2) varying label distributions: if clusters had differing label distributions, performance differences could be primarily due to label distribution mismatches between training and test data. hence, when using sds k-means generated data folds with similar label distributions per cluster, differences in model performance can be attributed to qualitative rather than quantitative differences between the folds. in the experiments of this paper, we keep the label distribution fixed. if the user wants to deviate from the default case, s/he can also use the sds k-means algorithm to generate folds with varying label distributions, and thereby also investigate the effects of different label distributions on model performance. in the following, we describe technical details and the derivation of our algorithm. all algorithms described in this section produce k clusters that are intended to be used in k-fold cross-validation. ## clusterdatasplit tool the tool clusterdatasplit consists of three jupyter notebooks. hence, using the tool requires basic python skills. communication between the tool and user code for machine learning models is based on .tsv files containing the text data instances and labels. figure 2 illustrates the workflow and the separation of tasks between the tool and client code. currently, our tool and algorithm only support sequence classification tasks with a single label per dataset instance. we leave extensions for sequence tagging tasks to future work. ## worked examples in this section, we give worked examples of using our proposed data splitting method for two sequence classification tasks, the stanford sentiment treebank (sst) binary sentiment analysis task and a patent multi-class classification task. ## conclusion and outlook in this paper, we have introduced the concept of clustering-based data splits for model evaluation of sequence classification tasks. we have outlined the steps necessary to generate clustering-based data splits and described different k-means based algorithms for the creation of clustering-based data splits. our newly proposed sds k-means algorithm is able to generate clusters with equal (or controllable) size and label distribution. these properties make the algorithm perfectly suited for generating clustering-based data splits for challenging cross-validation experiments. our worked examples show that model evaluation on clusteringbased data splits generated by the sds k-means algorithm is more challenging than model evaluation on randomly selected data splits. the experiments conducted in this paper present a first step in exploring clustering-based data splits. directions for future research involve the steps necessary to generate the clustering-based data splits, further exploration of the data splits and additional use cases. with regard to the generation of clustering-based data splits, different text vector representations or clustering algorithms, like for example density-based approaches, could be explored. experiments with different text vector representations and clustering algorithms could shed some light on the impact of different cluster structures on the evaluation setup. moreover, it would be interesting to study why clustering-based data splits seem to have a stronger effect on some datasets than on others. @xcite introduce an experiment setting to compare the predictive performance of model evaluation on different data splits with regard to test sets coming from the same domain as the training data. applying our clustering-based data splits in this experiment setting thus could deliver important information about the predictive quality of model performance scores obtained on clustering-based data splits. currently, our method works only for classification tasks and clustering is performed mainly based on lexical information. hence, another interesting direction for future work is extending our ideas to data splitting for sequence tagging tasks, and to integrate other types of information such as syntactic features.