We Need to Talk About train-dev-test Splits (2021)

Abstract: Standard train-dev-test splits used to benchmark multiple models against each other are ubiquitously used in Natural Language Processing (NLP). In this setup, the train data is used for training the model, the development set for evaluating different versions of the proposed model(s) during development, and the test set to confirm the answers to the main research question(s). However, the introduction of neural networks in NLP has led to a different use of these standard splits; the development set is now often used for model selection during the training procedure. Because of this, comparing multiple versions of the same model during development leads to overestimation on the development data. As an effect, people have started to compare an increasing amount of models on the test data, leading to faster overfitting and “expiration” of our test sets. We propose to use a tune-set when developing neural network methods, which can be used for model picking so that comparing the different versions of a new model can safely be done on the development data.

## the tune split the solution we propose to the problem introduced in section 1.2 follows logically from the observation that we do not have a data split left for comparing the models. we simply introduce an additional data split, which we call the tune split (figure 1(c) ). this tune data can be used to pick the best model, thereby leaving the development set out of the training procedure. then the best model b out of b 1...n to compare against model a can be picked based on the development data, and the superiority of model b i can be confirmed on the test data. this also makes a comparison to traditional machine learning models fairer, as they also do not make use of the dev data during training. one clear downside of this approach is that there is less data remaining for the other splits. to overcome this, one could also pick the best hyperparameters/settings for model b based on the dev split, while using the tune split for model picking, and then for the final comparison add the tune split to the train split and use the development data for picking the best model. this procedure is the same as it would be in a shared task setup, where the train+dev data can be used however the participants see fit, but the test data remains unseen until the final comparison. it should be noted that in cross-domain or crosslingual setups, similar solutions have recently been proposed. in these setups, people commonly use the source dataset dev split for model picking @xcite . to have a pure cross-domain or cross-lingual setup, it is important to not tune on all target domains/languages as you are likely to overestimate performance when no target data is available. @xcite therefore argue to only use the dev set of one target language and report test results on other languages. another case where a similar solution was sometimes used, is the devtest set in machine translation, which is used at least since the wmt 2006 shared task @xcite . this split is an effect of having many sequential shared task, where new test-data is added every year. in some work, the dev split is used for model-picking and the testdev split is used as development data. however, to the best of our knowledge, there is no offical use (nor guidelines) on the function of the devtest split. an alternative solution is introduced by @xcite , who propose methods for picking the best model that do not rely on any labeled data. ## case study to evaluate the effect of having a separate tune split, we perform a case study in which we fine-tune a transition-based @xcite bi-lstm @xcite parser and a transformer-based @xcite deep biaffine parser @xcite on the same datasets. we use the universal dependencies (ud) 2.8 data @xcite as benchmark, and use the uuparser @xcite and the machamp @xcite implementations of the corresponding parsers. ## conclusion we have reflected on the default dataset splits used in nlp (and actually also more widely in machine learning) to tune design decisions (architectures, hyperparameters, etc.) of neural network based models, which can easily lead to overfitting on the test data. this is an effect of the fact that in standard setups, neural networks use the development data during training, and it thus became more common to compare multiple versions of the same model on the test data. the solution to this problem is simple, we need another data split to do model picking, or avoid using the dev set in the training procedure, by learning which model to pick using other heuristics @xcite . we call this split the tune-split. the only downside of using a separate tune-split, is that there is less data available for the other splits. this can be circumvented by using train+tune for the final (test-)runs of the model. we evaluated the effect of the tune-split for two common nlp benchmarks by tuning two different types of models. one of them showed to be more robust against the evaluated hyperparameter ranges, whereas the other showed a clear performance improvement when using a tune-split. this proposed solution is orthogonal to other proposed practices for a hygienic experimental setup like significance testing, random splits, and evaluating specific abilities of our models.