We Need to Talk About Reproducibility in NLP Model Comparison (2023)

Abstract: NLPers frequently face reproducibility crisis in a comparison of various models of a real-world NLP task. Many studies have empirically showed that the standard splits tend to produce low reproducible and unreliable conclusions, and they attempted to improve the splits by using more random repetitions. However, the improvement on the reproducibility in a comparison of NLP models is limited attributed to a lack of investigation on the relationship between the reproducibility and the estimator induced by a splitting strategy. In this paper, we formulate the reproducibility in a model comparison into a probabilistic function with regard to a conclusion. Furthermore, we theoretically illustrate that the reproducibility is qualitatively dominated by the signal-to-noise ratio (SNR) of a model performance estimator obtained on a corpus splitting strategy. Specifically, a higher value of the SNR of an estimator probably indicates a better reproducibility. On the basis of the theoretical motivations, we develop a novel mixture estimator of the performance of an NLP model with a regularized corpus splitting strategy based on a blocked 3\times 2 cross-validation. We conduct numerical experiments on multiple NLP tasks to show that the proposed estimator achieves a high SNR, and it substantially increases the reproducibility. Therefore, we recommend the NLP practitioners to use the proposed method to compare NLP models instead of the methods based on the widely-used standard splits and the random splits with multiple repetitions.

## introduction in nlp domain, the reproducibility of empirical experimental conclusions has attracted most researchers' attention because the reproducibility crisis becomes more and more serious along with the increasing complexity of the proposed model architectures for a specific nlp task @xcite . in particular, in the past few decades, many researchers found that the majority of the conclusions in model comparisons can not be well reproduced in similar or slightly different experimental settings involving many real-world nlp tasks @xcite @xcite , including pos tagging, parsing, text summarization, machine translation and so on. although a checklist methodology has been recommended in many nlp conferences, the checklist could merely address the reproducibility problems in the levels of methods and results rather than in the level of inferential conclusions @xcite . considering that about 70% of papers had an unpromising low reproducibility @xcite , whether a conclusion in an nlp model comparison is reproducible if one draw it from a different experimental setup is still an important problem that is not sufficiently addressed. in actual, the reproducibility of a conclusion depends on multiple variates or factors in an nlp experiment, such as source code, text corpus and so on. an important factor that affects the reproducibility is the splitting strategy of a text corpus, which has been well recognized in recent years. as a thumb rule, standard splits on a text corpus are frequently used to ensure a fair comparison. however, @xcite found that the conclusion obtained on the standard splits is less reproducible and instable, and thereby they recommended random splits for nlp practitioners. different from their views, @xcite pointed out that the random splits lead to overly optimistic performance estimators and harm the nlp model comparison. thus, they recommended multiple biased splits for assessing nlp models. moreover, @xcite proposed a novel splitting strategy and introduced a tune set to select an optimal neural model in hundreds of epochs for fairness in subsequent comparisons. these studies provided sufficient empirical evidences to show the importance of a splitting strategy on the reproducibility in a comparison but lacking of a necessary theoretical investigation on the relationship between the reproducibility and the estimator induced from a splitting strategy. thus, the improvement on the standard splits is limited. essentially, various splitting strategies lead to different estimators of the performance of an nlp model. the variance and the expectation of a performance estimator are critical to the reproducibility of the comparison of two nlp models. therefore, in order to select an optimal splitting strategy and a reasonable estimation method, it is necessary to establish the theoretical relationship between the reproducibility and a performance estimator in a model comparison. in this paper, we formulate the reproducibility in an nlp model comparison from a probabilistic perspective and theoretically establish the relationship between the reproducibility and the signal-to-noise ratio (snr) of an estimator of model performance. we illustrate that an estimator with a higher snr probably leads to a better reproducibility in a comparison. therefore, we establish an optimization problem to maximize the snr of an estimator in a model comparison. we further summarize that the solution to the problem should satisfy two criteria with regard to the variance and the expectation of the estimator, respectively. motivated by the criteria, we develop a novel method that uses a blocked 3 × 2 cross-validation (bcv) coupled with a mixture estimator of nlp model performance and provide theoretical explanations of the method. we conduct numerical experiments on multiple nlp tasks to show that the proposed method produces an estimator with a high snr and substantially increases the reproducibility of a conclusion in an nlp model comparison. ## reproducibility in model comparison in this paper, we merely consider the reproducibility in a comparison of two nlp models from the perspectives of splitting strategy and estimation method. when the number of models is larger than two, the comparison can be decomposed into a collection of comparisons of any two nlp models among the candidates, and the comparison results on multiple pairs of models can be further reduced toward the final result with a proper correction, such as the bonferroni procedure @xcite . we denote a text corpus as d n where n is the corpus size, and all the samples in d n are iid-sampled from population d. let a and b be two nlp models built on the corpus d n , and the corresponding model instances are denoted as a(d n ) and b(d n ). furthermore, let ν be a commonly-used performance metric, such as accuracy, f 1 score and so on. without loss of generality, we assume that higher values of ν indicate better performance. furthermore, if a previous empirical study compared the performance of models a(d n ) and b(d n ) based on a particular splitting strategy s and published a conclusion of ν(a(d n )) > ν(b(d n )) in a paper, then we have to ask a question: how the splitting strategy s affects the reproducibility of the con- for example, assume the conclusion is obtained based on standard splits. if another research group uses slightly different splits to perform a similar comparison, to what extent the conclusion is reproduced? it is hard to solve the optimization problem in eq. ( 2 ) since the distribution of νa-b 1 ) and obtain the following lower bound. furthermore, according to the definition of snr in eq. ( 4 ), the solution to eq. ( 5 ) should satisfy the following two criteria. criterion i. the optimal splitting strategy and estimation method should reduce the variance v ar[ν a-b s ] as much as possible. criterion ii. the optimal splitting strategy and estimation method should maximize the expectation e[ν a-b s ] as much as possible. ## our approach although the optimization problem in eq. ( 2 ) is relaxed into a second best problem in eq. ( 5 ), the latter is still not solved through an analytic methodology. despite this, the two criteria obtained from eq. ( 5 ) may shed light on a heuristic design of a better splitting strategy and estimation method. ## experiments we compare our proposed method with the baselines in section 3.3 on the following nlp tasks. semantic role labeling task (srl): identify the boundaries of all semantic role chunks in a chinese sentence with gold word segmentation and a pre-defined target word based on the chinese framenet annotation convention. a chinese framenet training corpus is used as d n that contains 35,473 chinese sentences and 78,749 semantic chunks. a bi-lstm algorithm coupled with linguistic features of word, pos, target word, and frame name, is used to train and validate an srl model. considering the embedding size of a target word is an important hyper-parameter, we compare the two srl models obtained from the settings of the embedding size being 10 and 20, respectively. named entity recognition task (ner): identify the boundaries of all ner chunks without recognizing their type labels. we use conll 2003 english ner training set as d n that contains 14,987 sentences and 23,499 chunks. the corpus contains four types of named entities, including per, loc, org, and misc. we consider the ner task as a sequence labeling problem and use crfs to train and validate an ner model. two labeling sets of task sample means (%) standard errors (×10 -4 ) confidence intervals (%) iob2 and iobes are used to induce two different ner models that are compared in our experiments. chinese base-phrases chunking task (chunking): identify the boundaries of all base-phrases in a chinese sentence with gold word segmentation. the chinese base-phrases corpus is developed by @xcite . the corpus is used as d n that contains 14,248 sentences and 186,319 chunks. we use crfs based on different labeling sets of iob2 and iobes to obtain two different chunking models that are compared in our experiments. the above three tasks usually adopt precision, recall, and f 1 score as evaluation metrics. due to space limitation, we merely report the results about f 1 score. for the three nlp tasks, we consider the following empirical conclusions in model comparisons. to check the correctness of the above empirical conclusions, i.e., ν(a(d n )) > ν(b(d n )), we conduct numerical simulations to obtain the performance of models a and b. specifically, we combine the original training and test sets and randomly sample a training set and a test set in a without-replacement manner from the combined data set. the sampling ratio is the same as the original splitting ratio. then, we independently perform the process 1,000 times and use the sample mean of the 1,000 values of estimators ν(a(d n )) and ν(b(d n )) to approximate ν(a(d n )) and ν(b(d n )). considering that the 1,000 estimators are iid, we further compute the standard errors and the confidence intervals with α = 0.05. the results listed in on the basis of a text corpus and a splitting strategy, we randomly generate the corresponding splits and use the splits to train and validate the compared nlp models and obtain the corresponding estimators of the difference of the performance of the two compared models. this process is independently repeated in 1,000 times, and we obtain 1,000 values of an estimator that is used to compute the corresponding snr and reproducibility. for the computation of the snr, we approximate the expectation and the variance of an estimator with the sample mean and the sample variance of the 1,000 values of an estimator. then, we use eq. (4) to compute the numerical value of the snr. for the computation of the reproducibility, we compute the proportion of the 1,000 values greater than zero. then, we obtain the numerical value of the reproducibility by replacing the probability in eq. ( 1 ) with the empirical proportion. we organize experiments into three groups to illustrate (1) the relationship between the snr and the reproducibility; and (2) the distribution of an estimator in each of different methods; and (3) the reproducibility over different baselines and our method. ## related work in recent decades, a task of reproducing and/or replicating results and conclusions in published nlp papers has attracted many researchers' attention. many studies have illustrated that the reproducibility in nlp models has a close relationship with many factors, including availability of source code @xcite , randomization manner of instances in a text corpus @xcite , nature of an nlp task and under-specified or missing details in a published paper @xcite , readability of a published paper and number of hyperlinks @xcite , splitting strategy of a corpus @xcite , statistical test used in an nlp task @xcite , tradition of publishing reproduction attempts @xcite , and so on. therefore, several systematic reviews and meta analysis related to the reproducibility in nlp models are conducted in recent years, including but not limited to @xcite @xcite @xcite . furthermore, many effective methods are recommended to ensure a high reproducibility in an nlp study. for example, @xcite proposed a novel replicability analysis framework based on partial conjunction testing. @xcite developed a deep dominance method to properly compare neural nlp models. @xcite recommended a quantitative measure of reproducibility from a metrological perspective. @xcite investigated the commonly-used checklist method. besides the above-mentioned studies, an attractive direction is to investigate the relationship between a splitting strategy of a text corpus and the reproducibility in a comparison of two nlp models. @xcite showed that the optimized values of hyper-parameters in an nlp model are highly sensitive to how a text data set is partitioned. @xcite illustrated that the popular standard splits are insufficient to reproduce a conclusion in the task of nlp model comparison. they recommended multiple random splits of a text corpus. @xcite further made an improvement on the splitting strategy of a text corpus and proposed biased splits for nlp practitioners. rob (2021) introduced a tune set to make the optimization process of a neural nlp model more reliable. these studies shed lights on the direction and inspired us to further optimize the splitting strategy from a different perspective. ## conclusions in this paper, we theoretically analyze the relationship between the reproducibility in an nlp model comparison and the snr of an estimator of the difference of the performance of two nlp models. we further show that a higher value of the snr probably indicates a better reproducibility. to increase the reproducibility, we formulate an optimization problem to maximize the snr and further establish two criteria for figuring out an optimal splitting strategy and a better estimation method. we develop a novel method that uses a 3 × 2 bcv coupled with a mixture estimator and illustrate the superiority of the proposed method through multiple real-world nlp tasks. we recommend the nlp practitioners to use a 3 × 2 bcv and the mixture estimator to compare their nlp models. in future, we aim to investigate a comparison of multiple nlp models and try our best to figure out a better method to improve the reproducibility of the comparison based on the splitting strategy of an m × 2 bcv.