The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions (2020)

Abstract: We find that the performance of state-of-the-art models on Natural Language Inference (NLI) and Reading Comprehension (RC) analysis/stress sets can be highly unstable. This raises three questions: (1) How will the instability affect the reliability of the conclusions drawn based on these analysis sets? (2) Where does this instability come from? (3) How should we handle this instability and what are some potential solutions? For the first question, we conduct a thorough empirical study over analysis sets and find that in addition to the unstable final performance, the instability exists all along the training curve. We also observe lower-than-expected correlations between the analysis validation set and standard validation set, questioning the effectiveness of the current model-selection routine. Next, to answer the second question, we give both theoretical explanations and empirical evidence regarding the source of the instability, demonstrating that the instability mainly comes from high inter-example correlations within analysis sets. Finally, for the third question, we discuss an initial attempt to mitigate the instability and suggest guidelines for future work such as reporting the decomposed variance for more interpretable results and fair comparison across models.

## introduction neural network models have significantly pushed forward performances on natural language processing benchmarks with the development of largescale language model pre-training @xcite @xcite @xcite . for example, on two semantically challenging tasks, natural language inference (nli) and reading comprehension (rc), the state-of-the-art results have reached or even surpassed the estimated human performance on certain benchmark datasets @xcite @xcite . these astounding improvements, in turn, motivate a new trend of research to analyze what language understanding and reasoning skills are actually achieved, versus what is still missing within these current models. following this trend, numerous analysis approaches have been proposed to examine models' ability to capture different linguistic phenomena (e.g., named entities, syntax, lexical inference, etc.). those studies are often conducted in 3 steps: (1) proposing assumptions about a certain ability of the model; past analysis studies have led to many key dis-coveries in nlp models, such as over-stability @xcite , surface pattern overfitting @xcite , but recently @xcite found that the results of different runs of bert nli models have large non-negligible variances on the hans @xcite analysis datasets, contrasting sharply with their stable results on standard validation set across multiple seeds. this finding raises concerns regarding the reliability of individual results reported on those datasets, the conclusions made upon these results, and lack of reproducibility @xcite . thus, to help consolidate further developments, we conduct a deep investigation on model instability, showing how unstable the results are, and how such instability compromises the feedback loop between model analysis and model development. we start our investigation from a thorough empirical study of several representative models on both nli and rc. overall, we observe four worrisome observations in our experiments: (1) the final results of the same model with different random seeds on several analysis sets are of significantly high variance. the largest variance is more than 27 times of that for standard development set; next, in order to grasp a better understanding of this instability issue, we explore theoretical explanations behind this instability. through our theoretical analysis and empirical demonstration, we show that inter-examples correlation within the dataset is the dominating factor causing this performance instability. specifically, the variance of model accuracy on the entire analysis set can be decomposed into two terms: (1) the sum of single-data variance (the variance caused by individual prediction randomness on each example), and (2) the sum of inter-data covariance (caused by the correlation between different predictions). to understand the latter term better, consider the following case: if there are many examples correlated with each other in the evaluation set, then the change of prediction on one example will influence predictions on all the correlated examples, causing high variances in final accuracy. we estimate these two terms with multiple runs of experiments and show that inter-data covariance contributes significantly more than single-data variance to final accuracy variance, indicating its major role in the cause of instability. finally, in order for the continuous progress of the community to be built upon trustworthy and interpretable results, we provide initial suggestions on how to perceive the implication of this instability issue and how we should potentially handle it. for this, we encourage future research to: (1) when reporting means and variance over multiple runs, also report two decomposed variance terms (i.e., sum of single data variance and sum of inter-data covariance) for more interpretable results and fair comparison across models; (2) focus on designing models with better inductive and structural biases, and datasets with higher linguistic diversity. our contribution is 3-fold. first, we provide a thorough empirical study of the instability issue in models' performance on analysis datasets. second, we demonstrate theoretically and empirically that the performance variance is attributed mostly to inter-example correlations. finally, we provide suggestions on how to deal with instability, including reporting the decomposed variance for more interpretable evaluation and better comparison. ## tracking instability before answering the question how to handle these instabilities, we first seek the source of the instability to get a better understanding of the issue. we start with the intuition that high variance could be the result of high inter-example correlation within the dataset, and then provide hints from experimental observations. next, we show theoretical evidence to formalize our claim. finally, we conclude that the major source of variance is the interexample correlations based on empirical results. ## implications, suggestions, and discussion so far, we have demonstrated how severe this instability issue is and how the instability can be traced back to the high correlation between predictions of certain example clusters. now based on all the previous analysis results, we discuss potential ways of how to deal with this instability issue. we first want to point out that this instability issue is not a simple problem that can be solved by trivial modifications of the dataset, model, or training algorithm. here, below we first present one initial attempt at illustrating the difficulty of solving this issue via dataset resplitting. ## conclusions auxiliary analysis datasets are meant to be important resources for debugging and understanding models. however, large instability of current models on some of these analysis sets undermine such benefits and bring non-ignorable obstacles for future research. in this paper, we examine the issue of instability in detail, provide theoretical and empirical evidence discovering the high inter-example correlation that causes this issue. finally, we give suggestions on future research directions and on better analysis variance reporting. we hope this paper will guide researchers on how to handle instability and inspire future work in this direction. snli-hard @xcite , nli stress test @xcite and @xcite . we use advsquad @xcite as the rc analysis dataset. break nli. 7 the examples in break nli resemble the examples in snli. the hypothesis is generated by swapping words in the premise so that lexical or world knowledge is required to make the correct prediction. snli-hard. foot_7 snli hard dataset is a subset of the test set of snli. the examples that can be predicted correctly by only looking at the annotation artifacts in the premise sentence are removed. nli stress. foot_8 nli stress datasets is a collection of datasets modified from mnli. each dataset targets one specific linguistic phenomenon, including word overlap (str-o), negation (str-ne), antonyms (str-a), numerical reasoning (str-nu), length mismatch (str-l), and spelling errors (str-s). models with certain weaknesses will get low performance on the corresponding dataset. in our experiments, we use the mismatched set if there hans. 11 the examples in hans are created to reveal three heuristics used by models: the lexical overlap heuristic, the sub-sequence heuristic, and the constituent heuristic. for each heuristic, examples are generated using 5 different templates.