Showing Your Work Doesn’t Always Work (2020)

Abstract: In natural language processing, a recently popular line of work explores how to best report the experimental results of neural networks. One exemplar publication, titled “Show Your Work: Improved Reporting of Experimental Results” (Dodge et al., 2019), advocates for reporting the expected validation effectiveness of the best-tuned model, with respect to the computational budget. In the present work, we critically examine this paper. As far as statistical generalizability is concerned, we find unspoken pitfalls and caveats with this approach. We analytically show that their estimator is biased and uses error-prone assumptions. We find that the estimator favors negative errors and yields poor bootstrapped confidence intervals. We derive an unbiased alternative and bolster our claims with empirical evidence from statistical simulation. Our codebase is at https://github.com/castorini/meanmax .

## introduction questionable answers and irreproducible results represent a formidable beast in natural language processing research. worryingly, countless experimental papers lack empirical rigor, disregarding necessities such as the reporting of statistical significance tests @xcite and computational environments @xcite . as @xcite concisely lament, explorimentation, the act of tinkering with metaparameters and praying for success, while helpful in brainstorming, does not constitute a rigorous scientific effort. against the crashing wave of explorimentation, though, a few brave souls have resisted the urge to feed the beast. @xcite argue for the reporting of neural network score distributions. @xcite demonstrate that deterministic dataset splits yield less robust results than random ones for neural networks. @xcite advocate for reporting the expected validation quality as a function of the computation budget used for hyperparameter tuning, which is paramount to robust conclusions. but carefully tread we must. papers that advocate for scientific rigor must be held to the very same standards that they espouse, lest they birth a new beast altogether. in this work, we critically examine one such paper from @xcite . we acknowledge the validity of their technical contribution, but we find several notable caveats, as far as statistical generalizability is concerned. analytically, we show that their estimator is negatively biased and uses assumptions that are subject to large errors. based on our theoretical results, we hypothesize that this estimator strongly prefers underestimates to overestimates and yields poor confidence intervals with the common bootstrap method @xcite . our main contributions are as follows: first, we prove that their estimator is biased under weak conditions and provide an unbiased solution. second, we show that one of their core approximations often contains large errors, leading to poorly controlled bootstrapped confidence intervals. finally, we empirically confirm the practical hypothesis using the results of neural networks for document classification and sentiment analysis. ## analysis of showing your work in "show your work: improved reporting of experimental results," @xcite realize the ramifications of underreporting the hyperparameter tuning policy and its associated budget. one of their key findings is that, given different computation quotas for hyperparameter tuning, researchers may arrive at drastically different conclusions for the same model. given a small tuning budget, a researcher may conclude that a smaller model outperforms a bigger one, while they may reach the opposite conclusion for a larger budget. to ameliorate this issue, @xcite argue for fully reporting the expected maximum of the score as a function of the budget. concretely, the parameters of interest are θ 1 , . . . , θ b , where in other words, θ n is precisely the expected value of the n th order statistic for a sample of size n drawn i.i.d. at tuning time. for this quantity, they propose an estimator, derived as follows: first, observe that the cdf of for approximating the cdf, @xcite use the ecdf f n b (v), constructed from some sample s := (v 1 , . . . , v b ), i.e., to construct an estimator θn for θ n , @xcite then replace the cdf with the ecdf: ## experiments 4.1 experimental setup to support the validity of our conclusions, we opt for cleanroom monte carlo simulations, which enable us to determine the true parameter and draw millions of samples. to maintain the realism of our study, we apply kernel density estimation to actual results, using the resulting probability density (or discretized mass) function as the ground truth distribution. specifically, we examine the experimental results of the following neural networks: document classification. we first conduct hyperparameter search over neural networks for document classification, namely a multilayer perceptron (mlp) and a long short-term memory (lstm; hochreiter and schmidhuber, 1997) model representing state of the art (for lstms) from @xcite . for our dataset and evaluation metric, we choose reuters @xcite and the f 1 score, respectively. next, we fit discretized kernel density estimators to the results-see the appendix for experimental details. we name the distributions after their models, mlp and lstm. sentiment analysis. similar to @xcite , on the task of sentiment analysis, we tune the hyperparameters of two lstms-one ingesting embeddings from language models (elmo; @xcite , the other shallow word vectors (glove; @xcite . we choose the binary stanford sentiment treebank @xcite dataset and apply the same kernel density estimation method. we denote the distributions by their embedding types, glove and elmo. ## conclusions in this work, we provide a dual-pronged theoretical and empirical analysis of @xcite . we find unspoken caveats in their work-namely, that the estimator is statistically biased under weak conditions and uses an ecdf assumption that is subject to large errors. we empirically study its practical effects on tasks in document classification and sentiment analysis. we demonstrate that it prefers negative errors and that bootstrapping leads to poorly controlled confidence intervals.