Paper:
A Boo(n) for Evaluating Architecture Performance
We point out important problems with the common practice of using the best single model performance for comparing deep learning architectures, and we propose a method that corrects these flaws. Each time a model is trained, one gets a different result due to random factors in the training process, which include random parameter initialization and random data shuffling. Reporting the best single model performance does not appropriately address this stochasticity. We propose a normalized expected best-out-of-n performance (Boo n ) as a way to correct these problems.

Review:
This paper addresses multiple issues arising from the fact that commonly reported best model performance numbers are a single sample from a performance distribution. These problems are very real, and they deserve significant attention from the ML community.  However, I feel that the proposed solution may actually compound the issues highlighted.
Firstly, the proposed metric requires calculation of multiple test set experiments for every evaluation. In the paper up to 100 experiments were used. This may be reasonable in scenarios where the test set is hidden, and individual test numbers are never revealed. It also may be reasonable if we cynically assume that researchers are already running many test-set evaluations. But I am very opposed to any suggestion that we should relax the maxim that the test set should be used only once, or as close to once as is possible. Even the idea of researchers knowing their test set variance makes me very uneasy.
Secondly, this paper tries to account for variation in results due to different degrees of hyper-parameter tuning. This is certainly an admirable aim, since different research groups have access to very different types of resources. However, the suggested approach relies on randomly picking hyper-parameters from "a range that we previously found to work reasonably well". This randomization does not account for the many experiments that were required to find this range. And the randomization is also not extended to parameters controlling the model architecture (I suspect that a number of experiments went into picking the 32 layers in the ResNet used by this paper). Without a solid and consistent basis for these hyper-parameter perturbations, I worry that this approach will fail to normalize the effect of experiment numbers while also giving researchers an excuse to avoid reporting their experimental process.
I think this is a nice idea and the metric does merge the stability and low variance of mean score with the aspirations of best score. The metric may be very useful at development time in helping researchers build a reasonable expectation of test time performance in cases where the dev and test sets are strongly correlated. However, for the reasons outlined above, I don't think the proposed approach solves the problems that it addresses. Ultimately, the decision about this paper is a subjective one. Are we willing to increase the risk of inadvertent hyper-parameter tuning on the test set for the sake of a more stable metric?