A B OON FOR E VALUATING A RCHITECTURE P ERFORMANCE
Anonymous authors
Paper under double-blind review
A BSTRACT
We point out several important problems with the common practice of using the best single model performance for comparing Deep Learning architectures, and we propose a method that corrects these ﬂaws.
Each time a model is trained, one gets a different result due to random factors in the training process, which include random parameter initialization and random data shufﬂing.
Reporting the best single model performance does not appropriately deal with this stochasticity.
Furthermore, the expected best result increases with the number of experiments run, among other problems.
We propose a normalized expected best-out-of- n performance (Boo n ) as a way to correct these problems.
I NTRODUCTION
Replicating results in Deep Learning (DL) research is often hard.
This harms the usefulness of the research to industry and can lead to signiﬁcant waste of effort by other researchers, not to mention very limited scientiﬁc value of such results.
One reason for these problems may be that insufﬁcient information for replication is provided in many papers.
Many details of the experimental setup can have a signiﬁcant impact on the results (Henderson et al., 2017;
Fokkens et al., 2013; Raeder et al., 2010), so these should be provided at least in appendices, ideally alongside the source code, as was strongly emphasized e.g. by Ince et al.
However, there is a second factor that can hinder replicability: the stochasticity of experiment results, which is inherent to practically all DL training methods, and which usually comes from random data ordering in stochastic gradient descent and from random parameter initialization, though there can be additional sources of randomness such as Dropout or gradient noise.
Consequently, even if we ﬁx the model architecture and the experimental setup (including the hyper-parameters), we obtain a different result each time we run the experiment.
Hence, statistical techniques are needed to handle this variability when reporting results of experiments.
However, in DL research such techniques are heavily underused.
What is usually done instead?
The most common way of reporting quantitative results in DL research is currently the performance of the best single model – as we will show for some sub-domains.
Given the result stochasticity, this method is statistically ﬂawed.
The best model performance is not robust under experiment replication, and its expected value improves with an increasing number of experiments performed, among other problems.
Since these problems are largely ignored in current DL research publications, we dedicate the ﬁrst part of this article to explaining and illustrating them in some detail.
These problems point to the necessity of statistical techniques for evaluating (and comparing) the performance of Machine Learning (ML) architectures.
Methodology exists for such comparisons (a great introduction to such methods is given for instance by Cohen (1995)).
However, most existing methodology focuses on comparing the mean performance.
This may be one of the reasons why statistical methods are being underused, since mean may be unattractive to researchers in certain situations.
There may be multiple reasons for this.
The one that we do consider sound 1 is that when deploying models in practice, it is often natural to train multiple instances of a model and then deploy the best one to production based on a validation set evaluation.
2 Underperforming models can be discarded, so the ﬁnal deployed model does come from the higher tier of the model performance population, and the use of mean may be inappropriate.
Hence, in this paper, rather than completely abandon the method, we propose a way to ﬁx the ﬂaws of reporting the performance of the best model.
We do this by estimating the expected Best out of n (Boo n ) performance by running more than n experiments, which gives the estimate statistical validity if a sufﬁcient number of experiments are run.
We discuss how this measure relates to the performance distribution of the model and also give a method to empirically estimate Boo n .
B EST SINGLE MODEL PERFORMANCE
In most articles proposing new DL architectures, the performance is often reported as the score of the “best single model” or simply “single model”.
In practice, this usually means that the researchers train multiple instances of the proposed architecture, often with different sets of hyper-parameters, evaluate these instances on some validation set, select the best-performing model, which is then evaluated on a test set – this test score is then reported as the score characterizing the architecture and used for comparing it to previous models.
If the score is better than those reported in previous work, the architecture is presented as superior.
This current practice results in several issues:
Population variance
The ﬁrst problem is that since results of experiments are stochastic, the performance of the single model is just a single instance drawn from a possibly disparate population.
If others train the model on their own, they get another sample from the architecture’s performance distribution, which may substantially differ from the one listed in the original paper.
The paper thus gives insufﬁcient information about what to expect from the new architecture, which should be one of the article’s main points.
One may object that the result published in the paper is not chosen from the population at random – it is selected using a validation result.
However, the correlation between the validation and test results is generally imperfect; in fact, in some of our experiments, it is almost zero, as we show in Section 4.
Furthermore, if we indeed do have a strong correlation, we get another problem:
Increasing expected result with increasing number of experiments Simply put, the more samples from a population we get, the more extreme the best among them is likely to be.
In other words, the expectation of the best result is dependent on the number of experiments that the researchers run.
There are three closely related problems with this: Firstly, this makes the number of experiments run an important explanatory variable; however, this variable is usually unreported, which is a severe methodological ﬂaw in itself.
However, it also leads to the second problem: since each research team runs a different number of experiments, the results are not directly comparable.
Thirdly, this motivates researchers to run more experiments and gives an advantage to those that are able to run more experiments.
This pushes publishing quantitative results towards a race in computational power rather than a fair comparison of architectures themselves.
Best model performance is not a meaningful characteristic of the performance distribution
Even if we knew the underlying theoretical performance distribution – that is, if we had perfect information about the architecture’s performance – it would not be clear what we would mean by
"best model performance" without specifying the size of the pool from which we are choosing the best model.
Imagine some architecture having a Gaussian performance distribution.
Asking what is the best possible performance does not make sense in such a case, so using the best-model performance to characterize the performance distribution is not meaningful.
Even for capped metrics such as accuracy, where the performance distribution necessarily has bounded support, the best (possible) model 3 may be so unlikely, that it would be of no practical importance.
Hence, best model performance is not a meaningful characteristic of the performance distribution.
Generality / Falsiﬁability
Finally, there is the question of what the authors are trying to express.
Using “best single model performance”, the authors are essentially claiming: “There once existed an instance of our model that had accuracy X % on a dataset Y”.
Such fact is not of that much interest to the scientiﬁc community, which would rather need to know how the architecture behaves generally .
Relatedly, a frequently given characteristic of science is falsiﬁability of theories (Popper, 1959).
A theory claiming that there are invisible unicorns running among us is not science, since we cannot think of any potential empirical evidence that could prove the theory false.
Similarly, any number of replication experiments that produce substantially worse results cannot prove the above claim wrong.
If, for instance, a conﬁdence interval were given, replications could very quickly show a published result at least extremely implausible, if not false.
We will quantify the former two problems for two concrete architectures in Section 4.
2. P REVALENCE
Despite all these problems, reporting the performance of the best model is still the main way of reporting results in many areas of ML, and such practice seems to be tolerated by reviewers even at prime conferences.
For instance, what concerns models published on the popular Children’s Book
Test dataset for Reading Comprehension (on which we run experiments later), none of the (more than ten) papers used any form of statistical testing or conﬁdence intervals, and most reported only performance of the best single model without even mentioning the total number of experiments run.
These include papers published at NIPS (Hermann et al., 2015), ICLR (Hill et al., 2016; Munkhdalai & Yu, 2017), ACL (Chen et al., 2016; Dhingra et al., 2017;
Cui et al., 2016), or EMNLP (Trischler et al., 2016).
The same is true for the recently popular SQuAD dataset: for instance, none of the four papers (Yang et al., 2017; Wang & Jiang, 2017; Seo et al., 2017; Xiong et al., 2017) that published results on this dataset at ICLR 2017 have used any statistical testing or conﬁdence intervals nor published mean (or otherwise aggregated) results across multiple runs.
Overall, while a signiﬁcant proportion of papers at ICLR do contain an empirical evaluation of models or methods (as a rough guide, 174 out of 194 ICLR papers have "experiment" in a (sub-)section heading), only 11 mention terms related to hypothesis testing 4 and 11 contain the string "conﬁdence interval".
Further details can be found in Appendix A.
While this is certainly not meant as a comprehensive survey, it does suggest that while DL research is to a large extent an empirical science, the usage of statistical methods may be lagging behind and need improving.
E XPECTED BEST - OUT - OF - n (B OO n ) PERFORMANCE
The issues outlined above point to desiderata for a more suitable way of reporting an architecture’s performance.
Firstly, it should provide information about general behaviour of the architecture under speciﬁed conditions.
Secondly, it should be invariant under the number of experiments run (and other similar variables).
Finally, the resulting conclusion should be falsiﬁable and provide information relevant to the rest of the scientiﬁc community.
[Figure] [Figure]
CBT
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
T e s t a cc u r a c y
[Figure]
[Figure]
Given these requirements, traditional statistical measures, such as mean or median, probably come to mind of many readers.
They do indeed ﬁx the above issues; still, they express only the performance of a typical member of the population.
However, in many ML applications, it may be the best model from a pool that is of interest.
When practitioners are choosing a model for deployment, they may indeed train multiple models and then deploy the best-performing one 5 .
This gives some justiﬁcation to the practice of reporting the performance of the best model and gives us a reason to attempt to ﬁx its problems rather than completely dismiss it.
Such corrected best-model measure would be more useful than mean or median in situations outlined above.
A natural way to remove the lack of comparability between models, each evaluated in a different number of experiments, is to normalize the results to the expected result if the number of experiments were the same, say n .
This can be quite easily estimated, if we run m experiments, m ≥ n .
The greater the number of experiments m , the more robust the estimate of the expected best, which also helps us eliminate the problem of statistical robustness.
We are proposing the expected best-out-of- n performance Boo n to be used where the performance of the best model from a pool seems as an appropriate measure.
Let us ﬁrst examine how the expected best-out-of- n (Boo n ) performance relates to a known theoretical performance distribution; we will then proceed with empirical estimation, which is of value in practice.
3. B OO n OF A PROBABILITY DISTRIBUTION
We will now examine how to calculate Boo n from a known theoretical probability distribution.
This will serve two purposes: Firstly, since we are proposing Boo n as a way to characterize the performance distribution, this will make it explicit how Boo n relates to the underlying theoretical distribution.
Secondly, in some cases we may be able to make an assumption about the family to which the theoretical distribution belongs.
The analytic calculation below will allow us to leverage this information when empirically estimating Boo n , which may be especially useful when our sample size m is small.
3.1. S INGLE EVALUATION
Let us ﬁrst look on the case of validation performance as it is easier to grasp: How do we calculate an expected best Boo n ( P ) of independent identically distributed (i.i.d.)
random variables X 1 , ..., X n with probability distribution P (the performance distribution of an architecture) with a probability density function (p.d.f.)
f and a cumulative distribution function (c.d.f.)
F ?
In the case where best means maximal (minimum can be calculated similarly), the maximum max { X 1 , ..., X n } has a c.d.f.
equal to
[Equation]
using the independence of the X i s in the last step.
To obtain the p.d.f.
of the maximum (in case of a continuous distribution), we just need to differentiate with respect to x:
[Equation]
Using the p.d.f.
, we can now calculate the expected value of the maximum as
[Equation]
We can get a precise numerical estimate of the above integrals in any major numerical computation package such as numpy .
For illustration, for the standard normal distribution we have
Boo 5 ( N (0 , 1))
≈ 1 . 163 , Boo 10 ( N (0 , 1))
≈ 1 .
More generally, Boo n (cid:0) N ( µ, σ 2 ) (cid:1) can then be expressed as µ + σ Boo n ( N (0 , 1)) .
Thanks to this form we can get numerical estimates of
Boo n (cid:0) N ( µ, σ 2 ) (cid:1) just by estimating the two usual parameters of the Gaussian, Boo n ( N (0 , 1)) becoming just a constant coefﬁcient if we ﬁx n .
The full details of calculation for the Gaussian distribution can be found in Appendix B.
In the case of a discrete performance distribution, which will be useful for empirical estimation below, we get a probability mass function
[Equation]
so if p i is the probability weight associated with value x
i , i.e. P [ X i = x j ] = p j for all i , this gives us
[Equation]
3. V ALIDATION - TEST EVALUATION
In the previous part, we were choosing the best model with respect to the metric whose expectation we were calculating.
Hence, that method can be used to calculate the expected best validation performance of n models.
In practice, the best model is usually chosen with respect to the validation performance, while test performance is then of interest.
To calculate the expectation of the test performance of the best-validation model, we need to substitute the direct value of x in Equation 1, with the expectation of the test performance X test conditional on the validation performance x val ,
[Equation]
yielding an expression for the expected test performance of the best-validation model chosen from a pool of size n
[Equation]
where P val is the marginal probability distribution of the validation performance.
Similar simple substitution can be done in the discrete case.
Expanding the expression for the bivariate Gaussian distribution with mean test performance µ test and test-validation covariance cov ( X valid , X test ) as in Appendix B.2 again yields a convenient expression
[Equation]
which can again be used for parametric estimation.
3. E MPIRICAL ESTIMATION
We usually do not know the exact performance distribution of the model; we only have samples from this distribution – the results of our experiments.
In such case, we can estimate the expected maximum empirically, and in fact it is the empirical estimates that are likely to be used in practice to compare models.
To get a non-parametric estimator, for which we do not make any assumption about the family of the performance distribution, we take the empirical distribution arising from our sample as an approximation of the architecture’s true performance distribution, similarly to Bootstrap methods.
The empirical performance distribution (cid:98) P assigns a probability weight of 1 m to each of our m samples.
We approximate Boo n of the true performance distribution by Boo n of this empirical distribution.
For the uniform empirical distribution, all the p i in Equation 2 are equal to 1 m .
Hence if we rank our samples from worst-validation to best-validation as ( x valid i 1 , x test i 1 ) , . .
, ( x valid i m , x test i m ) we get
[Equation]
This estimator does not make any assumption about the performance distribution from which our observations are drawn.
If we do use such an assumption (e.g. we know that the performance distribution of our architecture usually belongs to a certain family, e.g. Gaussian), we can add information to our estimator and possibly get an even better estimate (i.e. one with lower sampling error).
For the Gaussian distribution, we can use the standard estimators of the parameters in
Equation 3 to get a parametric estimator
[Equation]
where (cid:100) µ test and (cid:99) cov are standard estimators of mean and covariance respectively.
A similar parametric estimator could be calculated for other distributions.
3. C HOICE OF n
Our chief motivation for the above normalization was the elimination of the dependence of the performance score on the number of experiments, m .
However, we still need to choose n , the number of experiments to which we normalize.
This is similar to the choice one is facing when using a quantile – should one use the 75% one, the 95% one, or some other?
The choice of n most useful to a reader is when n is the number of candidate models that a practitioner would train before choosing the best one for some target application.
Such number will differ from domain to domain and will heavily depend on the computational cost of training models in such domain.
Even though the optimal choice is a matter of opinion, what is important is to create some convention to ensure comparability of models within a single domain.
For end-users, providing results for multiple values of n would be most useful, however such practice may bring more confusion into the comparison of models in articles, since it allows for cherry-picking for the value of n , and consequently, performing statistical tests would require corrections for multiple testing.
In our experiments we decided to use n = 5 .
3. A CCOUNTING FOR ESTIMATOR UNCERTAINTY
However, even Boo n is still just a single number whose estimate can be noisy.
Hence, with Boo n , as well as with mean and other ways of aggregating results of a wider population, we should always use appropriate statistical methods when trying to compare the quantitative performance of a new model against a baseline.
This can be done using signiﬁcance testing (such as the t -test), or the uncertainty in performance estimation can be expressed using a conﬁdence interval, which seems to be the method currently preferred by a signiﬁcant part of the scientiﬁc community, since it allows us to disentangle the effect size from the uncertainty associated with noise and sample size (Berrar & Lozano, 2013;
Gardner & Altman, 1986).
For some theoretical distributions, there exist ways to calculate the hypothesis test or conﬁdence interval analytically (e.g. using the t-test or standard normal quantiles for the Gaussian).
However in cases where the family of the performance distribution or of the estimator is not known, we need to resort to computational methods - usually Monte Carlo (if we do know at least the family of the performance distribution) or Bootstrap (Efron, 1979)
(if we do not).
[Figure]
E XPERIMENTAL RESULTS
We have run several experiments to quantify the scope of the problems outlined in Section 2.
We just brieﬂy summarize the main results here for illustration; a more detailed description of the experiments can be found in Appendix C.
Performance variation To estimate the random variation of the results, we repeatedly 7 trained models from two domains of DL: the ResNet (Huang et al., 2017) on the CIFAR-100 dataset (Krizhevsky & Hinton, 2009) in image recognition and the Attention Sum Reader (AS Reader) (Kadlec et al., 2016) on
the Children’s Book Test Common Nouns (CBT CN)
(Hill et al., 2016) to represent Reading Comprehension.
The resulting performance distributions are illustrated in Figure 1.
If we ﬁx all hyper-parameters, the interquartile ranges of the models’ error rates are 0 .
98% and 1 .
20% (absolute).
Compare this to the median improvements of subsequent published results on these datasets: 0 .
86% and 1 . 15% respectively.
Hence, random variation in performance cannot be considered negligible as is currently often done.
Furthermore, if we allow the hyper-parameters to vary (in our case by random search), the result variance further increases, which further ampliﬁes the outlined effects.
In the case of the AS Reader the interquartile range increased to 2 .
9% when we randomly picked hyper-parameters from a range that we previously found to work reasonably well.
Several other articles conﬁrm signiﬁcant variation in model performance due to different random seed: e.g van den Berg et al.
(2016) in Speech Recognition, Henderson et al.
(2017) in Deep Reinforcement
Learning, or Reimers & Gurevych (2017) in Named Entity Recognition.
They all conclude that reporting performance scores of single models is insufﬁcient to characterize model performance.
Best-model performance improves with the number of experiments We also mentioned that if only the performance of the best model is reported, the more experiments are run, the better the expected result.
Figure 2b illustrates that this effect can indeed be fairly strong, if the validation performance is a good predictor of the test performance, as is the case of the AS Reader with random hyper-parameter search, where the expectation of the best single model performance increases from 61 .
3% if we train it once, to 63 .
3% if we train it 5 times, to 63 .
5% for 20 times.
This effect is further explained e.g. by Jensen & Cohen (2000).
It gives a further argument for refraining from using this method and certainly also for publishing the number of experiments run, which is often not done.
Boo n is not subject to this effect.
Validation-test correlation However, note that the assumption that validation performance is a good predictor of the test performance is sometimes not true.
In the two cases with ﬁxed hyperparameters that we looked at, Spearman correlation between validation and test results was only 0 .
10 and 0 .
18 respectively for the two models.
The correlation signiﬁcantly increases if we allow the hyper-parameters to vary – to 0 .
83 for the AS Reader.
These results are also illustrated in
Figure 1.
Also larger validation sets are likely to improve this correlation, which can be understood as the degree of generalization from validation to test.
Note that the problem of increasing expected performance mentioned above is relevant only in the case of correlation between validation and test results.
However, the effect becomes very strong in the case where the performance we are reporting is also used for choosing the best model, which emphasizes the need for honest separation of validation and test data.
Estimator variance Figure 2 shows the 95% conﬁdence intervals of best single model results compared to the Boo 5 performance for a range of result-pool sizes m .
This is shown for the cases of both strong and weak test-validation correlation.
In both cases Boo 5 is signiﬁcantly less noisy than the best-single-model result.
In fact in the case of random hyper-parameter search, Boo n shows even smaller variation than the mean (due to the negative skew of the performance distribution).
C ONCLUSION
In the ﬁrst part of this article, we have outlined several major problems with the common practice of using the “best single model” performance as a common way of reporting quantitative performance of Deep Learning architectures.
We do not think these ﬂaws will come as a surprise to anyone active in this area; still, they seem being ignored, which makes them worth pointing out.
We believe each of the problems would be sufﬁcient to make the practice inappropriate.
Hence, we need a better way of characterizing the performance distributions of proposed architectures.
Before returning to our case of single-number characterization, we should emphasize that one ought to attempt to characterize the distribution as fully as possible to give the reader reasonable expectations about how the model behaves – for instance, if the performance distribution seems to be (approximately) Gaussian, authors can provide the distribution’s parameters (the mean and the variance in this case), ideally with appropriate conﬁdence intervals.
This allows readers to subsequently choose the aspect of the distribution that they ﬁnd the most interesting, whether it be median or the expected best-out-of- n performance.
Of course, to complete the picture, such distribution should ideally be provided for multiple test datasets and associated metrics, since each brings to light a different aspect of the model performance, and such practice helps avoid ﬁne-adjusting models to particular datasets, which harms the generalizability of the results.
However, alongside this detailed characterization, describing an architecture’s performance by a single number may still have its appeal, especially for the purpose of comparison among architectures and choosing the best one according to some criterion 8 .
We have presented elements of one such possible method, which could be summarized as follows.
After choosing the most appropriate metric, such as accuracy, together with an appropriate test task – or possibly multiple of these – one should choose a suitable way of aggregating the results of models, to get some sensible characteristic of the architecture’s performance distribution – this can be the traditional mean or median, it can be the expected best model performance from a pool of n , which we have proposed as a substitute for the currently used “best single model” performance, or it can be some other characteristic.
This choice can differ domain from domain.
However, such method should be reasonably robust with respect to observation sampling, and it should be invariant under choices of experiment design that often differ between research teams, such as the number of experiments run, which was discussed in this article.
Once one chooses the appropriate measure of converting the pool of observations into a score characterizing the architecture’s performance distribution, one also needs to account for the uncertainty in the estimation of such score.
We described how to construct conﬁdence intervals using the Bootstrap as a possible way of accounting for this uncertainty.
This may allow researchers to assess the significance of the quantitative improvement that their architecture brings, which is unfortunately done rarely in recent research.
We believe that methodology currently common in DL research is lagging behind the standards usual in most other sciences.
Improving it, which can be worked on by both authors and reviewers, would bring us better reproducibility of results, as well as make the results more meaningful to both other researchers and to industry practitioners.
The main new theoretical contribution of the paper was in proposing a method that ﬁxes the main problems of the popular practice of reporting best single model performance.
The article ﬁrstly described both how it relates to the underlying (theoretical) performance distribution.
Secondly, it looked on how to estimate the the expected best-out-of- n performance from empirical observations.
We do not claim that the proposed method is appropriate for all situations – for instance, when we are interested chieﬂy in the performance of a typical model from a population, mean or median may be a better choice.
However, the target application often involves deploying the best model from a pool, and in that case, the performance of the expected best model out of n may be more informative.
Hence we believe it will be a useful contribution to the toolbox for describing and comparing the performance of ML models.
A PPENDIX A S URVEY OF ICLR 2017 PAPERS : METHOD
To roughly estimate the usage of statistical methods in ICLR 2017 papers, we downloaded the pdfs of all accepted papers 9 , extracted text from them using the OpenSource Xpdf package 10 and then searched the resulting text documents using the grep command as follows.
Firstly, to roughly estimate the usage of experiments in the papers, we searched for the capitalized string "EXPERIMENT" in the documents, since all (sub-)section headings are capitalized in the
ICLR format.
This was matched in 174 documents.
Further 6 contained the string "EVALUATION" yielding a total of 180 out of 194 papers containing one of the two strings, which suggests that many ICLR papers indeed have an empirical component, though our rough method is only very approximate.
We then searched for the string "conﬁdence interval", which was matched in only 11 papers, and further 11 documents matched one of expressions related to hypothesis testing (curiously, a set completely disjoint from the "conﬁdence interval" set).
These terms were: "hypothesis test", "pvalue", "t-test" "conﬁdence level", "signiﬁcance level", "ANOVA", "analysis of variance", "Wilcoxon", and "sign test".
This may be only a lower bound since mentioning the term does not necessarily mean that the method was employed in the experimental setup.
A PPENDIX B B OO n OF THE G AUSSIAN DISTRIBUTION
In this section we will calculate Boo n of the Gaussian distribution.
This can serve as a basis for a parametric estimator of Boo n , when we assume a performance distribution to be (approximately)
Gaussian, which was the case of some of the performance distributions we have examined, for instance the AS Reader with ﬁxed hyper-parameters.
B. S INGLE EVALUATION DATASET
In the simpler case in which the best model is chosen with respect to the same dataset on which the performance is then reported, we can substitute the p.d.f. and c.d.f.
of the Gaussian distribution into
Equation 1 to get
[Equation]
where Φ is the c.d.f.
of a standard normal random variable.
Substituting z
= x − µσ , dx = σdz , yields
[Equation]
(the ﬁrst integrand has the form of the p.d.f.
found above and hence integrates to one) so the expected maximum is neatly expressed in terms of a maximum of a standard normal and is linearly proportional to both the mean and the standard deviation.
Once n is ﬁxed for comparison purposes, Boo n ( N (0 , 1)) is just a constant, e.g. Boo 5 ( N (0 , 1))
≈ 1 . 163 , Boo 10 ( N (0 , 1))
≈ 1 .
B. T EST - VALIDATION EVALUATION
Let us turn to the case of reporting a test set performance of a best-validation model.
If we model the validation and test performances by a Bivariate Normal Distribution with valid-test correlation ρ , means µ val , µ test , and variances σ val , σ test , then given a validation performance x val , the test performance is distributed normally with conditional expectation
[Equation]
which gives
[Equation]
Using the same two tricks as above, this can be simpliﬁed to
[Equation]
where cov ( X valid , X test ) is the covariance between the validation and test performances, and
Boo n ( N (0 , 1)) is the single-evaluation expected maximum of the standard normal distribution as deﬁned above.
A PPENDIX C E XPERIMENTS : D ETAILS
Note: We will provide data and code for our experimental analysis when the paper is no longer subject to the anonymity requirement.
Here we provide further details of our experiments quantifying the extent of result stochasticity and the resulting effects.
C. M ODELS
To run our experiments we have chosen Open Source implementations 11 of models from two popular domains of Deep Learning, namely ResNet (Huang et al., 2017) on the CIFAR-100 dataset (Krizhevsky & Hinton, 2009) for Image Classiﬁcation and the AS Reader (Kadlec et al., 2016) on the CBT CN dataset (Hill et al., 2016) for Reading Comprehension.
We believe these two models representative of models in their respective areas – Resnet is based on a deep convolutional network architecture as most models in Machine vision, while the AS Reader is based on a bidirectional GRU network with attention, as is the case for many models in Natural Language Processing.
C. D ATA COLLECTION
To collect the data for our experiments, we repeatedly trained the two models.
Each training instance had a different random parameter initialization and random data shufﬂing.
We evaluated the model on the validation and test datasets at least once per epoch.
We then took the validation and test performance at the best-validation epoch as a data point for our further analyses.
All training was done on Ubuntu 14.04 on a single GPU per training, either Nvidia Tesla K80 or GTX 1080.
C.2. R ESNET
Resnet was trained with a single set of hyperparameters, the default ones for the above Open Source implementation.
That means 5 residual units resulting in a 32-layer Resnet.
The model was trained using the 0.9 momentum optimizer, with batch size 128, initial learning rate of 0.1 lowered to 0.
after 40,000 steps and to 0.001 after 60,000 steps.
Data augmentation included padding to 36x36 and then random cropping, horizontal ﬂipping and per-image whitening.
L2 regularization weight was set 0.002.
Training was done using Tensorﬂow 1.3.
C. AS R EADER
The AS Reader was trained in two different settings.
Firstly 370 times with hyper-parameters ﬁxed to embedding dimension of 128 and 384 hidden dimensions in the GRU units, with all other hyper-parameters as used in the original AS Reader paper (Kadlec et al., 2016).
In the second setting, the hyper-parameters for each training instance were chosen randomly from the following ranges: The batch size was chosen from the range [16 , 128] , and the embedding size and hidden state size were each chosen from the range [16 , 512] with the log 2 value of the parameter being distributed uniformly in the interval.
Parameters from these ranges worked reasonably well in our preliminary experiments.
Training was done using Theano 0.9.0 and Blocks 0.2.0.
C. P ERFORMANCE DISTRIBUTION RESULTS
Figure 1 plots the histograms of test performances of the evaluated models.
The mean test accuracy for Resnet was 68 .
41% with standard deviation of 0 .
67% (absolute), the range was 67 .
31% − 69 .
For AS reader with ﬁxed hyperparameters the mean was 63 .
16% with standard deviation 0 .
94% and range of 61 . 52% − 64 .
In the case of random hyper-parameter search the mean was 61 .
26% , standard deviation 2 .
48% , and values ranged from 56 .
61% to 64 .
In both cases with ﬁxed hyper-parameters the collected results are consistent with coming from a
Gaussian distribution according to the Anderson-Darling test 12 Anderson & Darling (1954); the histograms also make it appear plausible that the performance distribution is approximately Gaussian.
This is not the case for the random hyper-parameter search where the distribution has a clear negative skew.
To put the above numbers into context, we also examined the margin of improvement of successive architectures published on the corresponding datasets, as listed in Munkhdalai & Yu (2017); Huang et al.
We sorted the results with respect to the test performance and then calculated the differences between successive models.
The median difference was for 0 .
86% for Resnet and 1 .
15% for the AS Reader.
Note that the median differences are smaller than two standard deviations for each model.
Two standard deviations from the mean approximately give the 95% conﬁdence interval for a Gaussian distribution – hence we could typically ﬁt three successive published results within the width of one such conﬁdence interval.
The magnitude of the performance variation due to random initialization and data shufﬂing is therefore not negligible compared to the improvements in performance, which often hold an important place within articles in which they are presented.
We hence think it is inappropriate to completely ignore this random variation in evaluation protocols, which is currently the usual practice.
C. T EST - VALIDATION CORRELATION
The best model is usually selected using validation performance 13 .
This practice is based on the assumption that the validation accuracy is a reasonably good predictor of test accuracy.
The results of our experiments, illustrated also in Figure 1, suggest that this assumption holds for performance variation due to hyper-parameter choice.
However, if we ﬁx the hyper-parameters, the correlation almost disappears.
To some extent, this implies that selecting the best validation model means we are picking randomly with respect to the test performance.
Since we are picking from a random test performance distribution, this further calls for better characterization of the distribution than a single instance drawn from it.
On the other hand if the correlation is strong, as seems to be the case if we do perform hyper-parameter search, we face the second problem with reporting the best-validation performance:
C. E FFECT OF THE NUMBER OF EXPERIMENTS
If the validation performance is a good predictor of the test performance, then the more models we train the better the best-validation model is likely to be even on the test set since we are able to select models high up the right tail of the performance distribution.
This effect has been described in more detail in Jensen & Cohen (2000), though with focus on induction algorithms; here we present an estimate of its effect in the case of Resnet and AS Reader.
To test this effect we took the pool of trained modelsn.
For each m in the range from 1 to 50 (orfor the AS Reader), we randomly sampled 100 , 000 samples of size m from the pool, and selected the best-validation model from each sample.
The mean test performance across the 100 , 000 samples for each m is plotted in Figure 2.
The results show that when there is suitable correlation between validation and test performances, increasing the number of experiments does increase the expected performance of the best-validation model.
This makes the number of experiments an important explanatory variable, which however usually goes unreported.
Furthermore, it makes results reported by different research teams not directly comparable.
Finally, it gives an advantage to those that can run more experiments.
We believe that this again makes the practice of reporting the performance of the best single model unsuitable.
A PPENDIX D D EALING WITH ESTIMATOR UNCERTAINTY D. C ONFIDENCE INTERVALS
If an estimator characterizing a performance distribution, say (cid:91) Boo n or average, is calculated from experimental observations, it is subject to random variation, so if another research team tries to reproduce the experiments, they generally get a different estimate.
The more observations are collected, the more precise the estimate generally is.
Conﬁdence intervals provide a natural way to express this uncertainty.
Their usage also gives a sense whether the number of performed experiments was sufﬁcient to reduce the uncertainly to a reasonable level, which is again not frequently addressed in ML papers.
The construction of the conﬁdence interval would be trivial if we knew the distribution from which our estimate was drawn – it is simply the interval between the appropriate quantiles, e.g. the 2.5th and 97.5th quantiles in the case of the 95% conﬁdence interval.
Such distribution has been studied extensively for instance in the case of a mean of Gaussian random variables.
However, in other cases, it is not known.
If we know at least the distribution from which the individual observations were drawn, we can use Monte Carlo methods to precisely estimate the conﬁdence interval; however, if we are not able to make an assumption about the underlying distribution, we need to use only what we have: our samples from the distribution.
In such case the variability of our estimator can be approximated using the Bootstrap (Efron, 1979) or similar methods.
The Bootstrap consists of repeatedly sampling with replacement m random observations from our pool of m observations, say B times.
Each such sample is then used to calculate an estimate of our quantity of interest, say Boo n or mean.
This creates a sample of B values of the estimator.
The conﬁdence interval can then be easily estimated taking the appropriate quantiles from this resulting
Bootstrap distribution of the estimator, which should be approximating the unknown underlying sampling distribution.
The Bootstrap distribution has been shown to converge to the true underlying performance distribution.
If we know the underlying distribution (up to some parameters), we can estimate its parameters and then generate a simulated Monte Carlo sample from the distribution, which can be used to calculate a sample of the estimator and the corresponding conﬁdence interval in a similar way as above with the advantage of the distribution being smoother.
Beside estimating the conﬁdence interval for the value of Boo n or mean itself, either re-sampling method can be used to construct a conﬁdence interval for the relative improvement of the newly proposed architecture compared to a baseline.
The improvement can then be considered signiﬁcant if zero is not included in the conﬁdence interval.
More details on constructing Bootstrap conﬁdence intervals can be found in many standard texts on computational statistics, for instance in Efron (1987).
For illustration, we calculated the Bootstrap conﬁdence interval for several sample sizes m for Resnet and the AS Reader.
Each was constructed using B = 100 , 000 .
The results are plotted in Figure 2. [Figure]
A PPENDIX E C OMPARISON OF ESTIMATORS
Figure 3 shows the comparison of the non-parametric and Gaussian parametric estimators of Boo n , both introduced in Section 3.3, in terms of their variance for various sample sizes.
The parametric estimator shows a somewhat lower variance.
This is an advantage if the performance distribution is indeed approximately Gaussian, which is the case for both cases with ﬁxed hyperparameters that we tested in our experiments.
However, this can introduce bias if the true performance distribution differs from the theoretical distribution assumed by a parametric estimator, so one should be prudent to use it.