Reproducibility Report: Your Classiﬁer is Secretly an Energy Based Model and You Should Treat it Like One
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility Summary Scope of Reproducibility
We validated the Joint Energy-Based model (JEM) training technique, recently developed by Grathwohl et al.
Speciﬁcally, we checked performance on image classiﬁcation, generation, and uncertainty calibration.
Methodology
We re-implemented the paper’s pipeline from scratch, based on the algorithm described in the paper.
We only referredto the authors’ code for subtleties such as data normalization which were not explicitly mentioned in the paper.
Training
JEM took about 12 hours using a Wide ResNet 28-2 architecture [2].
Results
We veriﬁed that JEM performed similarly to how it was presented in [1].
We could not reproduce the exact numericalresults of the paper due to the long training time of the algorithm.
What was easy
The paper is well-written, with the algorithm and motivation clearly explained.
What was difﬁcult
We were not able to reproduce the exact results of the paper since constraints on computation time forced us to use asmaller network than the authors.
Running the authors’ method with the model and hyperparameters they described intheir original paper would have required about 80 hours to train on our hardware (which we believe is comparable tothe authors’ hardware), rather than the 36 hours the authors reported in the paper.
While the training method produces a well-calibrated hybrid model, training itself is unstable.
We needed to restarttraining a few times due to the loss diverging.
Communication with original authors
We spoke to the authors who corroborated the second of the above difﬁculties, however the origin of the lengthy trainingtime is less clear.
Submitted to ML Reproducibility Challenge 2020.
Do not distribute.
Introduction
Research in generative models has historically been motivated by the potential beneﬁt to tasks such as semisupervisedlearning, imputing missing data and/or data augmentation for discriminitive tasks, and calibrating uncertainty inpredictive modelling.
However, much of the recent literature in the subject is heavily motivated by generated samplequality.
At present, there is a large gap in performance between generative modelling and downstream discriminativetasks.
This is motivated, at least in part, by stark contrast between generative and discriminative deep neural networkarchitectures.
The paper we reproduce in this report [1] bridges the gap between discrimination and generation by using a novelenergy-based approach.
Typically discriminative classiﬁers deﬁne a conditional probability distribution p ( y | x ) overthe labels y ∈ R K (for K classes), where (e.g. for images) x ∈ R c × h × w (channels, height, width).
However, byreinterpreting of the pre-softmax network outputs of a neural network, the authors of [1] deﬁne a method to model thedata distribution p θ ( x ) as
[Equation]
where E θ , known as the energy function , depends on the pre-softmax network outputs and continuously maps eachimage to a scalar.
The normalization Z ( θ ) ≡ (cid:82) x exp( − E θ ( x )) is known as the partition function .
Performing maximumlikelihood estimation by differentiating (1) with respect to θ yields
[Equation]
The authors approximate the intractable expectation in 2 by stochastic gradient langevin descent (SGLD) [3], a type of
Markov chain Monte Carlo (MCMC) algorithm.
We refer the reader to the references (particularly [1], [3]) for detailson how this is implemented in practice.
The full likelihood function that we must optimize with respect to the networkparameters θ is then given summing the generative and discriminative contributions:
[Equation]
For the purpose of this reproducibility study, we chose to replicate Algorithm 1 of [1] described in Appendix A ofthe paper.
This is the primary novel algorithm proposed, known as joint energy-based model (JEM) training forjointly training a discriminative/generative ( hybrid ) model.
This training scheme is a subset of the class hybrid modeltraining methods (hybrid due to the joint generative/discriminative nature).
The paper shows that JEM outperformsstate-of-the-art hybrid models at both generative as well as discriminative modeling.
NB: If we mention "the paper" without citation, we are referring to [1]
Scope of reproducibility
We verify the following key claims made in the paper:
We verify the validity of the JEM-based training approach by implementing the model and training pipeline
from scratch.
Due to issues with training time, we could not verify the results for the exact ResNet
architecture
described in the paper (see Section 5 section for details).
However, we trained a Wide ResNet (WRN)
28-2 (scaled down from the WRN 28-10 trained in the paper) and produced comparable generative and
discriminative performance.
We verify that training with JEM improves uncertainty calibration compared to standard cross entropy
minimization.
Methodology
We made every effort to perform an independent reproduction relying on the author’s paper, however we occasionallyhad to refer to the author’s code to obtain information not presented in the paper, particularly image preprocessinghyperparameters and the train/test split.
3.1 Model descriptions
While the paper used a Wide Resnet (WRN) 28-10, we scaled down to a WRN 28-2 for reasons of training time.
(See
Sec 5 for more details).
We did not implement the WRN architectures ourselves, but found an existing implementation(based on this code) which we used.
3.2 Datasets
We used CIFAR10 [4] for all experiments, as in the original paper.
3.3 Hyperparameters
To train the WRN 28-2 we used the same hyperparameters that the paper used to train the WRN 28-10.
While wedid not have time to optimize hyperparameters for our scaled down network, we found that we were able to producecomparable results to what was found in the paper.
Full hyperparameters are given in Table 1.
[Table]
3.4 Experimental setup and computational requirements
[Table]
Our code is available here.
All experiments were run on an AWS p3.2xlarge instance with 8 CPUs and 1 NVIDIA V100 GPU.
Training took about12 hours for JEM hybrid training and about 2 hours to train an ordinary classiﬁer using the hyperparameters given in
Table 1.
Before we scaled down our model architecture, we tried training a WRN 28-10 on an AWS p2.xlarge instancewith 4 CPUs and an NVIDIA K80 GPU; this took about 40 hours.
As a time test, we ran the authors’ code on a AWS p3.2xlarge instance.
However we projected a training time of about80 hours (as opposed to 36 as they described in their paper) based on time per iteration.
This is why we decided to scaledown our architecture.
Results
We trained a WRN 28-2 architecture using the hyperparameters described in section 3.3.
In particular, we trained both apure classiﬁer as well as a hybrid model using the same architecture.
In analogy with lines 5 and 6 of Table 1 of thepaper, we report both the accuracy of the pure discriminative performance as well as the accuracy and inception score(IS)
[5] of the hybrid model.
We also went beyond the paper by generating samples from uniform noise in 4.3, andcomputed their inception scores as well.
To compute the inception score, we adapt the code from here (code for ref [6]).
4.1 Result 1: Accuracy and Generative Samples
Table 2 compares the accuracy of the two architectures, where the 28-2 is our result and the 28-10 is taken from Table 1of the paper.
Table 3 compares both the generative and discriminative performances of the two architectures trained using JEM.
The inception scores are computed using the top 10% of most conﬁdent inception-network-classiﬁedsamples from the training buffer.
In our training pipeline, we saved checkpoints every 5 epochs.
We also did a further study of the inception scorescomputed from (a) the ﬁnal checkpointed training buffer (epoch 145), and (b) the ensembled last 5 training buffersbetween epochs 125 and 145.
For both of these, we computed the inception score over the top k most conﬁdent samples(according to the inception network) for k ∈ { 100 , 500 , 1000 , 5000 , 10000 } .
The results for these are shown in Tables4 and 5 respectively.
Figure 1 shows the 64 most conﬁdent (according to the inception network) samples taken from thelast saved checkpointed buffer.
[Table] [Table]
4.2 Result 2: Calibration
[Table]
The expected calibration error (ECE) is a method to measure how well a model is calibrated – the extent to which amodel’s conﬁdence predicts the true likelihood.
Given training samples x i , a classiﬁer’s conﬁdence on an exampleis deﬁned as max y p ( y | x i ) .
Binning these conﬁdences into equally spaced buckets { B m } Mm =1 , we can compute theaccuracy as well as conﬁdence of the model evaluated on all examples in B m for each m .
These are given by
[Table]
[Table]
[Equation]
where ˆ y i and ˆ p i are respectively the prediction and conﬁdence of each example.
The ECE is then given by
[Equation]
An ideally calibrated model has an ECE of zero.
We provide calibration plots for both ordinary supervised training aswell as JEM for the WRN 28-2 architecture in Figure 2. [Figure]
4.3 Additional results not present in the original paper
We note that that in the original paper, "generated" samples were drawn from the training buffer of the last savedcheckpoint, rather than having images generated from scratch.
To emulate a true generative model, we created a "buffer"of size 100, initializing it with random uniform noise over [ − 1 , 1] .
We then evolved it for 5, 10, 20, 50, and 100 SGLDsteps to see how well the model could generate images from scratch.
See Figure 3 for generated samples at theseevolution steps.
For reference, we also compute the inception scores of each of SGLD-evolved generated samples, which are given in Table 6.
[Table]
Discussion
Our results generally tend to corroborate the validity of the JEM method to train hybrid generative/discriminativemodels.
In particular, we were able to achieve comparable discriminative accuracy according to Table 3, and thesamples we generated both from the ﬁnal training buffer (Figure 1) as well as from uniform random noise (Figure 3)produced slightly unusual but decent-quality images.
We, however, note that the inception scores we obtained for the generated images were puzzling, as we would expectthem to be lower than what was obtained in the paper (appendix B, table 5 (unconditional) – 7.82 for the last checkpointand 7.79 for the ensemble).
However we obtain larger scores; it seems odd that a less complex model would produceseemingly higher quality images.
One possible source of this discrepancy is the way in we calculated the scores.
Wepicked the images with the top k most conﬁdent predictions according to the inception network, rather than with respect to the WRN architecture we trained.
Looking above Table 5 in Appendix B of the paper, we notice that they say thatthey compute the inception score by "keeping the top 10 percentile samples with highest p ( y | x ) values".
We assumethat p ( y | x ) refers to the network they trained, and not inception.
What’s further interesting is that the images generated from uniform random noise have signiﬁcantly lower inceptionscores than those sampled from the training buffer of the last checkpoint.
We hypothesize that this is because the trainingbuffer images evolve many more SGLD steps over the course of training.
While the generated images qualitatively lookgood, we plan to run more experiments with different architectures in the next iteration to understand the distribution ofthe inception score better.
5.1 What was easy
This paper was well-written, and the training algorithm (Algorithm 1 of Appendix A of [1]) was clearly described sowe were able to implement it almost directly from the paper.
5.2 What was difﬁcult
Due to runtime, we found that we were not able to exactly replicate the results of the paper – we found that running theauthors’ original code on the p3 instance would have taken about 80 hours to train a WRN-28-10 model, rather than 36as they’d said in their paper.
However, even though we used this training method on a scaled-down model (WRN 28-2rather than WRN 28-10 like done in the paper), we feel that the method does generalize to other architectures.
In the paper [1], the authors claimed to be able to train a WRN-28-10 model using the JEM training method in about 36hours on a single GPU.
However, even when we ran their code on our hardware (mentioned in Sec 3.4), we extrapolatedthat it would take at least 80 hours to train a full-scale 28-10 model.
Hence, for this initial reproducibility, we decidedto scale down the model architecture while trying to test the efﬁcacy of the method.
One other subtle difﬁculty which we encountered was that while the result of training neural networks with the JEMmethod produces a well-performing hybrid model, the training itself is highly unstable.
The JEM loss function (eqs.(2), (8) of [1]) is prone to diverging, hence requiring restarting training from the most recent saved checkpoint.
Whilethis was mentioned in Appendix H3 of [1], we felt that this could have been stated and explored more clearly.
5.3 Communication with original authors
We did communicate with the original authors, who corroborated the second of the above difﬁculties described.
Theorigin of the lengthy training time, however, is less clear.