This paper studies probability distributions ofpenultimate activations of classification networks. We show that, when a classification network istrained with the cross-entropy loss, its final classi-fication layer forms aGenerative-Discriminativepairwith a generative classifier based on a specificdistribution of penultimate activations. More im-portantly , the distribution is parameterized by theweights of the final fully-connected layer, and canbe considered as a generative model that synthe-sizes the penultimate activations without feedinginput data. We empirically demonstrate that thisgenerative model enables stable knowledge dis-tillation in the presence of domain shift, and cantransfer knowledge from a classifier to variationalautoencoders and generative adversarial networksfor class-conditional image generation.
This paper studies probability distributions of penultimate activations of classification networks. We show that, when a classification network is trained with the cross-entropy loss, its final classification layer forms a Generative-Discriminative pair with a generative classifier based on a specific distribution of penultimate activations. More importantly , the distribution is parameterized by the weights of the final fully-connected layer, and can be considered as a generative model that synthesizes the penultimate activations without feeding input data. We empirically demonstrate that this generative model enables stable knowledge distillation in the presence of domain shift, and can transfer knowledge from a classifier to variational autoencoders and generative adversarial networks for class-conditional image generation.
R
ofpenultimate
of penultimate
R
istrained
is trained
R
classi-fication layer forms aGenerative-Discriminativepairwith a
classification layer forms a Generative-Discriminative pair with a
R
specificdistribution
specific distribution
R
im-portantly
importantly
R
theweights
the weights
R
canbe
can be
R
synthe-sizes
synthesizes
R
feedinginput
feeding input
R
thisgenerative
this generative
R
dis-tillation
distillation
R
cantransfer
can transfer
R
variationalautoencoders
variational autoencoders
R
networksfor
networks for