  This paper theoretically investigates the following empirical phenomenon:
given a high-complexity network with poor generalization bounds, one can
distill it into a network with nearly identical predictions but low complexity
and vastly smaller generalization bounds. The main contribution is an analysis
showing that the original network inherits this good generalization bound from
its distillation, assuming the use of well-behaved data augmentation. This
bound is presented both in an abstract and in a concrete form, the latter
complemented by a reduction technique to handle modern computation graphs
featuring convolutional layers, fully-connected layers, and skip connections,
to name a few. To round out the story, a (looser) classical uniform convergence
analysis of compression is also presented, as well as a variety of experiments
on cifar and mnist demonstrating similar generalization performance between the
original network and its distillation.
