G ENERALIZATION BOUNDS VIA DISTILLATION
This paper provides a suite of mathematical tools to bound the generalization error of networks that possess low-complexity distillations — that is, when there exist simple networks whose softmax outputs approximately match those of the original network.
The primary contribution is the aforementioned bound, which upper bounds the test error of a network by the sum of its training error, the distillation error, and the complexity of the distilled network.
Supporting this, secondary contributions include: a generalization bound which can handle convolutions and skip connections, a generalization analysis of the compression step leading to a bound with small width- and depth-dependence via weight matrix stable ranks, and a sampling theorem to sparsify dense networks.
The bounds and their behavior are illustrated empirically on the standard mnist and cifar datasets.
O VERVIEW AND MAIN RESULTS
Generalization bounds are statistical tools which take as input various measurements of a predictor on training data and output a performance estimate for unseen data — that is, they estimate how well the predictor generalizes to unseen data.
Despite extensive development spanning many decades (Anthony & Bartlett, 1999), there is growing concern that these bounds are not only disastrously loose (Dziugaite & Roy, 2017), but worse that they do not correlate with the underlying phenomena (Jiang et al., 2019b) and even that the basic method of proof is doomed (Zhang et al., 2016; Nagarajan & Kolter, 2019).
As a quick calculation backing the issue the largest network in
Figure 1 has a modest size: 6 layers each with 1024 nodes and generalizes just ﬁne — but the VC dimension is 30 million, which is 500 times larger than the amount of data, giving an upper bound on the probability of misclassiﬁcation of roughly √ 500 .
Contrary to this dilemma there is evidence that these networks can often be compressed or distilled into simpler networks — ones, ostensibly, whose generalization is reasonably analyzed.
Indeed, this line of thought predates the preceding generalization concerns (Bucilu ˘ u et al., 2006), but has received increasing attention in recent years (Hinton et al., 2015).
Indeed, it can be found in many guises, for instance throughout the pruning or lottery ticket literature, which posits that vast swaths of parameters can simply be deleted with minimal harm to network performance (Frankle & Carbin, 2019; Frankle et al., 2020; Su et al., 2020).
The purpose of this work is to directly apply the distillation insight to tackle the generalization dilemma.
Conceptually, this goes beyond the standard distillation perspective, whose goal is to start with a complicated network, but to eventually discard it and utilize a simpler network which has hopefully inherited certain key properties.
Instead, the perspective here is that the distilled network absorbs the original network so wholly that it can be used to answer questions about the original network.
Concretely, the questions we answer here about networks via their distillations are generalization upper bounds.
Rather than a standard generalization upper bound in terms of network complexity, our core contribution is an elementary and widely-applicable upper bound on generalization via distillation properties:
test error (cid:46) training error + distillation error + distillation complexity .
As will be outlined momentarily, this work also provides (a) an instantiation of this lemma applicable to residual networks, (b) an explicit compression analysis via pruning with a corresponding
[Figure]
generalization bound, (c) empirical support, most notably showing width-independence subsequent to distillation.
1. C ORE DISTILLATION LEMMA , AND A CONVENIENT COROLLARY
This subsection will present the core distillation lemma in an abstract form, which lacks any mention of deep networks.
After some discussion, a corollary for general computation graphs will follow.
To build up to the lemma, ﬁrst recall the distillation setup.
Given a multi-class predictor f : R d → R k , the goal is to ﬁnd another predictor g : R d → R k which is simpler, but still similar in distillation error Φ γ,n , deﬁned along with φ γ , the softmax with temperature γ > 0 , as Φ γ,n ( f, g ) := 1 n n (cid:88) i = (cid:12)(cid:12) φ
[Equation]
( f ( x
[Equation]
) , y i ) − φ
[Equation]
( g ( x i ) , y i )
[Equation]
In other words, the softmax outputs for the original predictor f and its distillation g must be close on average along coordinates corresponding to the true label y .
The quantity γ > 0 is sometimes called a temperature (Hinton et al., 2015), but in the present work it also plays the role of a margin : indeed, the statistical learning theory literature has used such a γ in a softmax-like loss function to produce sophisticated generalization bounds via the margins perspective (Srebro et al., 2010, Section 5.1).
There are many variants of this scheme, for instance matching the outputs of f and g directly (Bucilu˘u et al., 2006), but the softmax is relied upon heavily in the present work.
The core lemma is as follows, but here is its ﬂavor in words.
We are handed a function f , and we don’t understand its generalization properties.
However, we have some simple function classes ( G j ) j ≥ 1 which we understand pretty well, and we have some prior belief on which G j are close to f in the sense of distillation error, namely Φ γ,n .
We compare f to these G j one by one, and when the distillation error is small, we announce that the original f inherits the generalization properties of G j , despite not necessarily looking anything like the functions in G j .
Lemma 1.1.
Let temperature (margin) parameter γ > 0 be given, along with any sets of multiclass predictors ( G j ) j ≥ 1 with G j (cid:51) g : R d → R k , along with prior weights ( p j ) j ≥ 1 on these sets, where p j ≥ 0 and (cid:80) j p j = 1 .
Then with probability at least 1 − δ over an iid draw of data (( x i , y i ))
n i =1 , every measurable function f : R d → R k satisﬁes
[Equation]
As was foreshadowed, the left hand side is the test error, and the right hand side searches over the classes G j , penalized by the prior ln(1 /p j ) , considering the best g ∈ G j within each in terms of distillation measure Φ γ,n .
One thing that was not foreshadowed is that the training error is measured with the softmax as well — this is ﬁne, since, in fact, it is upper bounded by the cross-entropy loss, which is the typical objective function in the setting.
The other new element is the clunky distillation complexity term; but in fact, it is just the Rademacher complexity of the multi-class margins g ( x ) y − max i (cid:54) =
y g ( x ) i of G j , and as is usual in margin analyses is scaled by 1 / γ .
Now that the bound has been stated, let us return to the initial context of the paper, namely the difﬁculty of producing generalization bounds for deep networks.
Suppose, as was discussed extensively in (Zhang et al., 2016), that a deep network is trained on random labels, and this is plugged in as f to
Lemma 1.1.
Achieving good distillation error Φ γ,n will therefore require high complexity, scaling with the number of examples, and it will not be possible to select a simple g .
On the other hand, if f is trained on true labels, then ﬁnding a simple g should be possible.
Indeed, Figure 1 aims to capture this by training 3 different network architectures on mnist — each fully-connected with 6 layers and 2 skip connections, the architectures differing by choosing widths { 64 , 256 , 1024 } .
Despite the original generalization bounds (and corresponding margin histograms) differing greatly, they all distill to a nearly identical simple predictor with good generalization properties.
As mentioned, and as will be discussed in detail in Section 2, the bounds are normalized by a speciﬁc generalization bound, meaning a speciﬁc choice of G j and Rademacher complexity upper bound.
Skip connections have nice properties when combined with distillation, so another contribution of this work is a Rademacher complexity bound which can handle ﬂexible computation graphs including convolutions, multi-variate gates, and skip connections.
Theorem 1.2.
Let temperature (margin) parameter γ > 0 be given, along with a ﬁxed computation graph architecture G (as detailed in Section 1.3), parameterized by per-layer batch bounds (cid:126)b , perlayer parameter (2 , 1) -norm bounds (cid:126)r layer operator norm bounds (cid:126)s .
With probability at least 1 − δ over an iid draw of data (( x i , y i ))
ni =1 , every measurable function f : R d → R k satisﬁes
[Equation]
where b i depends on ( x i ) ni =1 and should be interpreted as O ( √ n ) .
This bound and its proof will be discussed in detail in Section 3.
Brieﬂy, it follows from a simpliﬁed proof of the covering number-based bound due to Bartlett et al.
(2017a)
; this original proof was also heavily extended to analyze a different computation graph formalism by Wei & Ma (2019).
In addition to the generality, the bound also uses b i to upper bound the Frobenius norm of the input to each layer, which lends signiﬁcant numerical beneﬁts and again simpliﬁes the proof.
Compared with Lemma 1.1, this bound makes a ﬁxed choice of ( G j ) and their prior probabilities, namely a ﬁxed computation graph, and preferring smaller norms.
This bounds was used to produce
Figure 1, as will be discussed in Section 2.
1. D ISCUSSION : why CAN PREDICTORS BE DISTILLED ?
The focus of this paper is on arguing that if a predictor can be distilled to a simple network, then it generalizes well.
A separate question, only scratched here, is why networks can be distilled to simpler ones.
This subsection provides a generalization bound where, in a sense, the distillation is explicitly analyzed.
Concretely, the bound ﬁrst simpliﬁes a network by sparsifying its internal connections, and then applies a generalization bound to the sparsiﬁed network.
The sparsiﬁcation step can be interpreted as node pruning, and could be of independent interest; it is discussed in Section 3.
Lemma 1.3.
Let data matrix X ∈ R n × d be given.
Let H denote networks of the form x
(cid:55)→ σ L ( W L · · · σ 1 ( W 1 x )) with (cid:107) W i (cid:107) 2 ≤ s i and σ i is 1 -Lipschitz and 1-homogeneous, and (cid:107) W
i (cid:107) F ≤ R i and width at most m .
Then
[Equation]
The term (cid:107) X (cid:107) F /n 3 / 4 should be interpreted as O ( n − 1 / 4 ) , and deserves mention.
It arises because the network is ﬁrst sparsiﬁed with knowledge of the data , effectively baking data into the network, and making it unclear, at least with the present proof, if the usual 1 / √ n is possible.
On the other hand, the bound does not seems to have any explicit appearance of network width: the term R
i /s
i is the square root of the stable rank , and there is hope it is insensitive to ambient network width.
While the bound has certain interesting properties, it is empirically usually weaker than Theorem 1.2, though it is slightly better in the early distillation phase (where both bounds are vacuously large).
Is sparsiﬁcation the only thing that can lead to distillation?
Probably not; this question is left to future work.
1. N OTATION
Given data ( z i ) ni =1 , the Rademacher complexity of predictors H is
[Equation]
where (cid:15) i i.i.d.
∼ Uniform ( {− 1 , +1 } ) .
The Rademacher complexity is most common tool in generalization theory (Shalev-Shwartz & BenDavid, 2014), and is incorporated in Lemma 1.1 due to its convenience and wide use.
Canonical computation graphs are a ﬂexible way to write down feedforward networks which allow for dense linear layers, convolutional layers, skip connections, and multivariate gates amongst other options, while still having an economical expression.
To start the deﬁnition, given matrices
A ∈ R m × d and B ∈ R n × d let [ A |(cid:105) B ] ∈ R ( m + n ) ×
d denote their row-wise concatenation .
Given a minibatch X ∈ R n × d , a canonical computation graph is recursively deﬁned via
[Equation]
where: σ i is a multivariate-to-multivariate 1-Lipschitz function (measured over minibatches on either side with Frobenius norm), where the “ 1 ” is for notational convenience; F i is a ﬁxed matrix, for instance an identity mapping as in a residual network’s skip connection; D i is a ﬁxed diagonal matrix selecting certain coordinates, for instance the non-skip part in a residual network; Π i is a
Frobenius norm projection of a full minibatch; W i is a weight matrix, the trainable parameters.
In
Theorem 1.2, r i is an upper bound on (cid:107) W T i (cid:107) 2 , 1 ≤ r i (sum of the (cid:107) · (cid:107) 2 -norms of the rows), b i is the radius of the Frobenius norm ball which Π
i is projecting onto, and s i is the operator norm of
X (cid:55)→ [ W i Π
i D i X T |(cid:105)
F i X T ] .
While the deﬁnition is intricate, it can not only model basic residual networks, but it is sensitive enough to be able to have s i
= 1 and r i =
when residual blocks are fully zero’d out: this indeed happens during distillation (cf.
Figure 3b), and motivated this deﬁnition and its analysis.
A key property or “trick” of this architecture is that it always operates at the level of batches.
In particular, since every layer takes as input matrices of examples, it can output matrices with seemingly [Figure] more examples.
This is how convolution layers are handled: the input tensor is cut up into many patches, each patch effectively counting as a new example, and this is then fed into the convolution ﬁlter which now can look like a regular dense linear layer.
Since the activations are arbitrary Lipschitz multivariate gates, they can encode this operation of exploding into patches and combining back into single tensors.
A few more pieces of notation.
Throughout, (cid:107) · (cid:107) =
(cid:107) · (cid:107) F will denote Frobenius norm, and (cid:107) · (cid:107)will denote spectral norm.
The sigmoid will often be used: σ s ( z )) :
= (1 + exp( − z )) − 1 .
I LLUSTRATIVE EMPIRICAL RESULTS
This section will summarize the experimental setup, and then provide further comments on the various experiments backing the ﬁgures throughout the paper.
Experimental setup.
Experiments were conducted in a simple manner, as follows.
The standard mnist data was combined with fully-connected networks with skip links in the following architecture: given a width m ∈ { 64 , 256 , 1024 } , the ﬁrst layer had shape m × d , and thereafter pairs of layers (each with shape m × m ) had parallel skip connections, with ReLUs throughout, with a ﬁnal fully connected layer mapping to the 10 output labels.
The standard cifar data used a similar architecture, but with the dense layers in the skip blocks replaced with 2-dimensional convolutions using a kernel width of 3 .
The general training procedure, as follows, was effectively an implementation of Theorem 1.
A network f was trained to roughly zero training error with mini-batch gradient descent on the cross-entropy loss with with no regularization or other techniques.
Thereafter, distilled networks were produced via a relaxation sequence : f was duplicated to produce a network g , which was then trained via gradient descent on the objective Φ 1 ,n ( f, g ) + λ (cid:80) j (cid:107) W T j (cid:107) 2 , 1 , where λ was gradually increased from roughly zero until g became the 0 network.
To select a single g from this sequence, a few candidates were compared via their normalized margin distribution (to be discussed in detail momentarily), and the g whose 10% margin quantile was largest was selected.
This explains why the plots can sometimes differ in Φ( f, g ) while still having margin distributions far to the right of zero.
Margin histograms.
These histograms originated in the study of boosting methods (Schapire &
Freund, 2012), but have proved useful in studying deep networks as well (Bartlett et al., 2017a; Jiang [Figure] et al., 2019a).
The concrete approach here, as introduced in (Bartlett et al., 2017a), is to normalize the entire margin distribution by (some upper bound on) the Rademacher complexity Rad ( G ) for the predictor family: given g ∈ G ,
[Equation]
By this choice, margin distributions can be compared across predictor families and even datasets.
Moreover, the margin diagrams have a concrete interpretation: if most of the normalized margins fall to the right of some value c , then a generalization bound of roughly 1 / c is implied.
Width indepedence.
Now that the experimental setup has been detailed, Figure 1 is saying: starting from networks of different widths and then slowly applying more regularization, the best networks amongst all regularization paths have similar generalization properties.
Note that the distilled networks have negative margins on a fraction of the data: this trend was observed across experiments, and seems to imply that distillation quickly gives up on difﬁcult examples in order to quickly decrease complexity.
Depth.
Figure 3 shows two effects coming from massive depth.
In Figure 3a, there are many blocks, but still distillation can pull the sharply concentrated margin distribution away from the origin towards a nearly common place.
Figure 3b shows the generalization bound (from Lemma 3.2), test error, and number of block deletions on a network with 8 blocks (20 layers), each of width 64, all as a function of distillation error Φ γ,n .
Notably, the test error remains ﬂat until just before Φ γ,n reaches 1 / 10 .
Interestingly, the ﬁrst block deletion (deﬁned as weight matrix norms falling below/ 100 has no initial effect on test error.
As Figure 3a is the only appearance of cifar and convolutional networks, it is worth pointing out that the bound is much looser in this regime, offering a valuable direction for future work.
Random labels.
A standard sanity check for generalization bounds is whether they can reﬂect the difﬁculty of ﬁtting random labels (Zhang et al., 2016).
While it has been empirically shown that
Rademacher bounds do sharply reﬂect the presence of random labels (Bartlett et al., 2017a, Figures 2 & 3), the effect is extremely clear with distillation: even randomizing just 25% shrinks the margin distribution by multiple orders of magnitude.
A NALYSIS OVERVIEW AND SKETCH OF PROOFS
This section sketches the proofs of all the main results, and provides further context and connections to the literature.
Full detailed proofs appear in the appendices.
3. P ROOF SKETCH OF CORE DISTILLATION L EMMA 1.
The core lemma, Lemma 1.1, is a corollary of the following more fundamental core lemma.
Lemma 3.1.
Let ( G j ) j ≥ 1 be a sequence of classes of bounded functions, meaning G j (cid:51) g : R d → [0 , 1] , and let ( p j ) j ≥ 1 with p j ≥ 0 and (cid:80) j p j = 1 be corresponding prior weights.
Then with probability at least 1 − δ over an iid draw of S :
= ( z 1 , . .
. , z n ) , every bounded measurable function f : R d → [0 , 1] satisﬁes
[Equation]
On the one hand, this bound appears natural, indeed it has the ﬂavor of model selection theorems.
On the other hand, the bound is on f , which is not chosen via model selection, and instead a model selection penalty is applied after the fact.
In this generality, this tool can be applied in multiple places throughout the literature.
For instance, if pruning results chose to minimize distillation error rather than achieving good prediction performance (Frankle & Carbin, 2019; Frankle et al., 2020; Su et al., 2020), they could imply generalization properties of the original network.
Secondly, there are many generalization bounds which apply to a compressed or distilled network, but not the original, and Lemma 3.1 can be used to close this gap and provide a generalization bound for the original network.
As a concrete example, Arora et al.
(2018) provide a procedure to compress/distill a network (cf.
their Algorithm 1) and a generalization bound for the distilled network (cf.
their Theorems 2.1, 4.1, 5.1, providing our distillation complexity term), but no bound on the test error of the original network (cf.
Remark (1) below their Theorem 2.1); consequently, our work can be used to complement theirs by computing the distillation error term of their compression procedure (either empirically or via their Lemma 3) and then applying
Lemma 3.1 to obtain a generalization bound for the original uncompressed network.
Turning now to the proof, let F denote bounded measurable functions, speciﬁcally F (cid:51) f
: R d → [0 , 1] .
It is now crucial that ( G j ) j ≥ 1 was chosen before seeing f or data: given a target distillation error τ k , carve up all of F into strips based on G j :
[Equation]
This quantity is akin to an empirical covering number, however instead of starting with some class of functions and deriving a covering set, we start with a covering set G j , and write down the largest subset of F which it can cover.
Now that the cover formalism has been pinned down, the rest of the proof is easy.
We can get a
Rademacher complexity bound on each F j,k in the standard covering number way (Shalev-Shwartz & Ben-David, 2014).
Then unioning over all over all ( G j ) j ≥ 1 (this is where the prior comes in) and over a grid of choices for τ k , we obtain a family of bounds which hold simultaneously for all F j,k , and can thus safely apply an inﬁmum, do some algebra, and obtain the desired bound.
Turning back to the proof of Lemma 1.1, consider taking each function class G j and composing it with 1 − φ γ , giving Q j := (cid:8) ( x, y )
(cid:55)→ 1 − φ γ ( g ( x ) , y ) : g ∈ G j (cid:9) , and similarly composing f with 1 − φ γ .
All of these post-composition functions are bounded, and Lemma A.1 can be readily applied, giving roughly Lemma 1.1, albeit with a left hand side in terms of 1 − φ γ .
To lower bound the left hand side with the misclassiﬁcation probability, we use the fact that 2(1 − φ γ ) upper bounds the misclassiﬁcation loss; this is given by Lemma A.1, which can also be used to massage the right hand side into the ﬁnal form in Lemma 1.1.
Full details are given in the appendices; moreover, as detailed in Appendix E.3, this general approach can replace 1 − φ γ with any bounded and wellbehaved loss, for instance a standard margin loss.
3. P ROOF OF T HEOREM 1.
The proof follows by plugging the following Rademacher complexity estimate for canonical computation graphs into Lemma 1.1.
Lemma 3.2.
Let data X ∈ R n × d be given.
Let computation graph G be given, where Π i projects to
Frobenius-norm balls of radius b i , and (cid:107) W T i (cid:107) 2 ,
≤ r i , and (cid:107) [ W i Π i D i |(cid:105)
F i ] (cid:107) 2 ≤ s i , and Lip ( σ i ) ≤for convenience, and all layers have width at most m .
Then for every (cid:15) > 0 there exists a covering set M satisfying
[Equation]
and
[Equation]
Consequently,
[Equation]
As discussed before, this proof follows the scheme from (Bartlett et al., 2017a), though simpliﬁed in various ways, owing mainly to the omission of general matrix norm penalties on weight matrices, and the omission of the reference matrices .
The reference matrices were a technique to center the weight norm balls away from the origin; a logical place to center them was at initialization.
However, in this distillation setting, it is in fact most natural to center everything at the origin, and apply regularization and shrink to a well-behaved function (rather than shrinking back to the random initialization, which after all deﬁnes a complicated function).
Also provided is a short re-proof of the core covering number bound for (2 , 1) -norm bounded matrices (cf.
Lemma B.3).
Other than that, the generality of computation graphs was in fact conveniently handled in the original proof scheme.
3. P ROOF SKETCH OF L EMMA 1.
As was mentioned before, the ﬁrst step of the proof is to sparsify the network.
The approach followed here is to sparsify each matrix product.
Concretely, given weights W i of layer i , and letting X T i −denote the input to this layer, then
[Equation]
Written this way, it seems natural that the matrix product should “concentrate”, and that considering all m outer products should not be necessary.
Indeed, exactly such an approach has been followed before to analyze randomized matrix multiplication schemes (Sarlos, 2006; Hsu et al., 2012).
As there is no goal of high probability here, the analysis is simpler, and follows from the Maurey lemma (cf.
Lemma B.1), as is used in the (2 , 1) -norm matrix covering lemma in Lemma B.3.
Lemma 3.3.
Let a network be given with ρ i -Lipschitz homogeneous activations σ i and weight matrices ( W 1 , . .
, W L ) of maximum width m , along with data matrix X ∈ R n × d and desired widths ( k 1 , . . .
, k L ) be given.
Then there exists a sparsiﬁed network output, recursively deﬁned via ˆ X T 0 :
= X T , and ˆ X T i :
= Π i σ i ( W i M i X
T i − 1 ) , where M i := (cid:88)
[Equation]
A e j (cid:107) , where S i is a multiset of k i = | S i | indices, Π i denotes projection onto the Frobenius-norm ball of radius (cid:107) X (cid:107) F (cid:81)
j ≤ i (cid:107) W j (cid:107) 2 , and the scaling term Z j satisﬁes Z j ≤ (cid:107) W k (cid:107) F (cid:112) m/k j , and (cid:107) σ L ( W L · · · σ 1 ( W 1 X T ) · · · ) − ˆ X T L (cid:107) F ≤ (cid:107) X (cid:107) F  
L (cid:89) i
= (cid:107) W
i
cid:107) 
L (cid:88) i = (cid:115) sr
( W i ) k i , where sr ( W i ) :
= (cid:107) W
i (cid:107) 2 F / (cid:107) W
i
(cid:107) 2 2 denotes the stable rank of W i .
As a brief note, it would be wonderful if a generalization bound with this error estimate could be proved, but this is only a sparsiﬁcation bound.
The statement of this lemma is lengthy and detailed because the exact guts of the construction are needed in the subsequent generalization proof.
Speciﬁcally, now that there are few nodes, a generalization bound sensitive to narrow networks can be applied.
On the surface, it seems reasonable to apply a VC dimension bound, but this approach did not seem to yield a rate better than n − 1 / 6 , and also had an explicit dependence on the depth of the network, times other terms visible in Lemma 1.3.
Instead, the approach here, aiming for a better dependence on n and also no explicit dependence on network depth, was to produce an ∞ -norm covering number bound (see (Long & Sedghi, 2019) for a related approach), with some minor adjustments (indeed, the ∞ -norm parameter covering approach was applied to obtain a Frobenius-norm bound, as in Lemma 3.2).
Unfortunately, the magnitudes of weight matrix entries must be controlled for this to work (unlike the VC approach), and this necessitated the detailed form of Lemma 3.3 above.
To close with a few pointers to the literature, as Lemma 3.3 is essentially a pruning bound, it is potentially of independent interest.
Secondly, there is already one generalization bound in the literature which exhibits spectral norms, due to (Suzuki et al., 2019); unfortunately, it also has an explicit dependence on network width. A CKNOWLEDGMENTS
ZJ and MT are grateful for support from the NSF under grant IIS-1750051, and from NVIDIA under a GPU grant.
A P ROOFS OF MAIN DISTILLATION BOUNDS , L EMMAS 1.1 AND 3.
The proof of the core abstract lemma, Lemma 3.1, is now provided in full.
Proof of Lemma 3.1.
For each j ≥ 1 and k ∈ { 1 , . . .
, lg( n ) } , deﬁne
[Equation]
k and
[Equation]
whereby (cid:80) j,k δ j,k = lg( n ) (cid:80) j p j δ/ lg( n ) = δ .
Let F denote all measurable functions from R d to [0 , 1] , and carve it up into strips corresponding to failure probabilities δ j,k : for each ( j, k ) , deﬁne
[Equation]
By these deﬁnitions,
[Equation]
By standard Rademacher complexity bounds (Shalev-Shwartz & Ben-David, 2014), with probability at least 1 − δ j,k , for every f ∈ F j,k ,
[Equation]
To prove the desired statement, discard the union of all preceding failure events, meaning that with probability at least 1 − δ = 1 − (cid:80) j,k δ j,k , eq.
(A.1) holds simultaneously for all f ∈ F and ( j, k ) .
Fixing f , for any j and g ∈ G j , then (cid:80) i | f
( z i ) − g
( z i ) | /n ≤ 2 , thus there exists k so that f ∈ F j,k , and we may safely choose k to be the smallest such choice.
This smallest k must satisfy
[Equation]
where the ﬁnal slop comes from handling the case k = lg( n ) , and thus
[Equation]
Consequently, applying an inﬁmum over ( j, k ) for all instantiations of eq.
(A.1),
[Equation]
[Equation]
n
[Equation]
and the ﬁnal bound in the statement comes via the general inequality √ a + b ≤ √ a + √ b for arbitrary nonnegative ( a, b ) .
To establish Lemma 1.1, convenient properties of φ γ are ﬁrst handled.
Lemma A.1.
For any v ∈ R k and y ∈ { 1 , . . . , k } , letting σ s ( z ) :
= (1 + exp( − z )) − 1 denote the sigmoid,
[Equation]
Proof.
First note that the sigmoid σ s ( z ) :=
(1 + exp( − z )) − 1 satisﬁes 2 σ s ( z ) ≥ 1 [ z ≥ 0] , since σ s is monotone nondecreasing and σ s (0) = 1 / 2 .
Therefore, setting u := v/γ for convenience, and recalling that
[Equation]
then
[Equation]
The proof of Lemma 1.1 is a combination of Lemma A.1 and Lemma 3.1.
Proof of Lemma 1.1.
Deﬁne ψ := 1 − φ γ and Q j :=
{ ψ ◦ g : g ∈ G j } .
Since ψ has range [0 , 1] , it follows by Lemma 3.1 with probability at least 1 − δ that for any measurable f : R d → R k , E [Equation] , y
[Equation]
To simplify this expression, note ﬁrstly that
[Equation]
for every i .
Secondly, letting σ s denote the sigmoid (which is 1 / 4 -Lipschitz), by the properties of φ γ in Lemma A.1 and the Lipschitz peeling property of Rademacher complexity (Shalev-Shwartz & Ben-David, 2014), [Equation] =
Rad (cid:32)(cid:26)
[Equation]
Rad
[Equation]
max
[Equation]
For the ﬁnal simpliﬁcation, note again by Lemma A.1 that
[Equation]
thus combining all simpliﬁcations and multiplying everything by 2 gives the ﬁnal bound.
B S AMPLING TOOLS
The proofs of Lemma 3.2 and Lemma 3.3 both make heavy use of sampling.
Lemma B.1 (Maurey (Pisier, 1980)) .
Suppose random variable V is almost surely supported on a subset S of some Hilbert space, and let ( V 1 , . .
, V k ) be k IID copies of V .
Then there exist ( ˆ V 1 , . . .
, ˆ V k ) ∈
S k with
[Table]
[Equation]
[Table]
Proof of Lemma B.1.
The ﬁrst inequality is via the probabilistic method.
For the remaining inequalities, by expanding the square multiple times,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
A ﬁrst key application of Lemma B.1 is to sparsify products, as used in Lemma 3.3.
Lemma B.2.
Let matrices A ∈ R d × m and B ∈ R n × m be given, along with sampling budget k .
Then there exists a selection ( i 1 , . .
, i k ) of indices and a corresponding diagonal sampling matrix
M with at most k nonzero entries satisfying
[Equation]
and
[Equation]
Proof of Lemma B.2.
For convenience, deﬁne columns a i :=
A e i and b i :=
B e i for i ∈ { 1 , . . .
, m } .
Deﬁne importance weighting β
i :=
( (cid:107) a i (cid:107) / (cid:107) A (cid:107) F ) 2 , whereby (cid:80) i β i = 1 , and let V be a random variable with
[Equation]
whereby
[Equation]
By Lemma B.1, there exist indices ( i 1 , . .
, i k ) and matrices
ˆ V j :=
β − 1 i j a i j b T i j with
[Equation]
AB
[Equation]
To ﬁnish
[Equation]
by the deﬁnition of M ,
[Equation]
A second is to cover the set of matrices W satisfying a norm bound (cid:107) W T (cid:107) 2 , 1 ≤ r .
The proof here is more succinct and explicit than the one in (Bartlett et al., 2017a, Lemma 3.2).
Lemma B.3 (See also (Bartlett et al., 2017a, Lemma 3.2)) .
Let norm bound r ≥ 0 , X ∈ R n × d , and integer k be given.
Deﬁne a family of matrices
[Equation]
Then
[Equation]
k .
Proof.
Let W ∈ R m × d be given with (cid:107) W T (cid:107) 2 , 1 ≤ r .
Deﬁne s ij :=
W ij / | W ij | , and note
[Equation]
Note by Cauchy-Schwarz
that
[Equation]
potentially with strict inequality, thus q is not a probability vector.
To remedy this, construct probability vector p from q by adding in, with equal weight, some U ij and its negation, so that the above summation form of WX T goes through equally with p and with q .
Now deﬁne IID random variables ( V 1 , . .
, V k ) ,
where
Pr [ V l = U ij ] = p ij ,
[Equation]
X (cid:107) F = r (cid:107) X (cid:107) F ,
E (cid:107) V l (cid:107) 2 = (cid:88) i,j p ij (cid:107) U ij (cid:107) 2 ≤ (cid:88) ij p ij r 2 (cid:107) X (cid:107) 2 F = r 2 (cid:107) X
(cid:107) 2 F .
By Lemma B.1, there exist ( ˆ V 1 , . . .
, ˆ V k ) ∈
S k with (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
WX T − 1 k (cid:88)
l ˆ V l (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ E (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
E V 1 − 1 k (cid:88) l
V l (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤k E (cid:107) V 1 (cid:107) 2 ≤ r 2 (cid:107) X (cid:107) 2 F k .
Furthermore, the matrices ˆ V l have the form k (cid:88)
l ˆ V l = 1 k (cid:88) l s l e i l
( X e j l ) T (cid:107)
X
e j l (cid:107) =
  1 k (cid:88)
l s l e i l e T j l (cid:107)
X e j l (cid:107) 
 X T
=: ˆ WX T , where ˆ W ∈ M .
Lastly, note |M| has cardinality at most (2 nd ) k .
C P ROOF OF T HEOREM 1.2 AND LEMMA 3.
The ﬁrst step is the Rademacher complexity bound in Lemma 3.2, which follows the proof scheme laid out in (Bartlett et al., 2017a), with simpliﬁcations due to the removal of “reference matrices” and some norm generality.
Proof of Lemma 3.2.
Let cover scale (cid:15) and per-layer scales ( (cid:15) 1 , . .
, (cid:15) L ) be given; the proof will develop a covering number parameterized by these per-layer scales, and then optimize them to derive the ﬁnal covering number in terms of (cid:15) .
From there, a Dudley integral will give the Rademacher bound.
As in the statement, recursively deﬁne
[Equation]
The proof will recursively construct an analogous cover via
[Equation]
where the choice of ˆ W i depends on ˆ X i − 1 , and thus the total cover cardinality will product (and not simply sum) across layers.
Speciﬁcally, the cover N i for ˆ W i is given by Lemma B.3 by plugging in (cid:107) Π i D
i ˆ X T
i − 1 (cid:107) F ≤ b i , and thus it sufﬁces to choose cover cardinality k :=
r 2 i b 2 i (cid:15) 2 i , whereby min
[Equation]
W
[Equation]
By this choice (and the cardinality estimate in Lemma B.3, the full cover N satisﬁes
[Equation]
To optimize the parameters ( (cid:15) 1 , . .
, (cid:15) L ) , the ﬁrst step is to show via induction that
[Equation]
The base case is simply (cid:107) X T 0 − ˆ X T (cid:107) =
(cid:107) X T − X T (cid:107) = 0 , thus consider layer i > 0 .
Using the inductive formula for ˆ X i and the cover guarantee on ˆ W i ,
[Equation]
F i ] (cid:13)(cid:13) 2 (cid:13)(cid:13)(cid:13)
X T i − 1 − ˆ X T i − 1 (cid:13)(cid:13)(cid:13) + (cid:13)(cid:13)(cid:13) [( W i − ˆ W i )
Π
i D
i ˆ X T
i − 1 |(cid:105)
( F i − F i )
ˆ X T
i − 1 ] (cid:13)(cid:13)(cid:13) ≤ s i (cid:88)
j ≤ i −(cid:15)
j i − 1 (cid:89) l
= j + s l + (cid:13)(cid:13)(cid:13) ( W i − ˆ W i )
Π
i D
i ˆ X T
i − 1 (cid:13)(cid:13)(cid:13) ≤ (cid:88)
j ≤ i −(cid:15)
j i (cid:89) l = j + s l + (cid:15) i ≤ (cid:88)
j ≤ i (cid:15)
j i (cid:89) l = j + s l .
To balance ( (cid:15) 1 , . .
, (cid:15) L ) , it sufﬁces to minimize a Lagrangian corresponding to the cover size subject to an error constraint, meaning
L ( (cid:126)(cid:15), λ ) =
L (cid:88) i = α i (cid:15)
i + λ  
L (cid:88) i = (cid:15) i β
i − (cid:15) 
 where α i :=
r 2 i b 2 i ln(2 m 2 ) , β i :=
L (cid:89) l = i + s l , whose unique critical point for (cid:126)(cid:15) > 0 implies the choice (cid:15) i :
= 1 Z (cid:18) 2 α
i β i (cid:19) 1 /where
Z := 1 (cid:15) (cid:88) i (
2 α i β 2 i ) 1 / 3 , whereby (cid:107) X T L − ˆ X T L (cid:107) ≤ (cid:15) automatically, and
[Equation]
as desired, with τ introduced for convenience in what is to come.
For the Rademacher complexity estimate, by a standard Dudley entropy integral (Shalev-Shwartz &
Ben-David, 2014), setting ˆ τ := max { τ, 1 / 3 } for convenience,
[Equation]
which is minimized at ζ = 3ˆ τ/ √ n , whereby
[Equation]
Combining Lemma 3.2 with Lemma 1.1 directly gives Theorem 1.2.
Proof of Theorem 1.2.
For each parameter to the model, e.g., b i , organize the corresponding parameter space into shells of radius j ∈ Z ≥ 1 , and associate a per-parameter prior weight q ( b i )
j = ( j ( j +1))
− 1 =
( b i ( b i +1)) 2 , whereby (cid:80) j q ( b i )
j = 1 .
Consequently, for each choice of parameters ( (cid:126)r,(cid:126)b,(cid:126)s i ) , the corresponding prior value products the priors for each parameter, and thus again sums to 1 (and let the overall prior ( p j ) j ≥ 1 be any enumeration of these products).
Moreover, for any prior value p ( (cid:126)r,(cid:126)b,(cid:126)s ) corresponding to the preceding product for some ( (cid:126)r,(cid:126)b,(cid:126)s i ) , by Lemma 3.2,
Rad ( G ( (cid:126)r [Equation]
= (cid:101) O (cid:32)
[Equation]
n (cid:88) i (cid:20) r
i b i
L (cid:89) l = i + s l (cid:21) 2 /
[Equation]
(cid:33) 3 /, and moreover note ln
[Equation]

[Equation]
i (cid:20) r
[Equation]
L (cid:89) l = i + s l (cid:21) 2 / 3 (cid:33) .
Lastly, note that ψ ( v, y ) := v y − max i (cid:54) =
y
v i is √ 2 -Lipschitz, and so can be safely appended to
G ( (cid:126)r,(cid:126)b,(cid:126)s i )) and then peeled when calculating Rademacher complexity (Shalev-Shwartz & BenDavid, 2014).
Combining these pieces, by Lemma 1.1, with probability at least 1 − δ , every mea-
[Table]
O (cid:32)(cid:88)
i (cid:20) r i b i
L (cid:89) l = i + s l (cid:21) 2 / 3 (cid:33) (cid:35) .
D P ROOF OF STABLE RANK BOUND , L EMMA 1.
The ﬁrst step is to establish the sparsiﬁcation lemma in Lemma 3.3, which in turn sparsiﬁes each matrix product, can not simply invoke Lemma B.2: pre-processing is necessary to control the elementwise magnitudes of the resulting matrix.
Lemma D.1.
Let matrices A ∈ R d × m and B ∈ R n × m be given, along with sampling budget k .
Then there exists a selection ( i 1 , . .
, i k ) of indices and a corresponding diagonal sampling matrix
M with at most k nonzero entries satisfying
M := k (cid:88)
[Equation]
where Z i j ≤ (cid:107) A (cid:107) F (cid:114) m k , and (cid:13)(cid:13) AB T − AMB T (cid:13)(cid:13) 2 ≤ 4 k (cid:107) A (cid:107) 2 (cid:107) B (cid:107) 2 .
Proof.
Let τ > 0 be a parameter to be optimized later, and deﬁne a subset of indices S :
= { i ∈ { 1 , . . .
, m } : (cid:107) A e
i (cid:107) ≥ τ } , with S c := { 1 , . . .
, m } \ S .
Let A τ denote the matrix obtained by zeroing out columns not in S , meaning
[Equation]
whereby
[Equation]
Applying Lemma B.2 to A τ B T gives
[Equation]
such that
[Equation]
where Z i j is speciﬁed by these equalities.
To simplify, note (cid:107)
A τ (cid:107) ≤ (cid:107) A (cid:107) , and A τ M = AM .
Combining the two inequalities,
[Equation]
To ﬁnish, setting τ := (cid:107) A (cid:107) / √ mk gives the bound, and ensures that the scaling term Z i j satisﬁes, for any i j ∈ S ,
[Equation]
With this tool in hand, the proof of Lemma 3.3 is as follows.
Proof of Lemma 3.3.
Let X j denote the network output after layer j , meaning
[Equation]
whereby
[Equation]
The proof will inductively choose sampling matrices ( M 1 , . .
, M L ) as in the statement and construct
[Equation]
where Π j denotes projection onto the Frobenius-norm ball of radius (cid:107)
X
[Equation]
i ≤ j (cid:107) W i (cid:107) 2 (whereby Π j X j = X j ), satisfying
[Equation]
which gives the desired bound after plugging in j = L .
Proceeding with the inductive construction, the base case is direct since ˆ X 0 =
X
= X [-
-]0 and (cid:13)(cid:13)(cid:13)
[Equation]
0 (cid:13)(cid:13)(cid:13) F = 0 , thus consider some j > 0 .
Applying Lemma D.1 to the matrix multiplication
W j ˆ X j − 1 with k j samples, there exists a multiset of S j coordinates and a corresponding sampling matrix M j , as speciﬁed in the statement, satisfying
[Equation]
Using the choice ˆ X T j :=
Π j σ j
W j M j ˆ X T j −
[Equation]
[Equation]
as desired.
To prove Lemma 1.3 via Lemma 3.3, the ﬁrst step is a quick tool to cover matrices element-wise.
Lemma D.2.
Let A denote matrices with at most k 2 nonzero rows and k 1 nonzero columns, entries bounded in absolute value by b , and total number of rows and columns each at most m .
Then there exists a cover set M ⊆
A satisfying |M| ≤ m k 1 + k(cid:32) 2 b √ k 1 k(cid:15) (cid:33) k 1 k, and sup A ∈A min ˆ A ∈M (cid:107) A − ˆ A (cid:107) F ≤ (cid:15).
Proof.
Consider some ﬁxed set of k 2 nonzero rows and k 1 nonzero columns, and let M 0 denote the covering set obtained by gridding the k 1 · k 2 entries at scale
[Equation]
, whereby
[Equation]
For any A ∈ A with these speciﬁc nonzero rows and columns, the ˆ A ∈ M 0 obtained by rounding each nonzero entry of A to the nearest grid element gives
[Equation]
The ﬁnal cover M is now obtained by unioning copies of M 0 for all (cid:0) mk 1 (cid:1)(cid:0) mk 2 (cid:1) ≤ m k 1 + k 2 possible submatrices of size k 2 × k 1 .
The proof of Lemma 1.3 now carefully combines the preceding pieces.
Proof of Lemma 1.3.
The proof proceeds in three steps, as follows.
A covering number is estimate for sparsiﬁed networks, as output by Lemma 3.3.
A covering number for general networks is computed by balancing the error terms from Lemma 3.3 and its cover computed here.
This covering number is plugged into a Dudley integral to obtain the desired Rademacher bound.
Proceeding with this plan, let ( ˆ X T 0 , . . .
, ˆ X T L ) be the layer outputs (and network input) exactly as provided by Lemma 3.3.
Additionally, deﬁne diagonal matrices
D j := (cid:80) l !
∈
S
j +1 e l e T l (with D L =
I , where the “!” denotes unique inclusion; these matrices capture the effect of the subsequent sparsiﬁcation, and can be safely inserted after each W j without affecting ˆ X T j , meaning
[Equation]
Let per-layer cover precisions ( (cid:15) 1 , . . .
, (cid:15) L ) be given, which will be optimized away later.
This proof will inductively construct
[Equation]
where ˜ W j is a cover element for D j W j M j , and inductively satisfying
[Equation]
To construct the per-layer cover elements ˜ W j , ﬁrst note by the form of M j (and the scaling Z i provided by Lemma 3.3) that
[Equation]
Consequently, by Lemma D.2, there exists a cover C j of matrices of the form D j W j M j satisfying |C j | ≤ m k j
[Equation]
k j [Equation] [Equation] k j
[Equation]
j
[Equation]
k j k j [Equation] W [Equation]
k
[Equation]
(cid:15) j
[Equation]
k j k j −, and the closest cover element ˜ W j C j to D j W j M j satisﬁes [Equation] D j W j M j
[Equation]
W
[Equation]
Proceeding with the induction, the base case has
[Equation]
X
[Equation]
=
[Equation]
X T − X T (cid:107) = 0 , thus consider j > 0 .
The ﬁrst step is to estimate the spectral norm of D j W j M j , which can be coarsely upper bounded via
[Equation]
By the form of ˆ X j and ˜ X j ,
[Equation]
which establishes the desired bound on the error
[Equation]
The next step is to optimize k j .
Let (cid:15) > 0 be arbitrary
[Equation]
and set (cid:15) − 1 j := (cid:15)
[Equation]
m
[Equation]
whereby
[Equation]
The overall network cover N is the product of the covers for all layers
[Equation]
and thus has cardinality satisfying
[Equation]


[Equation]
To choose ( k 1 , . .
. , k L ) , letting X T L denote the output of the original unsparsiﬁed network, note ﬁrstly that the full error bound satisﬁes
[Equation]
, where α i := (cid:107) X (cid:107) F  (cid:89)
i (cid:107) W i (cid:107)  (cid:112) sr
( W i ) .
To choose k i , the approach here is to minimize a Lagrangian corresponding to the cover cardinality, subject to the total cover error being (cid:15) .
Simplifying the previous expressions and noting 2 k j k j − 1 ≤ k 2 j + k 2 j − 1 , whereby the dominant term in ln |N| is (cid:80) j k 2 j , consider Lagrangian
[Equation]
which has critical points when each k i satisﬁes
[Equation]
thus k i :=
α 2 / 5 i /Z with Z := (cid:15) 2 / (2 (cid:80)
j α 4 / 5 j ) 2 .
As a sanity check (since it was baked into the
Lagrangian)
, plugging this into the cover error indeed gives
[Equation]
To upper bound the cover cardinality, ﬁrst note that
[Equation]
whereby
[Equation]
ln
[Equation]
(cid:105) = β (cid:15)where β = (cid:101) O (cid:18) (cid:107) X (cid:107) 4 F (cid:104)(cid:89)
[Equation]
(cid:107)
[Equation]
(cid:104)(cid:88)
[Equation]
W [Equation] (cid:105) 5 (cid:104)(cid:88)
i ln (cid:107) W
[Equation]
(cid:107)
[Equation]
(cid:105)(cid:19) .
The ﬁnal step is to apply a Dudley entropy integral (Shalev-Shwartz & Ben-David, 2014), which gives
[Equation]
Dropping the negative term gives an expression of the form aζ + b/ζ , which is convex in ζ > 0 and has critical point at ζ
2 = b/a , which after plugging back in gives an upper bound 2 √ ab , meaning
[Equation]
Dividing by n and expanding the deﬁnition of β gives the ﬁnal Rademacher complexity bound.
[Figure]
E R EVIEW RESPONSE MATERIAL E. C OMPARISON OF BOUNDS
In response primarily to comments by AnonReviewer1 and AnonReviewer4, but relevant to all reviewers, we have included Figure 4.
Here is a detailed description:
• The ﬁgures follows a single run of our distillation method as detailed at the start of Section 2.
That is, a sequence of distilled functions is produced by gradually increasing the regularization parameter λ .
The horizontal axis corresponds to this regularization strength λ , and the vertical axis to various bounds and other values.
• The architecture is fully-connected with three ReLU layers of width 2048 , and the data is mnist .
It is trained to perfect classiﬁcation accuracy before distillation.
Unlike most plots in the paper, there are no skip connections, which allows us to invoke prior bounds from the literature.
• The curves with circle markers denote the training error, testing error, distillation error, and 10% margin quantile of the distilled function g .
Note that for regularization strength 10 − 3 , the test error is still small, but the bounds have dropped by an order of magnitude.
• The curves labeled by triangles are bounds.
– The VC bound stays constant, since the architecture does not change (Bartlett et al., 2017b).
– The curve for Lemma 3.2 is close to the curve for (Bartlett et al., 2017a), which makes sense since their proofs are similar.
It also makes sense that both curves decrease monotonically, since the regularization scheme uses (cid:107) · (cid:107) 2 , 1 .
– The curve for Lemma 1.3 starts the best, but gradually all other curves dominate it; this makes some sense, since the proof itself involves a compression-like step.
– The Frobenius product bound from (Golowich et al., 2018) looks promising in the middle phase, but becomes worse.
It is possible it would remain the best if the regularization scheme replaced the (2 , 1) norm with a Frobenius norm.
We focused on the (2 , 1) norm, however, since it is used by our Lemma 3.2, which handles skip connections and convolution layers.
The most similar plot in the literature is Figure 4 in (Arora et al., 2018).
The main points of contrast are that the curves here include dimension factors for each bound which were dropped there, and of course that the bounds here also bound the original network.
E. A SUFFICIENT CONDITION FOR DISTILLATION
This section is in response to a comment by AnonReviewer4, and aims to provide a sufﬁcient condition for distillation.
Namely, the sufﬁcient condition is that the weight matrices can be written
W i + E i , where W i is deterministic and E i is random; the claim is that we can “distill away” E i .
This construction is meant more as an illustration than an explanation; it is meant to capture our intuition that distillation can work when there are many useless, canceling weights.
Randomness is one way to model such weights.
Proposition E.1.
Let δ > 0 be given, and suppose the data matrix X ∈ R n × d is normalized so that (cid:107) X (cid:107) F ≤ √ n , and the labels y i ∈ {± 1 } are binary for simplicity.
Consider a ReLU network f ( x ) =
A L σ ( A L − 1 · · · σ ( A 1 x ) · · · ) where the i th weight matrix can be written A i :=
W i + E i , where W i satisﬁes (cid:107) W
i
(cid:107) 2 ≥ 1 , and E i is random with iid N (0 , 1 /m ) coordinates and independent of ( W j ) j<i , and the width parameter m satisﬁes
[Equation]
Then, with probability at least 1 − 3 Lδ over the draw of ( E L , . . .
, E 1 ) , the following statements hold.
(Original network has bad complexity.)
It holds that (cid:107) E 1 (cid:107) F = Θ( √ d ) , and (cid:107) E L (cid:107) F = Θ(1) , and (cid:107) A i
[Equation] F = Θ [Equation] E i
[Equation]
F ) =
Θ( √ m ) for i ∈ { 2 , . .
, L − 1 } , and the Rademacher complexity upper bound bound from (Golowich et al., 2018, Theorem 1) for ReLU networks with the same weight matrix norms as f is √
[Equation]
(Distilled network has low complexity.)
Deﬁne the distilled network as
[Equation]
The Rademacher complexity bound from (Golowich et al., 2018, Theorem 1) for ReLU networks with the same weight matrix norms as g is
[Equation]
(Distilled
and original
networks are close.)
Φ
[Equation]
n ( [Equation] [Equation] The parameter m corresponds to the width of the original network, and is intended to be a free parameter that can be made arbitrarily large.
The message of Proposition E.1 is that we can make f have arbitrarily bad complexity, while remaining close to g which has low complexity.
From another perspective, Proposition E.1 captures one setting where distillation can vastly reduce the bound from (Golowich et al., 2018, Theorem 1), as depicted in Figure 4.
Proof of Proposition E.1.
The ﬁrst step is to bound E (cid:107) E i (cid:107) for each matrix.
By Jensen’s inequality,
[Equation]
where m 1 = d , m L = 1 , and m i = m otherwise.
For the lower bound, by the general
Taylor lower bound
[Equation]
setting x = (cid:107) E
i (cid:107) 2 and r := E (cid:107) E
i (cid:107) 2 = m i and noting E (cid:107) E
i (cid:107) 4 = E ( (cid:80) j,k ( E i ) 2 j,k ) 2 gives
E ( x − r ) 2 = (cid:88)
[Equation]
whereby
[Equation]
Moreover, since the mapping A (cid:55)→ (cid:107) A (cid:107) F is 1 -Lipschitz, then standard Gaussian concentration (Wainwright, 2019, Theorem 2.26) implies that with probability at least 1 − 2 δ ,
[Equation]
Combining this with the estimates on E (cid:107)
A i (cid:107) for i ∈ { 2 , . . .
, L } [Equation] note by the assumptions on m and the preceding derivations that
[Equation]
and in particular
[Equation]
and
[Equation]
and the lower bounds on m gives the desired bounds on (cid:107) E i (cid:107) (after unioning away the combined 2 Lδ failure probability mass)
The complexity estimate is now direct from (Golowich et al., 2018, Theorem 1).
This part is direct from the deﬁnition of g and (Golowich et al., 2018, Theorem 1).
For convenience throughout the proof, deﬁne
[Equation]
For convenience deﬁne a fake identity activation σ 0 ( z ) =
z , which is still 1 -homogeneous and 1 -Lipschitz, and recursively deﬁne
[Equation]
The ﬁrst part of the proof will show, inductively, with probability at least 1 − iδ , that
[Equation]
which in turn will imply
[Equation]
We’ll spend the bulk of the proof establishing this claim; before that, let’s see how to complete the proof from there.
By (Bartlett et al., 2017a, Lemma A.3), the margin mapping (ˆ y, y ) (cid:55)→ ˆ y y − max i (cid:54) =
y ˆ
y i is 2 -Lipschitz, and a similar calculation to Lemma A.1 implies that φ 1 is (1 / 4) -Lipschitz, therefore
[Equation]
Returning to the induction, the base case is immediate since Z 0 =
X T =
U 0 .
For the inductive step, ﬁrst note that the mapping
[Equation]
is (cid:107)
Z i − 1 (cid:107) F -Lipschitz, since, for any matrices A, B ,
[Equation]
which can be repeated with A and B swapped, and implies
[Equation]
As such, with probability 1 − δ , by the fact that σ i − 1 ( Z i − 1 ) and E i are independent and standard Gaussian concentration
(Wainwright, 2019, Theorem 2.26),
[Equation]
where whereby the 1-homogeneity and 1-Lipschitz properties of σ i − 1 allow further simpliﬁcation via
[Equation]
and similarly
[Equation]
Thanks to this concentration inequality
[Equation]
and unioning it with the ( i − 1) δ failure probability of the inductive hypothesis, then with probability 1 − iδ ,
[Equation]
i i − 1 (cid:88)
j = (cid:81)
ik =1 s k s j   ≤ (cid:15) (cid:107) X (cid:107) F (1 + (cid:15) )
i i (cid:88)
j = (cid:81)
ik =1 s k s j , which completes the induction.
For the ﬁnal simpliﬁed bound, By the conditions on m , then (1 + (cid:15) )
L ≤ (1 + 1 /L )
L ≤ 3 , and additionally by further using the conditions on m , and since min i (cid:107) W i (cid:107) ≥ 1 and (cid:107) X (cid:107) F ≤ √ n , (cid:15) (cid:107) X (cid:107) F (cid:15) (1+ (cid:15) )
L i (cid:88)
j = (cid:81)
{+Lk =
 s k s j ≤ 3 L(cid:15) (cid:107) X (cid:107) F L (cid:89) k+}
= s k
= 3 L (cid:112) 2 ln(1 /δ )
(cid:107) X (cid:107) F (cid:81)
Lk
{+=
+}
s k √ m ≤ √ n m 1 / 4 , as desired.
E. M ARGIN VERSION OF L EMMA 1.
To clarify a point by AnonReviewer3 and to provide a useful point of comparison to Lemma 1.1, we may replace the distillation loss with a margin loss and follow nearly exactly the same proof to give a margin-based distillation bound.
To develop this version, deﬁne the standard ramp (margin) loss as (cid:96) γ ( z ) :
=  
z < 0 , 1 − z / γ z ∈ [0 , γ ] , z > γ.
In the binary case, the corresponding loss is (cid:96) γ ( yf ( x )) ; for the multiclass case, overload notation so that
[Equation]
(This multiclass version appears in (Bartlett et al., 2017a), and still upper bounds misclassiﬁcation error.)
The corresponding distillation bound is as follows.
Proposition E.2.
Let temperature (margin) parameter γ > 0 be given, along with any sets of multiclass predictors ( G j ) j ≥ 1 with G j (cid:51) g : R d → R k , along with prior weights ( p j ) j ≥ 1 on these sets, where p j ≥ 0 and (cid:80) j p j = 1 .
Then with probability at least 1 − δ over an iid draw of data (( x i , y i ))
ni =1 , every measurable function f : R d → R k satisﬁes
[Equation]
Proof sketch of Proposition E.2.
Since (cid:96) γ is bounded, the proof goes through basically identically to the proof of Lemma 1.1 via Lemma 3.1.
One key difference is that whereas
[Equation]
we now have
[Equation]
which removes a factor of 2 from most terms in right hand side of the bound as compared with
Lemma 1.1.