Original: For any X = ( x 1 , . . .
Revised: For any X = ( x 1 , . . .

Original: , x n ) ∈ X n = R n × h and ϵ > 0 , the covering number
Revised: , x n ) ∈ X n = R n × h and ϵ > 0 , the covering number

Original: N ( F , ϵ, ∥·∥ X ) satisfies
Revised: N ( F , ϵ, ∥·∥ X ) satisfies

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We rewrite the coordinate-wise definition of the 1D convolutional operation from Eq.
Revised: We rewrite the coordinate-wise definition of the 1D convolutional operation from Eq.

Original: (48) with unit stride as
Revised: (48) with unit stride as

Original: [Equation]
Revised: [Equation]

Original: Thus, convolution is a linear map parametrized by a matrix M ∈ R h × h with entries
Revised: Thus, convolution is a linear map parametrized by a matrix M ∈ R h × h with entries

Original: [Equation]
Revised: [Equation]

Original: In particular, we can write M = (cid:80) ( k − 1) /
Revised: In particular, we can write M = (cid:80) ( k − 1) /

Original: 2 α = − ( k − 1) /
Revised: 2 α = − ( k − 1) /

Original: 2 K α M
Revised: 2 K α M

Original: ( α ) with M ( α ) ij = 1 j − i
Revised: ( α ) with M ( α ) ij = 1 j − i

Original: = α and note that for every x ∈ R h and every α , we have
Revised: = α and note that for every x ∈ R h and every α , we have

Original: [Equation]
Revised: [Equation]

Original: For example, if h = 5 and k = 3 , we have
Revised: For example, if h = 5 and k = 3 , we have

Original: M =  
Revised: M =  

Original: K K
Revised: K K

Original: K −
Revised: K −

Original: K K
Revised: K K

Original: K −
Revised: K −

Original: K K
Revised: K K

Original: K −
Revised: K −

Original: K K 

Revised: K K

Original: K −
Revised: K −

Original: K  ,
Revised: K  ,

Original: M ( − 1) =   0 0 0 01 0 0 00 1 0 00 0 1 00 0 0 1  , M (0) =
Revised: M ( − 1) =   0 0 0 01 0 0 00 1 0 00 0 1 00 0 0 1  , M (0) =

Original:   1 0 0 00 1 0 00 0 1 00 0 0 10 0 0 0  , M (1) =   0 1 0 00 0 1 00 0 0 10 0 0 00 0 0 0  .
Revised:   1 0 0 00 1 0 00 0 1 00 0 0 10 0 0 0  , M (1) =   0 1 0 00 0 1 00 0 0 10 0 0 00 0 0 0  .

Original: In summary, we have
Revised: In summary, we have

Original: [Equation]
Revised: [Equation]

Original: By assumption, we have (cid:80) α (cid:12) (cid:12) K α b (cid:12) (cid:12) =
Revised: By assumption, we have (cid:80) α (cid:12) (cid:12) K α b (cid:12) (cid:12) =

Original: [Equation]
Revised: [Equation]

Original: ∥
Revised: ∥

Original: [Equation]
Revised: [Equation]

Original: b ≤ 1 and so we can instantiate Maurey’s sparsification lemma (Lemma C.4) on the Hilbert space ( F| X , ∥·∥ X ) for { V 1 , . . .
Revised: b ≤ 1 and so we can instantiate Maurey’s sparsification lemma (Lemma C.4) on the Hilbert space ( F| X , ∥·∥ X ) for { V 1 , . . .

Original: V 2 d } = (cid:8) x
Revised: V 2 d } = (cid:8) x

Original: (cid:55)→ ± bM ( α ) x | α = 1 , . . .
Revised: (cid:55)→ ± bM ( α ) x | α = 1 , . . .

Original: , d (cid:9) ⊂ F .
Revised: , d (cid:9) ⊂ F .

Original: As a consequence, for any convolutional kernel K ∈ R k and any m ∈ N , there exist ( m 1 , . . .
Revised: As a consequence, for any convolutional kernel K ∈ R k and any m ∈ N , there exist ( m 1 , . . .

Original: , m 2 d ) with (cid:80) 2 di =

Revised: , m 2 d ) with (cid:80) 2 di =

Original: m i = m such that (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: m i = m such that (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: Thus, for fixed ϵ > 0 , if we choose m ∈ N such that 1 m max i
Revised: Thus, for fixed ϵ > 0 , if we choose m ∈ N such that 1 m max i

Original: ∥ V i ∥ 2 X ≤ ϵ 2 , then the solutions (in m i ), of (cid:80) 2 di =1 m i = m , define an ϵ -cover
Revised: ∥ V i ∥ 2 X ≤ ϵ 2 , then the solutions (in m i ), of (cid:80) 2 di =1 m i = m , define an ϵ -cover

Original: [Equation]
Revised: [Equation]

Original: As the number of non-negative 2 d -tuples that add up to m is equal to
Revised: As the number of non-negative 2 d -tuples that add up to m is equal to

Original: [Equation]
Revised: [Equation]

Original: this means that F has an ϵ -cover of cardinality at most N ( m, d ) , and so N ( F , ϵ, ∥·∥ X ) ≤ N ( m, d ) .
Revised: this means that F has an ϵ -cover of cardinality at most N ( m, d ) , and so N ( F , ϵ, ∥·∥ X ) ≤ N ( m, d ) .

Original: Since for all i ∈ { 1 , . . .
Revised: Since for all i ∈ { 1 , . . .

Original: , 2 d } , the norms ∥ V i ∥
Revised: , 2 d } , the norms ∥ V i ∥

Original: 2 X satisfy
Revised: 2 X satisfy

Original: [Equation]
Revised: [Equation]

Original: we can choose N ∋ m = (cid:108) b 2 ∥ X ∥ 2 ϵ
Revised: we can choose N ∋ m = (cid:108) b 2 ∥ X ∥ 2 ϵ

Original: [Equation]
Revised: [Equation]

Original: The theorem then follows from two particular bounds on N ( m, d ) , see Lemma C.24.
Revised: The theorem then follows from two particular bounds on N ( m, d ) , see Lemma C.24.

Original: These are
Revised: These are

Original: [Equation]
Revised: [Equation]

Original: which implies Eq.
Revised: which implies Eq.

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: which implies Eq.
Revised: which implies Eq.

Original: Remark C.6.
Revised: Remark C.6.

Original: The definition of the convolution operation in Eq.
Revised: The definition of the convolution operation in Eq.

Original: (48) corresponds to convolutional layers with zero-padding, such that the dimensionality h of the data remains unchanged (i.e., inputsize preserving).
Revised: (48) corresponds to convolutional layers with zero-padding, such that the dimensionality h of the data remains unchanged (i.e., inputsize preserving).

Original: The covering bound equally holds for other types of padding, corresponding to other matrices M ( α ) , as long as
Revised: The covering bound equally holds for other types of padding, corresponding to other matrices M ( α ) , as long as

Original: [Equation]
Revised: [Equation]

Original: for all x .
Revised: for all x .

Original: In particular, it holds for convolutional layers with circular padding, where the matrices
Revised: In particular, it holds for convolutional layers with circular padding, where the matrices

Original: M ( α ) become permutation matrices .
Revised: M ( α ) become permutation matrices .

Original: Next, we study the general case of 2D multi-channel convolutions with strides.
Revised: Next, we study the general case of 2D multi-channel convolutions with strides.

Original: To that end, let ϕ K be the map determined by a weight tensor K ∈ R c out × c in ×
Revised: To that end, let ϕ K be the map determined by a weight tensor K ∈ R c out × c in ×

Original: k h × k w , where c out , c in denote the number of output and input channels, resp., and ( k h , k w ) is the spatial extension of the kernel.
Revised: k h × k w , where c out , c in denote the number of output and input channels, resp., and ( k h , k w ) is the spatial extension of the kernel.

Original: For input images x ∈ R c in × h × w , convolution ϕ K, ( s h ,s w ) : R
Revised: For input images x ∈ R c in × h × w , convolution ϕ K, ( s h ,s w ) : R

Original: [Equation]
Revised: [Equation]

Original: ×
Revised: ×

Original: [Equation]
Revised: [Equation]

Original: × w → R c in ×⌈ h/s h ⌉×⌈ w/s w ⌉ with strides ( s h , s w ) is defined coordinate-wise as
Revised: × w → R c in ×⌈ h/s h ⌉×⌈ w/s w ⌉ with strides ( s h , s w ) is defined coordinate-wise as

Original: [Equation]
Revised: [Equation]

Original: Theorem C.7 (Single-layer covering number bound – General case) .
Revised: Theorem C.7 (Single-layer covering number bound – General case) .

Original: Let b > 0 .
Revised: Let b > 0 .

Original: Define the class of ( s h , s w ) -strided 2D convolutions parametrized by tensors K ∈ R c out × c in × k h × k w with W =
Revised: Define the class of ( s h , s w ) -strided 2D convolutions parametrized by tensors K ∈ R c out × c in × k h × k w with W =

Original: c out c in k h k w parameters and (2,1) group norm ∥ K ∥ 2 , 1 ≤ b as
Revised: c out c in k h k w parameters and (2,1) group norm ∥ K ∥ 2 , 1 ≤ b as

Original: [Equation]
Revised: [Equation]

Original: Then, for any X = ( x 1 , . . .
Revised: Then, for any X = ( x 1 , . . .

Original: , x n ) ∈ R n × c in × h × w and ϵ > 0 , the covering number N ( F , ϵ, ∥·∥ X ) satisfies
Revised: , x n ) ∈ R n × c in × h × w and ϵ > 0 , the covering number N ( F , ϵ, ∥·∥ X ) satisfies

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Remark C.8.
Revised: Remark C.8.

Original: Recall that convolution with kernel size 1 and input size 1 is a linear map on the input channels, determined by the matrix M K = K ·· 11 .
Revised: Recall that convolution with kernel size 1 and input size 1 is a linear map on the input channels, determined by the matrix M K = K ·· 11 .

Original: In this situation, the convolutional layer reduces to a fully-connected layer and our first bound reduces to [3, Lemma 3.2].
Revised: In this situation, the convolutional layer reduces to a fully-connected layer and our first bound reduces to [3, Lemma 3.2].

Original: Proof.
Revised: Proof.

Original: The proof is quite similar to the one of the special case in Theorem C.5.
Revised: The proof is quite similar to the one of the special case in Theorem C.5.

Original: Recall, that the convolution operation is defined coordinate-wise in Eq.
Revised: Recall, that the convolution operation is defined coordinate-wise in Eq.

Original: Using identities of the formt ∈{ 1 ,...,n } = (cid:80) ni
Revised: Using identities of the formt ∈{ 1 ,...,n } = (cid:80) ni

Original: =1 1 i = t , we write [ ϕ
Revised: =1 1 i = t , we write [ ϕ

Original: [Equation]
Revised: [Equation]

Original: )] σµν = c
Revised: )] σµν = c

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: α
Revised: α

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original:   and condense this into [ ϕ
Revised:   and condense this into [ ϕ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: h ,s w )
Revised: h ,s w )

Original: [Equation]
Revised: [Equation]

Original: x )]
Revised: x )]

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Thus, the convolution is a multilinear map R c in × h × w → R c in ×⌈ h/s h ⌉×⌈ w/s w ⌉ parametrized by
Revised: Thus, the convolution is a multilinear map R c in × h × w → R c in ×⌈ h/s h ⌉×⌈ w/s w ⌉ parametrized by

Original: [Equation]
Revised: [Equation]

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: K prij |(cid:33) 1 / 2 (cid:32) c in (cid:88)
Revised: K prij |(cid:33) 1 / 2 (cid:32) c in (cid:88)

Original: r = ∥ X · r ·· ∥(cid:33) 1 /(cid:124) (cid:123)(cid:122) (cid:125) =
Revised: r = ∥ X · r ·· ∥(cid:33) 1 /(cid:124) (cid:123)(cid:122) (cid:125) =

Original: ∥
Revised: ∥

Original: X ∥ =
Revised: X ∥ =

Original: [Equation]
Revised: [Equation]

Original: c out (cid:88)
Revised: c out (cid:88)

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: ⌊−
Revised: ⌊−

Original: [Equation]
Revised: [Equation]

Original: kw
Revised: kw

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: (cid:32) c
Revised: (cid:32) c

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: (cid:33)
Revised: (cid:33)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: b (by assumption)
Revised: b (by assumption)

Original: [Equation]
Revised: [Equation]

Original: 1 , we can instantiate Maurey’s sparsification lemma (Lemma C.4) on the Hilbert space ( F| X , ∥·∥ X ) for { V 1 ,
Revised: 1 , we can instantiate Maurey’s sparsification lemma (Lemma C.4) on the Hilbert space ( F| X , ∥·∥ X ) for { V 1 ,

Original: [Equation]
Revised: [Equation]

Original: , V
Revised: , V

Original: [Equation]
Revised: [Equation]

Original: } =
Revised: } =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: As a consequence, for any convolutional kernel K ∈ R c out × c in × k h × k w and any m ∈ N , there exist ( m 1 , . . .
Revised: As a consequence, for any convolutional kernel K ∈ R c out × c in × k h × k w and any m ∈ N , there exist ( m 1 , . . .

Original: , m 2 W ) with (cid:80) 2 Wi =
Revised: , m 2 W ) with (cid:80) 2 Wi =

Original: m i = m such that
Revised: m i = m such that

Original: [Equation]
Revised: [Equation]

Original: Thus, for fixed ϵ > 0 , if we choose m ∈ N such that 1 m max i
Revised: Thus, for fixed ϵ > 0 , if we choose m ∈ N such that 1 m max i

Original: ∥ V i ∥ 2 X ≤ ϵ 2 , then the solutions in m i of (cid:80) 

Revised: ∥ V i ∥ 2 X ≤ ϵ 2 , then the solutions in m i of (cid:80)

Original: Wi =

Revised: Wi =

Original: m i = m , define an ϵ -cover
Revised: m i = m , define an ϵ -cover

Original: [Equation]
Revised: [Equation]

Original: As the number of non-negative 2 W -tuples which add up to m , denoted as N ( m, W ) , is equal to
Revised: As the number of non-negative 2 W -tuples which add up to m , denoted as N ( m, W ) , is equal to

Original: [Equation]
Revised: [Equation]

Original: this means that F has an ϵ -cover of cardinality at most N ( m, W ) ; thus, N ( F , ϵ, ∥·∥ X ) ≤ N ( m, W ) .
Revised: this means that F has an ϵ -cover of cardinality at most N ( m, W ) ; thus, N ( F , ϵ, ∥·∥ X ) ≤ N ( m, W ) .

Original: In order to compute the norms ∥ V i ∥ 2 X , we use that for all ( p, r, i, j ) ∈ [ c out ] ×
Revised: In order to compute the norms ∥ V i ∥ 2 X , we use that for all ( p, r, i, j ) ∈ [ c out ] ×

Original: [ c in ] ×
Revised: [ c in ] ×

Original: [ k h ] ×
Revised: [ k h ] ×

Original: [ k w ] and for all x ∈ R c in × h × w , it holds that
Revised: [ k w ] and for all x ∈ R c in × h × w , it holds that

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: M
Revised: M

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original: [Table]
Revised: [Table]

Original: ≤
Revised: ≤

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Thus, for any t ∈ { 1 , . . .
Revised: Thus, for any t ∈ { 1 , . . .

Original: , 2 W } ,
Revised: , 2 W } ,

Original: [Equation]
Revised: [Equation]

Original: and we can choose N ∋ m = (cid:108) b 2 ∥ X ∥ 2 ϵ(cid:109) to get an ϵ -cover of F via Eq.
Revised: and we can choose N ∋ m = (cid:108) b 2 ∥ X ∥ 2 ϵ(cid:109) to get an ϵ -cover of F via Eq.

Original: The theorem then follows from two particular bounds on N ( m, W )
Revised: The theorem then follows from two particular bounds on N ( m, W )

Original: =
Revised: =

Original: (cid:0)
Revised: (cid:0)

Original: m +2 W − 1 2 W − 1 (cid:1) .
Revised: m +2 W − 1 2 W − 1 (cid:1) .

Original: These are (cid:18) m
Revised: These are (cid:18) m

Original: + 2 W −
Revised: + 2 W −

Original: [Equation]
Revised: [Equation]

Original: d −
Revised: d −

Original: [Equation]
Revised: [Equation]

Original: ≤ (2 W ) m , which implies Eq.
Revised: ≤ (2 W ) m , which implies Eq.

Original: (54), and (cid:18) m
Revised: (54), and (cid:18) m

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: 2 W
Revised: 2 W

Original: [Equation]
Revised: [Equation]

Original: (cid:19) ≤ (1 + m ) 2 W − 1 , which implies Eq.
Revised: (cid:19) ≤ (1 + m ) 2 W − 1 , which implies Eq.

Original: (55); see Lemma C.24 for details.
Revised: (55); see Lemma C.24 for details.

Original: Remark C.9.
Revised: Remark C.9.

Original: In the proof, we bound
Revised: In the proof, we bound

Original: [Equation]
Revised: [Equation]

Original: Under additional assumptions on the data X this result might be improved as, on average, one expects
Revised: Under additional assumptions on the data X this result might be improved as, on average, one expects

Original: [Equation]
Revised: [Equation]

Original: which would reduce the ∥ X ∥ 2 b 2 ϵterms in the bounds by the factor 1 / ( s h s w ) .
Revised: which would reduce the ∥ X ∥ 2 b 2 ϵterms in the bounds by the factor 1 / ( s h s w ) .

Original: C.3 Whole-network covering number bounds (general form)
Revised: C.3 Whole-network covering number bounds (general form)

Original: In order to prove covering number bounds for residual networks, we utilize the following basic observation: a residual network is a composition of residual blocks and each residual block corresponds to addition of two (compositions of) functions on the same input (one of them is typically the identity function).
Revised: In order to prove covering number bounds for residual networks, we utilize the following basic observation: a residual network is a composition of residual blocks and each residual block corresponds to addition of two (compositions of) functions on the same input (one of them is typically the identity function).

Original: Thus, if we know the covering numbers of compositions and additions, we can derive whole-network covering number bounds in an inductive way.
Revised: Thus, if we know the covering numbers of compositions and additions, we can derive whole-network covering number bounds in an inductive way.

Original: Importantly, the derived covering bounds hold for a broad class of network architectures, including the special cases of non-residual and residual networks.
Revised: Importantly, the derived covering bounds hold for a broad class of network architectures, including the special cases of non-residual and residual networks.

Original: C.3.1 Covering number bounds for compositions and summations
Revised: C.3.1 Covering number bounds for compositions and summations

Original: Given normed spaces ( X i , ∥·∥ X i ) and function spaces F i and G i of functions
Revised: Given normed spaces ( X i , ∥·∥ X i ) and function spaces F i and G i of functions

Original: X i → X i +1 , we present covering number bounds for the following derived function spaces:
Revised: X i → X i +1 , we present covering number bounds for the following derived function spaces:

Original: [Equation]
Revised: [Equation]

Original: Lemma C.10 (Compositions) .
Revised: Lemma C.10 (Compositions) .

Original: For i ∈ { 1 , 2 , 3 } , let ( X i , ∥·∥ X i ) be normed spaces and let F i be classes of functions
Revised: For i ∈ { 1 , 2 , 3 } , let ( X i , ∥·∥ X i ) be normed spaces and let F i be classes of functions

Original: X i → X i +1 with Lip( F i ) <
Revised: X i → X i +1 with Lip( F i ) <

Original: ∞ .
Revised: ∞ .

Original: Then, for any ϵ 1 , ϵ 2 > 0 and any X = ( x 1 , . . .
Revised: Then, for any ϵ 1 , ϵ 2 > 0 and any X = ( x 1 , . . .

Original: , x n ) ∈ X n 1 , the covering number of the class Comp( F 1 , F 2 ) is bounded by
Revised: , x n ) ∈ X n 1 , the covering number of the class Comp( F 1 , F 2 ) is bounded by

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: Comp
Revised: Comp

Original: [Equation]
Revised: [Equation]

Original: F 1 , F 2 ) , Lip
Revised: F 1 , F 2 ) , Lip

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: If F 2 = { f 2 } is a singleton, then
Revised: If F 2 = { f 2 } is a singleton, then

Original: [Equation]
Revised: [Equation]

Original: Remark C.11.
Revised: Remark C.11.

Original: There is an analogous result which holds for external covering numbers, i.e.,
Revised: There is an analogous result which holds for external covering numbers, i.e.,

Original: N
Revised: N

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Notably, in this case, the supremum is taken over all f : X 1 → X 2 .
Revised: Notably, in this case, the supremum is taken over all f : X 1 → X 2 .

Original: However, this form is unusable for deriving the whole-network covering number bounds in Section C.4 as we want to handle the supremum via an assumption on the Lipschitz constant of the layer.
Revised: However, this form is unusable for deriving the whole-network covering number bounds in Section C.4 as we want to handle the supremum via an assumption on the Lipschitz constant of the layer.

Original: Proof.
Revised: Proof.

Original: Fix ϵ 1 , ϵ 2 > 0 .
Revised: Fix ϵ 1 , ϵ 2 > 0 .

Original: Let
Revised: Let

Original: U F 1 ⊂ F 1 be a minimal ϵ 1 -cover of ( F 1 , ∥·∥ X ) , i.e., card( U F 1 ) =
Revised: U F 1 ⊂ F 1 be a minimal ϵ 1 -cover of ( F 1 , ∥·∥ X ) , i.e., card( U F 1 ) =

Original: N ( F 1 , ϵ 1 , ∥·∥ X ) .
Revised: N ( F 1 , ϵ 1 , ∥·∥ X ) .

Original: For any covering element v ∈ U F 1 , let U F 2 ( v ) ⊂
Revised: For any covering element v ∈ U F 1 , let U F 2 ( v ) ⊂

Original: F 2 be a minimal ϵ 2 -cover of ( F 2 , ∥·∥ v ( X ) ) , i.e., card( U F 2 ( v ))
Revised: F 2 be a minimal ϵ 2 -cover of ( F 2 , ∥·∥ v ( X ) ) , i.e., card( U F 2 ( v ))

Original: =
Revised: =

Original: N ( F 2 , ϵ 2 , ∥·∥ v ( X ) ) .
Revised: N ( F 2 , ϵ 2 , ∥·∥ v ( X ) ) .

Original: Denote c 2 =
Revised: Denote c 2 =

Original: Lip( F 2 ) .
Revised: Lip( F 2 ) .

Original: We will show that
Revised: We will show that

Original: [Equation]
Revised: [Equation]

Original: ) defines an ( ϵ 1 c 2 + ϵ 2 ) -cover of (Comp( F 1 , F 2 ) , ∥·∥ X ) , i.e., for any f 1 ∈ F 1 and any f 2 ∈ F 2 , there exist v ∈ U F 1 and w v ∈ U F 2 ( v ) such that
Revised: ) defines an ( ϵ 1 c 2 + ϵ 2 ) -cover of (Comp( F 1 , F 2 ) , ∥·∥ X ) , i.e., for any f 1 ∈ F 1 and any f 2 ∈ F 2 , there exist v ∈ U F 1 and w v ∈ U F 2 ( v ) such that

Original: [Equation]
Revised: [Equation]

Original: Indeed, since U F 1 is an ϵ 1 -cover of ( F 1 , ∥·∥ X ) , we can choose v ∈ U F 1 such that ∥ f 1 − v ∥ X ≤ ϵ 1 , and, since U F 2 ( v ) is an ϵ 2 -cover of ( F 2 , ∥·∥ v ( X ) ) , we can choose w v ∈ U F 2 ( v ) such that ∥ f 2 − w v ∥ v ( X ) ≤ ϵ 2 .
Revised: Indeed, since U F 1 is an ϵ 1 -cover of ( F 1 , ∥·∥ X ) , we can choose v ∈ U F 1 such that ∥ f 1 − v ∥ X ≤ ϵ 1 , and, since U F 2 ( v ) is an ϵ 2 -cover of ( F 2 , ∥·∥ v ( X ) ) , we can choose w v ∈ U F 2 ( v ) such that ∥ f 2 − w v ∥ v ( X ) ≤ ϵ 2 .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: where the second inequality follows from Eq.
Revised: where the second inequality follows from Eq.

Original: Therefore,
Revised: Therefore,

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: For ( ⋆ ) , we used that U F 1 ⊂ F 1 is an internal cover.
Revised: For ( ⋆ ) , we used that U F 1 ⊂ F 1 is an internal cover.

Original: The special case of F 2 = { f 2 } being a singleton is obvious, as we can choose U F 2 ( v ) =
Revised: The special case of F 2 = { f 2 } being a singleton is obvious, as we can choose U F 2 ( v ) =

Original: { f 2 } for every v ∈ U F 1 .
Revised: { f 2 } for every v ∈ U F 1 .

Original: Then, for every f 2 ∈ F 2 and every w v ∈ U F 2 ( v ) , it holds that card( U F 2 ( v ))
Revised: Then, for every f 2 ∈ F 2 and every w v ∈ U F 2 ( v ) , it holds that card( U F 2 ( v ))

Original: =
Revised: =

Original: and (cid:13)(cid:13)
Revised: and (cid:13)(cid:13)

Original: f 2 − w vj
Revised: f 2 − w vj

Original: (cid:13)(cid:13) v ( X ) = 0 .
Revised: (cid:13)(cid:13) v ( X ) = 0 .

Original: Lemma C.12 (Summations) .
Revised: Lemma C.12 (Summations) .

Original: Let ( X , ∥·∥ X ) and ( Y , ∥·∥ Y ) be normed spaces and let F , G be classes of functions
Revised: Let ( X , ∥·∥ X ) and ( Y , ∥·∥ Y ) be normed spaces and let F , G be classes of functions

Original: X → Y .
Revised: X → Y .

Original: Then, for each ϵ F , ϵ G > 0 and each X
Revised: Then, for each ϵ F , ϵ G > 0 and each X

Original: = ( x i , . . .
Revised: = ( x i , . . .

Original: , x n ) ∈ X n , the covering number of the class Sum( F , G ) is bounded by
Revised: , x n ) ∈ X n , the covering number of the class Sum( F , G ) is bounded by

Original: [Equation]
Revised: [Equation]

Original: If G = { g } is a singleton, then
Revised: If G = { g } is a singleton, then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Fix ϵ F , ϵ G > 0 .
Revised: Fix ϵ F , ϵ G > 0 .

Original: Let U F ⊂ F be a minimal ϵ F -cover of ( F , ∥·∥ X ) and let U G ⊂ G be a minimal ϵ G -cover of ( G , ∥·∥ X ) ,
Revised: Let U F ⊂ F be a minimal ϵ F -cover of ( F , ∥·∥ X ) and let U G ⊂ G be a minimal ϵ G -cover of ( G , ∥·∥ X ) ,

Original: i.e., N ( F , ϵ F , ∥·∥ X ) = card( U F ) and N ( G , ϵ G , ∥·∥ X )
Revised: i.e., N ( F , ϵ F , ∥·∥ X ) = card( U F ) and N ( G , ϵ G , ∥·∥ X )

Original: = card( U G ) .
Revised: = card( U G ) .

Original: We will show that
Revised: We will show that

Original: [Equation]
Revised: [Equation]

Original: defines an ( ϵ F + ϵ G ) -cover of (Sum( F , G ) , ∥·∥ X ) , i.e., for every f ∈ F and every g ∈ G , there exist v ∈ U F and w ∈ U G such that
Revised: defines an ( ϵ F + ϵ G ) -cover of (Sum( F , G ) , ∥·∥ X ) , i.e., for every f ∈ F and every g ∈ G , there exist v ∈ U F and w ∈ U G such that

Original: [Equation]
Revised: [Equation]

Original: Indeed, since U F is an ϵ F -cover of ( F , ∥·∥ X ) , we can choose v ∈ U F such that ∥ f − v ∥ X ≤ ϵ F and since U G is an ϵ G -cover of ( G , ∥·∥ X ) , we can choose w ∈ U G such that ∥ g − w ∥
Revised: Indeed, since U F is an ϵ F -cover of ( F , ∥·∥ X ) , we can choose v ∈ U F such that ∥ f − v ∥ X ≤ ϵ F and since U G is an ϵ G -cover of ( G , ∥·∥ X ) , we can choose w ∈ U G such that ∥ g − w ∥

Original: X ≤ ϵ G .
Revised: X ≤ ϵ G .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: g )
Revised: g )

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: The special case of G = { g } being a singleton is obvious, as we can choose U G
Revised: The special case of G = { g } being a singleton is obvious, as we can choose U G

Original: = { g } .
Revised: = { g } .

Original: Then, for every g ∈ G and every w ∈ U G , it holds that card( U G )
Revised: Then, for every g ∈ G and every w ∈ U G , it holds that card( U G )

Original: = 1 and ∥ g − w ∥ X = 0 .
Revised: = 1 and ∥ g − w ∥ X = 0 .

Original: Bound covering numbers
Revised: Bound covering numbers

Original: Identify structure
Revised: Identify structure

Original: F a
Revised: F a

Original: F bb
Revised: F bb

Original: F baa
Revised: F baa

Original: F bab
Revised: F bab

Original: F a
Revised: F a

Original: F b
Revised: F b

Original: F a
Revised: F a

Original: F bb
Revised: F bb

Original: F bab
Revised: F bab

Original: F baab
Revised: F baab

Original: F baaa
Revised: F baaa

Original: FF F ba
Revised: FF F ba

Original: F bb
Revised: F bb

Original: F a F a
Revised: F a F a

Original: F bb
Revised: F bb

Original: F bab
Revised: F bab

Original: F baab F baaa
Revised: F baab F baaa

Original: F a
Revised: F a

Original: F bb
Revised: F bb

Original: F baa
Revised: F baa

Original: F bab
Revised: F bab

Original: F a
Revised: F a

Original: F b
Revised: F b

Original: F ba
Revised: F ba

Original: F bb
Revised: F bb

Original: F a
Revised: F a

Original: Now that we know how to bound the covering numbers of compositions and summations, we can iteratively derive covering number bounds for all function classes obtained from these two operations.
Revised: Now that we know how to bound the covering numbers of compositions and summations, we can iteratively derive covering number bounds for all function classes obtained from these two operations.

Original: C.3.2 General strategy for bounding the covering numbers of complex classes
Revised: C.3.2 General strategy for bounding the covering numbers of complex classes

Original: Let F be a function class whose covering number is unknown to us.
Revised: Let F be a function class whose covering number is unknown to us.

Original: If F can be built iteratively by compositions and summations of function classes with known covering number (bounds), then we can derive covering number bounds for F via the following strategy.
Revised: If F can be built iteratively by compositions and summations of function classes with known covering number (bounds), then we can derive covering number bounds for F via the following strategy.

Original: In a first step, we identify the structure of F , i.e., how it is built from compositions and summations.
Revised: In a first step, we identify the structure of F , i.e., how it is built from compositions and summations.

Original: In a second step, starting with
Revised: In a second step, starting with

Original: F , we iteratively replace each function class by its simpler building blocks and the covering number of F by the respective bound.
Revised: F , we iteratively replace each function class by its simpler building blocks and the covering number of F by the respective bound.

Original: To be more specific, we know by Lemma C.10 and Lemma C.12 that for F = Comp( F a , F b ) , it holds that
Revised: To be more specific, we know by Lemma C.10 and Lemma C.12 that for F = Comp( F a , F b ) , it holds that

Original: [Equation]
Revised: [Equation]

Original: and for F = Sum( F a , F b ) , it holds that
Revised: and for F = Sum( F a , F b ) , it holds that

Original: [Equation]
Revised: [Equation]

Original: ≤ N ( F a
Revised: ≤ N ( F a

Original: [Equation]
Revised: [Equation]

Original: ϵ F a , ∥·∥ X ) N ( F b
Revised: ϵ F a , ∥·∥ X ) N ( F b

Original: [Equation]
Revised: [Equation]

Original: ϵ F b , ∥·∥ X ) .
Revised: ϵ F b , ∥·∥ X ) .

Original: Now, if (for x = a or x = b ) some class F x is of the form F x = Comp( F xa , F xb ) or F x =
Revised: Now, if (for x = a or x = b ) some class F x is of the form F x = Comp( F xa , F xb ) or F x =

Original: Sum( F xa , F xb ) , we bound the right-hand side of the equations above by the same argument.
Revised: Sum( F xa , F xb ) , we bound the right-hand side of the equations above by the same argument.

Original: We repeat this procedure until the right-hand side contains only terms of known covering number bounds.
Revised: We repeat this procedure until the right-hand side contains only terms of known covering number bounds.

Original: For an illustration of this stepwise process, see Fig.
Revised: For an illustration of this stepwise process, see Fig.

Original: C.3.3 Examples
Revised: C.3.3 Examples

Original: Example C.1 (Multi-composition) .
Revised: Example C.1 (Multi-composition) .

Original: Let ( X 1 , ∥·∥ X 1 ) , . . .
Revised: Let ( X 1 , ∥·∥ X 1 ) , . . .

Original: , ( X L , ∥·∥ X L ) be normed spaces.
Revised: , ( X L , ∥·∥ X L ) be normed spaces.

Original: Let F i be classes of functions
Revised: Let F i be classes of functions

Original: X i → X i +1 with bounded Lipschitz constants, i.e., Lip( F i ) <
Revised: X i → X i +1 with bounded Lipschitz constants, i.e., Lip( F i ) <

Original: ∞ .
Revised: ∞ .

Original: Denote →
Revised: Denote →

Original: F i =
Revised: F i =

Original: Comp( F 1 , . .
Revised: Comp( F 1 , . .

Original: , F i − 1 ) .
Revised: , F i − 1 ) .

Original: Then, for any ϵ i > 0 and any finite X = ( x 1 , . . .
Revised: Then, for any ϵ i > 0 and any finite X = ( x 1 , . . .

Original: , x n ) ∈ X n 1 , the covering number of the class F = Comp( F 1 , . .
Revised: , x n ) ∈ X n 1 , the covering number of the class F = Comp( F 1 , . .

Original: . , F L ) is bounded by
Revised: . , F L ) is bounded by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where the inequalities follow from Lemma C.
Revised: where the inequalities follow from Lemma C.

Original: [Equation]
Revised: [Equation]

Original: Remark C.13.
Revised: Remark C.13.

Original: We want to point out that Example C.1 implies the whole-network covering bound as in [3, Lemma A.7].
Revised: We want to point out that Example C.1 implies the whole-network covering bound as in [3, Lemma A.7].

Original: To see this, let X i
Revised: To see this, let X i

Original: = R d i , X i =
Revised: = R d i , X i =

Original: ( x i 1 , . . .
Revised: ( x i 1 , . . .

Original: , x i n ) ∈ R d i × n and let σ i be fixed ρ i -Lipschitz functions.
Revised: , x i n ) ∈ R d i × n and let σ i be fixed ρ i -Lipschitz functions.

Original: Further, let A i be sets of matrices
Revised: Further, let A i be sets of matrices

Original: A ∈ R d
Revised: A ∈ R d

Original: i +1 ×
Revised: i +1 ×

Original: d i .
Revised: d i .

Original: Then, the maps
Revised: Then, the maps

Original: [Equation]
Revised: [Equation]

Original: define isometries, because
Revised: define isometries, because

Original: [Equation]
Revised: [Equation]

Original: Consequently,
Revised: Consequently,

Original: [Equation]
Revised: [Equation]

Original: which are the factors on the right hand side of [3, Lemma A.7].
Revised: which are the factors on the right hand side of [3, Lemma A.7].

Original: Example C.2 (Addition block) .
Revised: Example C.2 (Addition block) .

Original: Let F be the function class of addition blocks, i.e.,
Revised: Let F be the function class of addition blocks, i.e.,

Original: F = Sum( G , H ) , where G =
Revised: F = Sum( G , H ) , where G =

Original: Comp( G 1 , . .
Revised: Comp( G 1 , . .

Original: , G L G ) and H =
Revised: , G L G ) and H =

Original: Comp( H 1 , . .
Revised: Comp( H 1 , . .

Original: , H L H ) .
Revised: , H L H ) .

Original: For brevity, we write → G i =
Revised: For brevity, we write → G i =

Original: Comp( G 1 , . .
Revised: Comp( G 1 , . .

Original: , G i − 1 ) , → H i =
Revised: , G i − 1 ) , → H i =

Original: Comp( H 1 , . .
Revised: Comp( H 1 , . .

Original: , H i − 1 ) .
Revised: , H i − 1 ) .

Original: The covering number of a block F = Sum( G , H ) is bounded by
Revised: The covering number of a block F = Sum( G , H ) is bounded by

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From Lemma C.12, we know that
Revised: From Lemma C.12, we know that

Original: [Equation]
Revised: [Equation]

Original: holds for every ϵ G > 0 and ϵ H > 0 .
Revised: holds for every ϵ G > 0 and ϵ H > 0 .

Original: Choosing
Revised: Choosing

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: and bounding each factor on the right-hand side via Example C.1 yields Eq.
Revised: and bounding each factor on the right-hand side via Example C.1 yields Eq.

Original: Example C.3 (Residual network) .
Revised: Example C.3 (Residual network) .

Original: In the setting of Example C.1, let the function classes
Revised: In the setting of Example C.1, let the function classes

Original: F i be residual blocks
Revised: F i be residual blocks

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: G i =
Revised: G i =

Original: Comp( G i 1 , . .
Revised: Comp( G i 1 , . .

Original: , G iL G i ) and H i =
Revised: , G iL G i ) and H i =

Original: Comp( H i 1 , . .
Revised: Comp( H i 1 , . .

Original: , H iL H i ) .
Revised: , H iL H i ) .

Original: Assume, that Lip( G ij ) , Lip( H ij ) < ∞ and that input data X is given.
Revised: Assume, that Lip( G ij ) , Lip( H ij ) < ∞ and that input data X is given.

Original: For brevity, we write → G ij =
Revised: For brevity, we write → G ij =

Original: Comp
Revised: Comp

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: The covering number of the residual network, F = Comp( F 1 , . .
Revised: The covering number of the residual network, F = Comp( F 1 , . .

Original: . , F L ) , is bounded by
Revised: . , F L ) , is bounded by

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: with ϵ F i =
Revised: with ϵ F i =

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: j =
Revised: j =

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: i (cid:89) k
Revised: i (cid:89) k

Original: = j +
Revised: = j +

Original: Lip( G ik
Revised: Lip( G ik

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: 
Revised: 

Original: ϵ G
Revised: ϵ G

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Assuming card( F i ) > 1 , we apply Example C.1 to F = Comp( F 1 , . .
Revised: Assuming card( F i ) > 1 , we apply Example C.1 to F = Comp( F 1 , . .

Original: . , F L ) to obtain
Revised: . , F L ) to obtain

Original: N ( F
Revised: N ( F

Original: [Equation]
Revised: [Equation]

Original: Bounding the covering number of each block F i via Example C.2 yields Eq.
Revised: Bounding the covering number of each block F i via Example C.2 yields Eq.

Original: C.3.4 Covering number bounds for concatenations
Revised: C.3.4 Covering number bounds for concatenations

Original: The general approach to bounding covering numbers of function classes, obtained from linking simple function classes via summations and compositions, can be easily extended.
Revised: The general approach to bounding covering numbers of function classes, obtained from linking simple function classes via summations and compositions, can be easily extended.

Original: As an example, we can incorporate concatenations , as typically used in DenseNets [22], via the following lemma.
Revised: As an example, we can incorporate concatenations , as typically used in DenseNets [22], via the following lemma.

Original: Lemma C.14 (Concatenations) .
Revised: Lemma C.14 (Concatenations) .

Original: Let ( X , ∥·∥ X ) be a normed space and let ( Y , ∥·∥ Y ) =
Revised: Let ( X , ∥·∥ X ) be a normed space and let ( Y , ∥·∥ Y ) =

Original: ( R d Y , ∥·∥ l 2 ) and ( Z , ∥·∥ Z ) =
Revised: ( R d Y , ∥·∥ l 2 ) and ( Z , ∥·∥ Z ) =

Original: ( R d Z , ∥·∥ l 2 ) .
Revised: ( R d Z , ∥·∥ l 2 ) .

Original: Let F , G be classes of functions X
Revised: Let F , G be classes of functions X

Original: → Y , resp.
Revised: → Y , resp.

Original: X → Z .
Revised: X → Z .

Original: Define the function class Cat( F , G ) of concatenations X
Revised: Define the function class Cat( F , G ) of concatenations X

Original: → Y × Z as
Revised: → Y × Z as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: If we equip Y × Z
Revised: If we equip Y × Z

Original: = R d Y d
Revised: = R d Y d

Original: Z with the l 2 -norm, then
Revised: Z with the l 2 norm, then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Fix ϵ F , ϵ G > 0 .
Revised: Fix ϵ F , ϵ G > 0 .

Original: Let U F ⊂ F be a minimal ϵ F -cover of ( F , ∥·∥ X ) and let U G ⊂ G be a minimal ϵ G -cover of ( G , ∥·∥ X ) ,
Revised: Let U F ⊂ F be a minimal ϵ F -cover of ( F , ∥·∥ X ) and let U G ⊂ G be a minimal ϵ G -cover of ( G , ∥·∥ X ) ,

Original: i.e., N ( F , ϵ F , ∥·∥ X ) = card( U F ) and N ( G , ϵ G , ∥·∥ X )
Revised: i.e., N ( F , ϵ F , ∥·∥ X ) = card( U F ) and N ( G , ϵ G , ∥·∥ X )

Original: = card( U G ) .
Revised: = card( U G ) .

Original: We will show that
Revised: We will show that

Original: [Equation]
Revised: [Equation]

Original: defines an (cid:113) ϵ 2 F + ϵ 2 G -cover of (Cat( F , G ) , ∥·∥ l 2 ) , i.e., for every f ∈ F and every g ∈ G , there exist v ∈ U F and w ∈ U G such that
Revised: defines an (cid:113) ϵ 2 F + ϵ 2 G -cover of (Cat( F , G ) , ∥·∥ l 2 ) , i.e., for every f ∈ F and every g ∈ G , there exist v ∈ U F and w ∈ U G such that

Original: [Equation]
Revised: [Equation]

Original: Indeed, since U F is an ϵ F -cover of ( F , ∥·∥ X ) , we can choose v such that ∥ f − v ∥ X ≤ ϵ F and since
Revised: Indeed, since U F is an ϵ F -cover of ( F , ∥·∥ X ) , we can choose v such that ∥ f − v ∥ X ≤ ϵ F and since

Original: U G is an ϵ G -cover of ( G , ∥·∥ X ) , we can choose w such that ∥ g − w ∥ X ≤ ϵ G .
Revised: U G is an ϵ G -cover of ( G , ∥·∥ X ) , we can choose w such that ∥ g − w ∥ X ≤ ϵ G .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: ≤

Original: [Equation]

Original: ( U
Revised: ( U

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: G ) ) =
Revised: ) =

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: , w
Revised: U F

Original: [Equation]
Revised: [Equation]

Original: | [Equation] ∈ UF
Revised: U

Original: [Equation]
Revised: [Equation]

Original: U G } ) =
Revised: ) =

Original: [Equation]
Revised: [Equation]

Original: C.4 Whole-network covering number bounds (convolutional & fully-connected)
Revised: C.4 Whole-network covering number bounds (convolutional & fully-connected)

Original: In order to compute covering number bounds for specific residual network architectures, we need to specify the function classes G i and H i .
Revised: In order to compute covering number bounds for specific residual network architectures, we need to specify the function classes G i and H i .

Original: We will present exemplary proofs for a simple residual network with fixed shortcuts (Theorem C.15, which corresponds to Theorem 3.4 in the main text) and the ResNet18 architecture [20] without batch normalization, see Example C.4.
Revised: We will present exemplary proofs for a simple residual network with fixed shortcuts (Theorem C.15, which corresponds to Theorem 3.4 in the main text) and the ResNet18 architecture [20] without batch normalization, see Example C.4.

Original: C.4.1 Bounds for residual networks
Revised: C.4.1 Bounds for residual networks

Original: Theorem C.15 (Covering numbers for residual networks) .
Revised: Theorem C.15 (Covering numbers for residual networks) .

Original: For i = 1 , . . .
Revised: For i = 1 , . . .

Original: , L let j = 1 , . . .
Revised: , L let j = 1 , . . .

Original: , L i , s ij > 0 and b ij > 0 .
Revised: , L i , s ij > 0 and b ij > 0 .

Original: Further, let F be the class of residual networks of the form
Revised: Further, let F be the class of residual networks of the form

Original: [Equation]
Revised: [Equation]

Original: with σ i fixed ρ i -Lipschitz functions satisfying σ i (0) = 0 , and f i residual blocks with fixed shortcuts g i , i.e.,
Revised: with σ i fixed ρ i -Lipschitz functions satisfying σ i (0) = 0 , and f i residual blocks with fixed shortcuts g i , i.e.,

Original: [Equation]
Revised: [Equation]

Original: where σ ij are fixed ρ ij -Lipschitz functions with σ ij (0) = 

Revised: where σ ij are fixed ρ ij -Lipschitz functions with σ ij (0) =

Original: and g i is Lipschitz with g i (0) = 0 .
Revised: and g i is Lipschitz with g i (0) = 0 .

Original: The fully-connected or convolutional layers f ij ∈ layer ij are parametrized respectively by matrices
Revised: The fully-connected or convolutional layers f ij ∈ layer ij are parametrized respectively by matrices

Original: A ij or weight tensors K ij .
Revised: A ij or weight tensors K ij .

Original: They satisfy Lipschitz constant constraints s ij and (2,1) group norm distance constraints b ij w.r.t.
Revised: They satisfy Lipschitz constant constraints s ij and (2,1) group norm distance constraints b ij w.r.t.

Original: reference weights M ij .
Revised: reference weights M ij .

Original: That is, for convolutions
Revised: That is, for convolutions

Original: [Equation]
Revised: [Equation]

Original: and for fully-connected layers
Revised: and for fully-connected layers

Original: [Equation]
Revised: [Equation]

Original: Upon letting W ij denote the number of parameters of each layer and defining
Revised: Upon letting W ij denote the number of parameters of each layer and defining

Original: [Equation]
Revised: [Equation]

Original: it holds that
Revised: it holds that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: As we consider residual networks with fixed shortcuts
Revised: As we consider residual networks with fixed shortcuts

Original: [Equation]
Revised: [Equation]

Original: the covering number bound from
Revised: the covering number bound from

Original: Example C.3 simplifies to
Revised: Example C.3 simplifies to

Original: [Equation]
Revised: [Equation]

Original: In our setting, F i = { σ i ◦ f i } with Lip( F i ) ≤ ρ i s i , H ij = Comp(layer ij , { σ ij } ) with Lip( H ij ) ≤ σ ij s ij .
Revised: In our setting, F i = { σ i ◦ f i } with Lip( F i ) ≤ ρ i s i , H ij = Comp(layer ij , { σ ij } ) with Lip( H ij ) ≤ σ ij s ij .

Original: Further, ϵ H ij = ρ ij ϵ ij
Revised: Further, ϵ H ij = ρ ij ϵ ij

Original: and
Revised: and

Original: L H ij =
Revised: L H ij =

Original: L ij .
Revised: L ij .

Original: As covering numbers decrease with the radius, it follows that
Revised: As covering numbers decrease with the radius, it follows that

Original: [Equation]
Revised: [Equation]

Original: Now, for each ij referring to convolutional layers, we have
Revised: Now, for each ij referring to convolutional layers, we have

Original: [Equation]
Revised: [Equation]

Original: In this chain of equalities
Revised: In this chain of equalities

Original: [Equation]
Revised: [Equation]

Original: we used the translation invariance of covering numbers, i.e., Lemma C. with one summand being the singleton (cid:8) − ϕ M ij (cid:9) , and the linearity of ϕ in the weights to accommodate the distance to initialization.
Revised: we used the translation invariance of covering numbers, i.e., Lemma C. with one summand being the singleton (cid:8) − ϕ M ij (cid:9) , and the linearity of ϕ in the weights to accommodate the distance to initialization.

Original: An analogous inequality holds for fully-connected layers.
Revised: An analogous inequality holds for fully-connected layers.

Original: Theorem C.7 provides bounds for the covering number of the superset
Revised: Theorem C.7 provides bounds for the covering number of the superset

Original: [Equation]
Revised: [Equation]

Original: Hence, to proceed
Revised: Hence, to proceed

Original: [Equation]
Revised: [Equation]

Original: we need to transition to external covering numbers
Revised: we need to transition to external covering numbers

Original: [Equation]
Revised: [Equation]

Original: which requires halving the radius ϵ .
Revised: which requires halving the radius ϵ .

Original: This yields
Revised: This yields

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: by Eq.
Revised: by Eq.

Original: [Equation]
Revised: [Equation]

Original: it holds that
Revised: it holds that

Original: [Equation]
Revised: [Equation]

Original: Notably
Revised: Notably

Original: [Equation]
Revised: [Equation]

Original: the second inequality requires the assumption that all g i , f ij , σ i and σ ij map zero to zero.
Revised: the second inequality requires the assumption that all g i , f ij , σ i and σ ij map zero to zero.

Original: The next step is to choose radii
Revised: The next step is to choose radii

Original: ϵ ij so that the right-hand side becomes small under the condition
Revised: ϵ ij so that the right-hand side becomes small under the condition

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: We choose
Revised: We choose

Original: [Equation]
Revised: [Equation]

Original: which would be optimal for the analogous optimization problem without ceiling functions.
Revised: which would be optimal for the analogous optimization problem without ceiling functions.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: L (cid:89)
Revised: L (cid:89)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: l
Revised: l

Original: [Equation]
Revised: [Equation]

Original: (cid:16)(cid:81)
Revised: (cid:16)(cid:81)

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: 
Revised: 

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: n ϵ
Revised: n ϵ

Original: [Equation]
Revised: [Equation]

Original: = log(2 W )  
Revised: = log(2 W )  

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: (cid:108) n
Revised: (cid:108) n

Original: [Equation]
Revised: [Equation]

Original: (cid:109) , which establishes the first covering number bound, i.e., Eq.
Revised: (cid:109) , which establishes the first covering number bound, i.e., Eq.

Original: (72), from Theorem C.15.
Revised: (72), from Theorem C.15.

Original: Similarly, Eq.
Revised: Similarly, Eq.

Original: (55) implies log N   F ,
Revised: (55) implies log N   F ,

Original: L (cid:88)
Revised: L (cid:88)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: ρ
Revised: ρ

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: (cid:33)
Revised: (cid:33)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: (cid:33)
Revised: (cid:33)

Original: [Equation]
Revised: [Equation]

Original: ij ϵ
Revised: ij ϵ

Original: [Equation]
Revised: [Equation]

Original: ij   
Revised: ij   

Original:  .
Revised:  .

Original: Again, we need to choose the ϵ ij such that Eq.
Revised: Again, we need to choose the ϵ ij such that Eq.

Original: (75) holds.
Revised: (75) holds.

Original: We choose
Revised: We choose

Original: [Equation]
Revised: [Equation]

Original: α
Revised: α

Original: [Equation]
Revised: [Equation]

Original: This simple choice yields the optimal solution for the problem of minimizing
Revised: This simple choice yields the optimal solution for the problem of minimizing

Original: [Equation]
Revised: [Equation]

Original: Hence, we expect it to be a good choice if the W ij are roughly equal and ϵ is small.
Revised: Hence, we expect it to be a good choice if the W ij are roughly equal and ϵ is small.

Original: Overall, we get log N ( F , ϵ, ∥·∥ X ) ≤
Revised: Overall, we get log N ( F , ϵ, ∥·∥ X ) ≤

Original: [Equation]
Revised: [Equation]

Original: which establishes the second covering number bound, i.e., Eq.
Revised: which establishes the second covering number bound, i.e., Eq.

Original: (73), from Theorem C.15.
Revised: (73), from Theorem C.15.

Original: Corollary C.16 (Covering numbers for non-residual networks) .
Revised: Corollary C.16 (Covering numbers for non-residual networks) .

Original: For i ∈ { 1 , . . .
Revised: For i ∈ { 1 , . . .

Original: , L } , let layer i be a function class with Lipschitz constraint s i and
Revised: , L } , let layer i be a function class with Lipschitz constraint s i and

Original: (2,1) group norm distance constraint b i with respect to a reference weight M i .
Revised: (2,1) group norm distance constraint b i with respect to a reference weight M i .

Original: In particular, if layer i is convolutional, then
Revised: In particular, if layer i is convolutional, then

Original: [Equation]
Revised: [Equation]

Original: and if layer i is fully-connected, then
Revised: and if layer i is fully-connected, then

Original: [Equation]
Revised: [Equation]

Original: We write W i for the number of parameters of each layer, i.e., the number of elements of each K i , resp.
Revised: We write W i for the number of parameters of each layer, i.e., the number of elements of each K i , resp.

Original: A i .
Revised: A i .

Original: Further, let F = { σ L ◦ f L ◦ · · · ◦ σ 1 ◦ f 1 | f i ∈ layer i } , where the maps σ i are ρ i -Lipschitz with σ i (0) = 0 , and define
Revised: Further, let F = { σ L ◦ f L ◦ · · · ◦ σ 1 ◦ f 1 | f i ∈ layer i } , where the maps σ i are ρ i -Lipschitz with σ i (0) = 0 , and define

Original: [Equation]
Revised: [Equation]

Original: Then, for every input data X = ( x 1 , . . .
Revised: Then, for every input data X = ( x 1 , . . .

Original: , x n ) and every ϵ > 0 , it holds that
Revised: , x n ) and every ϵ > 0 , it holds that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Follows directly from Theorem C.15, as the network can be considered as a single (long) residual block, whose shortcut g : x (cid:55)→ 0 is the zero map.
Revised: Follows directly from Theorem C.15, as the network can be considered as a single (long) residual block, whose shortcut g : x (cid:55)→ 0 is the zero map.

Original: Remark C.17.
Revised: Remark C.17.

Original: Similarly, we can derive covering number bounds for networks, where each block F i is a sum of w i parametrized maps, i.e., F i =
Revised: Similarly, we can derive covering number bounds for networks, where each block F i is a sum of w i parametrized maps, i.e., F i =

Original: Sum( G i 1 , . . .
Revised: Sum( G i 1 , . . .

Original: , G iw i ) , with G ij =
Revised: , G iw i ) , with G ij =

Original: Comp(layer ij 1 , σ ij 1 , . .
Revised: Comp(layer ij 1 , σ ij 1 , . .

Original: . , layer ijL ij , σ ijL ij ) .
Revised: . , layer ijL ij , σ ijL ij ) .

Original: In this setting, the whole-network covering number is bounded by
Revised: In this setting, the whole-network covering number is bounded by

Original: [Equation]
Revised: [Equation]

Original: (81) and log N ( F , ϵ, ∥·∥ X ) ≤
Revised: (81) and log N ( F , ϵ, ∥·∥ X ) ≤

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: (cid:109)(cid:17) (82) for
Revised: (cid:109)(cid:17) (82) for

Original: [Equation]
Revised: [Equation]

Original: (83) where s i =
Revised: (83) where s i =

Original: (cid:80) w i j =1 (cid:16)(cid:81)
Revised: (cid:80) w i j =1 (cid:16)(cid:81)

Original: L ij
Revised: L ij

Original: [Equation]
Revised: [Equation]

Original: =1 s
Revised: =1 s

Original: [Equation]
Revised: [Equation]

Original: , ρ ijk = Lip(
Revised: , ρ ijk = Lip(

Original: C.4.2 Application to specific architectures
Revised: C.4.2 Application to specific architectures

Original: Example C.4 (ResNet18) .
Revised: Example C.4 (ResNet18) .

Original: We derive covering number bounds for the ResNet18 architecture [20] without batch normalization, illustrated in Fig.
Revised: We derive covering number bounds for the ResNet18 architecture [20] without batch normalization, illustrated in Fig.

Original: We can think of the ResNet18 as a composition of 10 residual blocks, the first and last one having the zero map as shortcut and five blocks having identity shortcuts.
Revised: We can think of the ResNet18 as a composition of 10 residual blocks, the first and last one having the zero map as shortcut and five blocks having identity shortcuts.

Original: The remaining 3 blocks have downsampling shortcuts of the form σ ◦ ψ , where ψ is a 1x1 convolution and ρ is the ReLU activation function.
Revised: The remaining 3 blocks have downsampling shortcuts of the form σ ◦ ψ , where ψ is a 1x1 convolution and ρ is the ReLU activation function.

Original: These blocks are handled by Remark C.17.
Revised: These blocks are handled by Remark C.17.

Original: Furthermore, all nonlinearities are 1 -Lipschitz and map zero to zero.
Revised: Furthermore, all nonlinearities are 1 -Lipschitz and map zero to zero.

Original: For any data X = ( x 1 , . . .
Revised: For any data X = ( x 1 , . . .

Original: x n ) and any ϵ > 0 , the covering number of the function class F corresponding to the ResNet18 architecture without batch normalization, with no bias parameters and
Revised: x n ) and any ϵ > 0 , the covering number of the function class F corresponding to the ResNet18 architecture without batch normalization, with no bias parameters and

Original: C onv7x7 R e L U M a x P oo l
Revised: C onv7x7 R e L U M a x P oo l

Original: I d
Revised: I d

Original: C onv3x3 R e L U
Revised: C onv3x3 R e L U

Original: C onv3x
Revised: C onv3x

Original: R e L U
Revised: R e L U

Original: I d
Revised: I d

Original: C onv3x3 R e L U 
Revised: C onv3x3 R e L U C onv3x R e L U

Revised: C onv1x

Revised: C onv3x3 R e L U

Original: C onv3x
Revised: C onv3x

Original: R e L U
Revised: R e L U

Original: C onv1x
Revised: I d

Original: C onv3x3 R e L U
Revised: C onv3x3 R e L U

Original: C onv3x
Revised: C onv3x

Original: R e L U
Revised: R e L U

Original: I d

Original: C onv3x3 R e L U
Revised: C onv3x3 R e L U

Original: C onv3x
Revised: C onv3x

Original: R e L U
Revised: R e L U

Original: C onv3x3 R e L U C onv3x R e L U I d
Revised: I d

Original: C onv3x3 R e L U
Revised: C onv3x3 R e L U

Original: C onv3x
Revised: C onv3x

Original: R e L U
Revised: R e L U

Original: C onv3x3 R e L U
Revised: C onv3x3 R e L U

Original: C onv3x
Revised: C onv3x

Original: R e L U
Revised: R e L U

Original: I d
Revised: I d

Original: C onv3x3 R e L U
Revised: C onv3x3 R e L U

Original: C onv3x
Revised: C onv3x

Original: R e L U
Revised: R e L U

Original: A vg P oo l
Revised: A vg P oo l

Original: F C
Revised: F C

Original: C onv1x C onv1x
Revised: C onv1x C onv1x

Original: I npu t
Revised: I npu t

Original: P re d i c t i o n with distance and Lipschitz constrained layers, is approximately (ignoring ceiling functions) bounded by log
Revised: P re d i c t i o n with distance and Lipschitz constrained layers, is approximately (ignoring ceiling functions) bounded by log

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:89)
Revised: (cid:89)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:34)
Revised: (cid:34)

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: / 3s
Revised: / 3s

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:88) i ∈{ 2 , 3 , 5 , 7 , 9 } (1 + s i 1 s i 2 )
Revised: (cid:88) i ∈{ 2 , 3 , 5 , 7 , 9 } (1 + s i 1 s i 2 )

Original: [Equation]
Revised: [Equation]

Original: /(cid:32)
Revised: /(cid:32)

Original: [Equation]
Revised: [Equation]

Original: Here s i 1 , resp s i 2 , denotes the Lipschitz constraint on the first, resp.
Revised: Here s i 1 , resp s i 2 , denotes the Lipschitz constraint on the first, resp.

Original: second, layer in the i -th residual block and s i, down the constraint on the downsampling layer (1x1 convolution).
Revised: second, layer in the i -th residual block and s i, down the constraint on the downsampling layer (1x1 convolution).

Original: The (2,1)-distance constraints are denoted by b i 1 , b i 2 and b i, down .
Revised: The (2,1)-distance constraints are denoted by b i 1 , b i 2 and b i, down .

Original: C.5 Rademacher complexity & Generalization bounds
Revised: C.5 Rademacher complexity & Generalization bounds

Original: The empirical Rademacher complexity can be upper bounded via Dudley’s entropy integral.
Revised: The empirical Rademacher complexity can be upper bounded via Dudley’s entropy integral.

Original: In the following, we restate a variant of this standard result as it appears in Bartlett et al.
Revised: In the following, we restate a variant of this standard result as it appears in Bartlett et al.

Original: Theorem C.18 (Dudley entropy integral, cf.
Revised: Theorem C.18 (Dudley entropy integral, cf.

Original: [3, Lemma A.5]) .
Revised: [3, Lemma A.5]) .

Original: Let F be a class of functions mapping to [0 , 1] containing the zero function.
Revised: Let F be a class of functions mapping to [0 , 1] containing the zero function.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: We will compute Dudley’s entropy integral for the covering number bounds from Section C.4.
Revised: We will compute Dudley’s entropy integral for the covering number bounds from Section C.4.

Original: Theorem C.19 (Empirical Rademacher complexity for residual networks) .
Revised: Theorem C.19 (Empirical Rademacher complexity for residual networks) .

Original: For i = 1 , . . .
Revised: For i = 1 , . . .

Original: , L let j = 1 , . . .
Revised: , L let j = 1 , . . .

Original: , L i , s ij > 0 and b ij > 0 .
Revised: , L i , s ij > 0 and b ij > 0 .

Original: Further, let F be the class of residual networks of the form
Revised: Further, let F be the class of residual networks of the form

Original: [Equation]
Revised: [Equation]

Original: with σ i fixed ρ i -Lipschitz functions satisfying σ i (0) = 0 , and with f i residual blocks with identity shortcuts, i.e.,
Revised: with σ i fixed ρ i -Lipschitz functions satisfying σ i (0) = 0 , and with f i residual blocks with identity shortcuts, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where σ ij are fixed ρ ij -Lipschitz functions with σ ij (0) = 0 and f ij are convolutional or fullyconnected layers.
Revised: where σ ij are fixed ρ ij -Lipschitz functions with σ ij (0) = 0 and f ij are convolutional or fullyconnected layers.

Original: The layers f ij satisfy Lipschitz constraints Lip( f ij ) ≤ s ij and the corresponding weight tensors K ij , respectively weight matrices
Revised: The layers f ij satisfy Lipschitz constraints Lip( f ij ) ≤ s ij and the corresponding weight tensors K ij , respectively weight matrices

Original: A ij , satisfy distance constraints
Revised: A ij , satisfy distance constraints

Original: [Equation]
Revised: [Equation]

Original: respectively
Revised: respectively

Original: [Equation]
Revised: [Equation]

Original: ≤ b ij
Revised: ≤ b ij

Original: [Equation]
Revised: [Equation]

Original: with respect to reference weights K (0) ij , respectively A (0) ij .
Revised: with respect to reference weights K (0) ij , respectively A (0) ij .

Original: Upon letting W ij denote the number of parameters of each layer and defining
Revised: Upon letting W ij denote the number of parameters of each layer and defining

Original: [Equation]
Revised: [Equation]

Original: the empirical Rademacher complexity of the function class F γ , with margin parameter γ > 0 , satisfies
Revised: the empirical Rademacher complexity of the function class F γ , with margin parameter γ > 0 , satisfies

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: (cid:18)
Revised: (cid:18)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation]
Revised: [Equation]

Original: (cid:18)
Revised: (cid:18)

Original: [Equation]
Revised: [Equation]

Original: (cid:108)
Revised: (cid:108)

Original: [Equation]
Revised: [Equation]

Original: Here, H n − 1 = (cid:80) n − 1 m =1 1 m denotes the ( n − 1) -th
Revised: Here, H n − 1 = (cid:80) n − 1 m =1 1 m denotes the ( n − 1) -th

Original: harmonic number and ζ ( s, q ) = (cid:80) ∞ n = 1 ( q + n ) s the
Revised: harmonic number and ζ ( s, q ) = (cid:80) ∞ n = 1 ( q + n ) s the

Original: Hurwitz zeta function.
Revised: Hurwitz zeta function.

Original: Remark C.20.
Revised: Remark C.20.

Original: The harmonic number satisfies H n − 1 < log( n ) + γ ≈ log( n ) + 0 .
Revised: The harmonic number satisfies H n − 1 < log( n ) + γ ≈ log( n ) + 0 .

Original: The function ψ : x (cid:55)→ ζ (cid:0) 32 , 1 (cid:1) 1 / 3 ζ (cid:0) 32 , 1 + 1 /x (cid:1) 2 / 3 is monotonically increasing with ψ (0)
Revised: The function ψ : x (cid:55)→ ζ (cid:0) 32 , 1 (cid:1) 1 / 3 ζ (cid:0) 32 , 1 + 1 /x (cid:1) 2 / 3 is monotonically increasing with ψ (0)

Original: = 

Revised: =

Original: and upper bounded by ζ ( 32 )
Revised: and upper bounded by ζ ( 32 )

Original: ≈ 2 . 62 .
Revised: ≈ 2 . 62 .

Original: So, for large ˜ C = max ij ˜ C ij , the second summand is negligible and
Revised: So, for large ˜ C = max ij ˜ C ij , the second summand is negligible and

Original: Eq.
Revised: Eq.

Original: (87) scales as (cid:113) W log(¯ L 2 ˜ C ) .
Revised: (87) scales as (cid:113) W log(¯ L 2 ˜ C ) .

Original: Proof.
Revised: Proof.

Original: Both inequalities follow from a combination of Dudley’s entropy integral with a covering number bound from Theorem C.15.
Revised: Both inequalities follow from a combination of Dudley’s entropy integral with a covering number bound from Theorem C.15.

Original: Since ℓ γ ( −M ( · , · )) is a fixed 2 /γ -Lipschitz function, the covering number of F γ can be bounded as in Corollary C.16 with
Revised: Since ℓ γ ( −M ( · , · )) is a fixed 2 /γ -Lipschitz function, the covering number of F γ can be bounded as in Corollary C.16 with

Original: [Equation]
Revised: [Equation]

Original: To prove Eq.
Revised: To prove Eq.

Original: (86), we insert Eq.
Revised: (86), we insert Eq.

Original: (72) into Dudley’s entropy integral, which yields
Revised: (72) into Dudley’s entropy integral, which yields

Original: [Equation]
Revised: [Equation]

Original: The value of the integral is a harmonic number if 1 /t ∈ N , as then (cid:90)
Revised: The value of the integral is a harmonic number if 1 /t ∈ N , as then (cid:90)

Original: [Equation]
Revised: [Equation]

Original: Choosing t = 1 /n , establishes the inequality in Eq.
Revised: Choosing t = 1 /n , establishes the inequality in Eq.

Original: [Equation]
Revised: [Equation]

Original: To prove Eq.
Revised: To prove Eq.

Original: (87), we first observe that, by Jensen’s inequality, it holds that
Revised: (87), we first observe that, by Jensen’s inequality, it holds that

Original: [Equation]
Revised: [Equation]

Original: and thus
Revised: and thus

Original: [Equation]
Revised: [Equation]

Original: Then, recalling Eq.
Revised: Then, recalling Eq.

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e.,
Revised: e.,

Original: [Equation]
Revised: [Equation]

Original: yields
Revised: yields

Original: [Equation]
Revised: [Equation]

Original: Lemma
Revised: Lemma

Original: [Equation]
Revised: [Equation]

Original: The last inequality follows from Lemma C.25 (proof deferred to Section C.6).
Revised: The last inequality follows from Lemma C.25 (proof deferred to Section C.6).

Original: Overall, this implies ˆ
Revised: Overall, this implies ˆ

Original: [Equation]
Revised: [Equation]

Original: which establishes the inequality in Eq.
Revised: which establishes the inequality in Eq.

Original: Corollary C.21.
Revised: Corollary C.21.

Original: Let γ > 0 and let ˜ C
Revised: Let γ > 0 and let ˜ C

Original: i =
Revised: i =

Original: 2 C i /γ .
Revised: 2 C i /γ .

Original: For non-residual networks as specified in
Revised: For non-residual networks as specified in

Original: Corollary C.
Revised: Corollary C.

Original: [Equation]
Revised: [Equation]

Original: the empirical Rademacher complexity of F γ satisfies
Revised: the empirical Rademacher complexity of F γ satisfies

Original: [Equation]
Revised: [Equation]

Original: ≤ 4 n + 12 H n − 1 √ n (cid:112) log(2 W ) (cid:32) L (cid:88)
Revised: ≤ 4 n + 12 H n − 1 √ n (cid:112) log(2 W ) (cid:32) L (cid:88)

Original: i = (cid:108)
Revised: i = (cid:108)

Original: ˜ C 2 / 3 i (cid:109)(cid:33) 3 /(89) and ˆ R X
Revised: ˜ C 2 / 3 i (cid:109)(cid:33) 3 /(89) and ˆ R X

Original: ( F γ )
Revised: ( F γ )

Original: [Equation]
Revised: [Equation]

Original: 12 √ n (cid:118)(cid:117)(cid:117)(cid:116)
Revised: 12 √ n (cid:118)(cid:117)(cid:117)(cid:116)

Original: L (cid:88)
Revised: L (cid:88)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: log (cid:16)
Revised: log (cid:16)

Original: [Equation]
Revised: [Equation]

Original: (cid:108)
Revised: (cid:108)

Original: [Equation]
Revised: [Equation]

Original: (cid:109)(cid:17)
Revised: (cid:109)(cid:17)

Original: [Equation]
Revised: [Equation]

Original: (cid:18)
Revised: (cid:18)

Original: [Equation]
Revised: [Equation]

Original: 1 (cid:19)
Revised: 1 (cid:19)

Original: [Equation]
Revised: [Equation]

Original: (cid:18)
Revised: (cid:18)

Original: [Equation]
Revised: [Equation]

Original: 1 +
Revised: 1 +

Original: [Equation]
Revised: [Equation]

Original: (cid:108)
Revised: (cid:108)

Original: [Equation]
Revised: [Equation]

Original: ˜
Revised: ˜

Original: [Equation]
Revised: [Equation]

Original: (cid:109)(cid:19)
Revised: (cid:109)(cid:19)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For the sake of completeness, we state the generalization bounds that result from the Rademacher complexity bounds for networks with a priori constrained weights.
Revised: For the sake of completeness, we state the generalization bounds that result from the Rademacher complexity bounds for networks with a priori constrained weights.

Original: Theorem C.22.
Revised: Theorem C.22.

Original: For i = 1 , . . .
Revised: For i = 1 , . . .

Original: , L let j = 1 , . . .
Revised: , L let j = 1 , . . .

Original: , L i , s ij > 0 and b ij > 0 .
Revised: , L i , s ij > 0 and b ij > 0 .

Original: Let F be the class of residual networks of the form
Revised: Let F be the class of residual networks of the form

Original: [Equation]
Revised: [Equation]

Original: with σ i fixed ρ i -Lipschitz functions satisfying σ i (0) = 0 , and with f i residual blocks with identity shortcuts, i.e.,
Revised: with σ i fixed ρ i -Lipschitz functions satisfying σ i (0) = 0 , and with f i residual blocks with identity shortcuts, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where σ ij are fixed ρ ij -Lipschitz functions with σ ij (0) = 0 and f ij are convolutional or fullyconnected layers whose weight tensors K ij , resp.
Revised: where σ ij are fixed ρ ij -Lipschitz functions with σ ij (0) = 0 and f ij are convolutional or fullyconnected layers whose weight tensors K ij , resp.

Original: weight matrices
Revised: weight matrices

Original: A ij , satisfy the distance constraints
Revised: A ij , satisfy the distance constraints

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: with respect to reference weights K (0) ij , resp.
Revised: with respect to reference weights K (0) ij , resp.

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: 0) ij , and the Lipschitz constraints Lip( f ij ) ≤ s ij .
Revised: 0) ij , and the Lipschitz constraints Lip( f ij ) ≤ s ij .

Original: Let W ij denote the number of parameters of each layer and define
Revised: Let W ij denote the number of parameters of each layer and define

Original: [Equation]
Revised: [Equation]

Original: Then, for fixed margin parameter γ > 0 , every network f ∈ F satisfies
Revised: Then, for fixed margin parameter γ > 0 , every network f ∈ F satisfies

Original: P [
Revised: P [

Original: [Equation]
Revised: [Equation]

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: with probability of at least 1 − δ over an i.i.d.
Revised: with probability of at least 1 − δ over an i.i.d.

Original: draw (( x 1 , y 1 ) , . . .
Revised: draw (( x 1 , y 1 ) , . . .

Original: , ( x n , y n )) .
Revised: , ( x n , y n )) .

Original: Proof.
Revised: Proof.

Original: Recall Lemma 3.1, i.e.,
Revised: Recall Lemma 3.1, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Bounding the empirical Rademacher complexity ˆ R S ( F ) via Theorem C.19 proves the theorem.
Revised: Bounding the empirical Rademacher complexity ˆ R S ( F ) via Theorem C.19 proves the theorem.

Original: Remark C.23.
Revised: Remark C.23.

Original: By a union bound argument over the constraint sets and the margin parameter, the generalization bound above can be transformed to a generalization bound which depends directly on the norms of the network weights and the Lipschitz constants instead of a priori constraints, see for example [3, Lemma A.9].
Revised: By a union bound argument over the constraint sets and the margin parameter, the generalization bound above can be transformed to a generalization bound which depends directly on the norms of the network weights and the Lipschitz constants instead of a priori constraints, see for example [3, Lemma A.9].

Original: Furthermore, Lipschitz augmentation [43] allows to replace the product of
Revised: Furthermore, Lipschitz augmentation [43] allows to replace the product of

Original: Lipschitz constants by empirical equivalents, i.e., norms of activations and norms of Jacobians.
Revised: Lipschitz constants by empirical equivalents, i.e., norms of activations and norms of Jacobians.

Original: C.6 Calculations
Revised: C.6 Calculations

Original: This section contains postponed calculations.
Revised: This section contains postponed calculations.

Original: Lemma C.24 (used in Theorem C.5 and Theorem C.7) .
Revised: Lemma C.24 (used in Theorem C.5 and Theorem C.7) .

Original: For any n ∈ N , it holds that
Revised: For any n ∈ N , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: To prove the first inequality, note that
Revised: To prove the first inequality, note that

Original: [Equation]
Revised: [Equation]

Original: Similarly,
Revised: Similarly,

Original: [Equation]
Revised: [Equation]

Original: Lemma C.25 (used in Theorem C.19) .
Revised: Lemma C.25 (used in Theorem C.19) .

Original: For any α > 0 , it holds that
Revised: For any α > 0 , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The function
Revised: The function

Original: [Equation]
Revised: [Equation]

Original: is piecewise constant and so its integral is defined as
Revised: is piecewise constant and so its integral is defined as

Original: [Equation]
Revised: [Equation]

Original: For any M ∈ N , the partial sums are
Revised: For any M ∈ N , the partial sums are

Original: [Equation]
Revised: [Equation]

Original: Since lim M →∞ log(1+ αM ) √ M = 0 for every α > 0 , we conclude
Revised: Since lim M →∞ log(1+ αM ) √ M = 0 for every α > 0 , we conclude

Original: [Equation]
Revised: [Equation]

Original: Since 1 1 /α
Revised: Since 1 1 /α

Original: [Equation]
Revised: [Equation]

Original: m ∈ (0 , 1) for any α > 0 and m ∈ N , the logarithm is given by the Mercator series
Revised: m ∈ (0 , 1) for any α > 0 and m ∈ N , the logarithm is given by the Mercator series

Original: [Equation]
Revised: [Equation]

Original: Inserting this into the series from above and exchanging the order of summation, we get
Revised: Inserting this into the series from above and exchanging the order of summation, we get

Original: [Equation]
Revised: [Equation]

Original: This is an alternating convergent series in k , so its first summand ∞
Revised: This is an alternating convergent series in k , so its first summand ∞

Original: [Equation]
Revised: [Equation]

Original: (using Hölder inequality) already provides an upper bound, i.e.,
Revised: (using Hölder inequality) already provides an upper bound, i.e.,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]