Original: On Measuring Excess Capacity in Neural Networks
Revised: On Measuring Excess Capacity in Neural Networks

Original: Abstract
Revised: Abstract

Original: We study the excess capacity of deep networks in the context of supervised classification.
Revised: We study the excess capacity of deep networks in the context of supervised classification.

Original: That is, given a capacity measure of the underlying hypothesis class – in our case, empirical Rademacher complexity – to what extent can we (a priori) constrain this class while retaining an empirical error on a par with the unconstrained regime?
Revised: That is, given a capacity measure of the underlying hypothesis class – in our case, empirical Rademacher complexity – to what extent can we (a priori) constrain this class while retaining an empirical error on a par with the unconstrained regime?

Original: To assess excess capacity in modern architectures (such as residual networks ), we extend and unify prior Rademacher complexity bounds to accommodate function composition and addition , as well as the structure of convolutions.
Revised: To assess excess capacity in modern architectures (such as residual networks ), we extend and unify prior Rademacher complexity bounds to accommodate function composition and addition , as well as the structure of convolutions.

Original: The capacitydriving terms in our bounds are the Lipschitz constants of the layers and a (2,1) group norm distance to the initializations of the convolution weights.
Revised: The capacitydriving terms in our bounds are the Lipschitz constants of the layers and a (2,1) group norm distance to the initializations of the convolution weights.

Original: Experiments on benchmark datasets of varying task difficulty indicate that (1) there is a substantial amount of excess capacity per task, and (2) capacity can be kept at a surprisingly similar level across tasks.
Revised: Experiments on benchmark datasets of varying task difficulty indicate that (1) there is a substantial amount of excess capacity per task, and (2) capacity can be kept at a surprisingly similar level across tasks.

Original: Overall, this suggests a notion of compressibility with respect to weight norms, complementary to classic compression via weight pruning.
Revised: Overall, this suggests a notion of compressibility with respect to weight norms, complementary to classic compression via weight pruning.

Original: Source code is available at https://github.com/rkwitt/excess_capacity .
Revised: Source code is available at https://github.com/rkwitt/excess_capacity .

Original:  Introduction
Revised:  Introduction

Original: Understanding the generalization behavior of deep networks in supervised classification is still a largely open problem, despite a long history of theoretical advances.
Revised: Understanding the generalization behavior of deep networks in supervised classification is still a largely open problem, despite a long history of theoretical advances.

Original: The observation that (overparametrized) models can easily fit—i e., reach zero training error—to randomly permuted training labels [45, 46] but, when trained on unpermuted labels, yield good generalization performance, has fueled much of the progress in this area.
Revised: The observation that (overparametrized) models can easily fit—i e., reach zero training error—to randomly permuted training labels [45, 46] but, when trained on unpermuted labels, yield good generalization performance, has fueled much of the progress in this area.

Original: Recent works range from relating generalization to weight norms [3, 14, 26, 37, 38], measures of the distance to initializations [36], implicit regularization induced by the optimization algorithm [8, 41], or model compression [2, 6, 42].
Revised: Recent works range from relating generalization to weight norms [3, 14, 26, 37, 38], measures of the distance to initializations [34], implicit regularization induced by the optimization algorithm [8, 41], or model compression [2, 6, 42].

Original: Other works study connections to optimal transport [9], or generalization in the neural tangent kernel setting [1, 23].
Revised: Other works study connections to optimal transport [9], or generalization in the neural tangent kernel setting [1, 23].

Original: When seeking to establish generalization guarantees within the classic uniform convergence regime, bounding a capacity measure, such as the Rademacher complexity [4], of the hypothesis class is the crucial step.
Revised: When seeking to establish generalization guarantees within the classic uniform convergence regime, bounding a capacity measure, such as the Rademacher complexity [4], of the hypothesis class is the crucial step.

Original: While the resultant generalization bounds are typically vacuous and can exhibit concerning behavior [35], the capacity bounds themselves offer invaluable insights through the behavior of the bound-driving quantities, such as various types of weight norms or Lipschitz constants.
Revised: While the resultant generalization bounds are typically vacuous and can exhibit concerning behavior [36], the capacity bounds themselves offer invaluable insights through the behavior of the bound-driving quantities, such as various types of weight norms or Lipschitz constants.

Original: Particularly relevant to our work is the observation that the bound-driving quantities tend to increase with task difficulty .
Revised: Particularly relevant to our work is the observation that the bound-driving quantities tend to increase with task difficulty .

Original: Fig.
Revised: Fig.

Original: 1 illustrates this behavior in terms of Lipschitz constants per layer and the distance of each layer’s weight to its initialization (measured via a group norm we develop in Section 3.1).
Revised: 1 illustrates this behavior in terms of Lipschitz constants per layer and the distance of each layer’s weight to its initialization (measured via a group norm we develop in Section 3.1).

Original: [Figure]
Revised: [Figure]

Original: This raises two immediate questions: First ( Q1 ), can a network maintain empirical testing performance at substantially lower capacity?
Revised: This raises two immediate questions: First ( Q1 ), can a network maintain empirical testing performance at substantially lower capacity?

Original: Second ( Q2 ), is any imposed capacity reduction inevitably tied to task difficulty, i.e., would a reduced-capacity model for an easy task fail on a difficult task?
Revised: Second ( Q2 ), is any imposed capacity reduction inevitably tied to task difficulty, i.e., would a reduced-capacity model for an easy task fail on a difficult task?

Original: Both of these questions aim at the amount of “unneeded” capacity, which we refer to as excess capacity in the remainder of this work.
Revised: Both of these questions aim at the amount of “unneeded” capacity, which we refer to as excess capacity in the remainder of this work.

Original: We will address questions ( Q1 ) and ( Q2 ) by means of controlling the empirical Rademacher complexity of a neural network.
Revised: We will address questions ( Q1 ) and ( Q2 ) by means of controlling the empirical Rademacher complexity of a neural network.

Original: To this end, we consolidate and extend prior results from the literature on Rademacher complexity bounds to accommodate a broad range of network components in a unified way, including convolutions and skip connections, two ubiquitous elements in state-of-the-art models.
Revised: To this end, we consolidate and extend prior results from the literature on Rademacher complexity bounds to accommodate a broad range of network components in a unified way, including convolutions and skip connections, two ubiquitous elements in state-of-the-art models.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: We establish two bounds (in Theorem 3.5) for the empirical Rademacher complexity of neural networks that use convolutions and implement functions built from composition and addition .
Revised: We establish two bounds (in Theorem 3.5) for the empirical Rademacher complexity of neural networks that use convolutions and implement functions built from composition and addition .

Original: Specifically, we introduce two novel, convolution-specific, single-layer covering number bounds in Section 3.2 and contrast them to prior art, then modularize the single-layer to multi-layer covering approach of Bartlett et al.
Revised: Specifically, we introduce two novel, convolution-specific, single-layer covering number bounds in Section 3.2 and contrast them to prior art, then modularize the single-layer to multi-layer covering approach of Bartlett et al.

Original: [3] in Section 3.3, and eventually present one incarnation of our framework for convolutional residual networks in Sections 3.4 and 3.5. 2.
Revised: [3] in Section 3.3, and eventually present one incarnation of our framework for convolutional residual networks in Sections 3.4 and 3.5. 2.

Original: We present an extensive set of experiments (in Section 4) with a ResNet18 model across benchmark datasets of varying task difficulty, demonstrating that model capacity, when measured via our weight norm based bound, (1) can be kept surprisingly small per task, and (2) can be kept at roughly the same level regardless of task difficulty.
Revised: We present an extensive set of experiments (in Section 4) with a ResNet18 model across benchmark datasets of varying task difficulty, demonstrating that model capacity, when measured via our weight norm based bound, (1) can be kept surprisingly small per task, and (2) can be kept at roughly the same level regardless of task difficulty.

Original: Both observations suggest compressibility of neural networks with respect to weight norms,
Revised: Both observations suggest compressibility of neural networks with respect to weight norms,

Original: complementary to the well-known compressibility property of neural networks with respect to the number of parameters [2, 42].
Revised: complementary to the well-known compressibility property of neural networks with respect to the number of parameters [2, 42].

Original:  Related Work
Revised:  Related Work

Original: Many prior works establish uniform-convergence type generalization bounds for neural networks through Rademacher complexity analysis.
Revised: Many prior works establish uniform-convergence type generalization bounds for neural networks through Rademacher complexity analysis.

Original: We review such approaches, highlighting challenges that arise with modern network architectures and the peculiarities of convolutional layers.
Revised: We review such approaches, highlighting challenges that arise with modern network architectures and the peculiarities of convolutional layers.

Original: One direct approach to bound the empirical Rademacher complexity is via a layer-peeling strategy [14, 38, 44] where the Rademacher complexity of L -layer networks is expressed by a factor times the
Revised: One direct approach to bound the empirical Rademacher complexity is via a layer-peeling strategy [14, 38, 44] where the Rademacher complexity of L -layer networks is expressed by a factor times the

Original: Rademacher complexity of ( L − 1) -layer networks; in other words, the last layer is peeled off.
Revised: Rademacher complexity of ( L − 1) -layer networks; in other words, the last layer is peeled off.

Original: This factor is typically a matrix l p,q group norm, and thus the bounds usually scale with the product of the latter.
Revised: This factor is typically a matrix l p,q group norm, and thus the bounds usually scale with the product of the latter.

Original: Notably, the nonlinearities need to be element-wise operations, and some approaches only work for specific nonlinearities, such as ReLUs or asymmetric activations.
Revised: Notably, the nonlinearities need to be element-wise operations, and some approaches only work for specific nonlinearities, such as ReLUs or asymmetric activations.

Original: A second strategy is to bound the empirical Rademacher complexity via a covering numbers approach [3, 26, 29, 47], typically achieved via Dudley’s entropy integral [11].
Revised: A second strategy is to bound the empirical Rademacher complexity via a covering numbers approach [3, 26, 29, 47], typically achieved via Dudley’s entropy integral [11].

Original: This strategy is particularly flexible as it allows for arbitrary (but fixed) nonlinearities and for various paths to bounding covering numbers of network parts, e.g., via Maurey’s sparsification lemma, or via parameter counting.
Revised: This strategy is particularly flexible as it allows for arbitrary (but fixed) nonlinearities and for various paths to bounding covering numbers of network parts, e.g., via Maurey’s sparsification lemma, or via parameter counting.

Original: The corresponding whole-network bounds typically scale with the product of each layer’s Lipschitz constant or local empirical estimates thereof [43].
Revised: The corresponding whole-network bounds typically scale with the product of each layer’s Lipschitz constant or local empirical estimates thereof [43].

Original: Irrespective of the particular proof strategy, most formal arguments only hold for neural networks constructed from function composition, i.e., maps of the form
Revised: Irrespective of the particular proof strategy, most formal arguments only hold for neural networks constructed from function composition, i.e., maps of the form

Original: [Equation]
Revised: [Equation]

Original: where σ i : R d i − 1 → R d i are nonlinearities and A
Revised: where σ i : R d i − 1 → R d i are nonlinearities and A

Original: i are weight matrices specifying the i -th linear map.
Revised: i are weight matrices specifying the i -th linear map.

Original: However, modern architectures often rely on operations specifically tailored to the data, such as convolutions, and typically incorporate skip connections as in residual networks [20], rendering many results inapplicable or suboptimal for such models.
Revised: However, modern architectures often rely on operations specifically tailored to the data, such as convolutions, and typically incorporate skip connections as in residual networks [20], rendering many results inapplicable or suboptimal for such models.

Original: In this work, we handle convolutions and skip connections, thus increasing the applicability and utility of such bounds.
Revised: In this work, we handle convolutions and skip connections, thus increasing the applicability and utility of such bounds.

Original: For example, while residual networks have been studied extensively, theory mostly focuses on expressivity or optimization aspects [3, 18, 28, 44].
Revised: For example, while residual networks have been studied extensively, theory mostly focuses on expressivity or optimization aspects [3, 18, 28, 44].

Original: Yun et al.
Revised: Yun et al.

Original: [44] provide a Rademacher complexity bound via layer-peeling for fully-connected layers and element-wise activations.
Revised: [44] provide a Rademacher complexity bound via layer-peeling for fully-connected layers and element-wise activations.

Original: He et al.
Revised: He et al.

Original: [Figure]
Revised: [Figure]

Original: establish a generalization bound for residual networks via covering number arguments, resting upon earlier work by Bartlett et al.
Revised: establish a generalization bound for residual networks via covering number arguments, resting upon earlier work by Bartlett et al.

Original: [3] for linear maps.
Revised: [3] for linear maps.

Original: However, when directly applied to convolutions, both bounds scale unfavorably w.r
Revised: However, when directly applied to convolutions, both bounds scale unfavorably w.r

Original: [Figure]
Revised: [Figure]

Original: t. the spatial input size (see Section 3.2).
Revised: t. the spatial input size (see Section 3.2).

Original: Other works provide generalization guarantees specifically tailored to convolutional networks, cf.
Revised: Other works provide generalization guarantees specifically tailored to convolutional networks, cf.

Original: [Figure]
Revised: [Figure]

Original: and, although such bounds scale benignly with input size, they only apply for models as in Eq.
Revised: and, although such bounds scale benignly with input size, they only apply for models as in Eq.

Original: An initial numerical comparison.
Revised: An initial numerical comparison.

Original: Bounds on the empirical Rademacher complexity differ in their dependence on various quantities, such as matrix l p,q group norms, Lipschitz constants, or the number of parameters.
Revised: Bounds on the empirical Rademacher complexity differ in their dependence on various quantities, such as matrix l p,q group norms, Lipschitz constants, or the number of parameters.

Original: Thus, a precise formal comparison is challenging and, depending on the setting, different bounds may be preferable.
Revised: Thus, a precise formal comparison is challenging and, depending on the setting, different bounds may be preferable.

Original: To provide some intuition about magnitude differences, we evaluated several existing bounds (including ours from Section 3.5) on two convolutional (ReLU) networks with 6 and 11 layers, see Fig.
Revised: To provide some intuition about magnitude differences, we evaluated several existing bounds (including ours from Section 3.5) on two convolutional (ReLU) networks with 6 and 11 layers, see Fig.

Original: 2 and Section A.2 for details.
Revised: 2 and Section A.2 for details.

Original:  Rademacher Complexity Analysis
Revised:  Rademacher Complexity Analysis

Original: To derive bounds on the empirical Rademacher complexity, we follow the margin-based multiclass learning formalism and the flexible proof strategy of Bartlett et al.
Revised: To derive bounds on the empirical Rademacher complexity, we follow the margin-based multiclass learning formalism and the flexible proof strategy of Bartlett et al.

Original: Section 3.2 introduces novel single-layer covering number bounds for convolutions.
Revised: Section 3.2 introduces novel single-layer covering number bounds for convolutions.

Original: Section 3.3 modularizes and extends the single- to multi-layer covering step to account for architectures such as residual networks (Section 3.4).
Revised: Section 3.3 modularizes and extends the single- to multi-layer covering step to account for architectures such as residual networks (Section 3.4).

Original: Last, Section 3.5 presents and discusses our Rademacher complexity bounds.
Revised: Last, Section 3.5 presents and discusses our Rademacher complexity bounds.

Original: 3.1 Preliminaries
Revised: 3.1 Preliminaries

Original: In a κ -class classification task, we are given n instance/label pairs S =
Revised: In a κ -class classification task, we are given n instance/label pairs S =

Original: (( x 1 , y 1 ) , . . .
Revised: (( x 1 , y 1 ) , . . .

Original: , ( x n , y n )) , drawn iid from a probability measure D on R d ×{ 1 , . .
Revised: , ( x n , y n )) , drawn iid from a probability measure D on R d ×{ 1 , . .

Original: . , κ } .
Revised: . , κ } .

Original: For a neural network f in a hypothesis class F ⊂ { f : R d → R κ } , a class label for input x is obtained by taking the argmax over the components of f ( x ) ∈ R κ .
Revised: For a neural network f in a hypothesis class F ⊂ { f : R d → R κ } , a class label for input x is obtained by taking the argmax over the components of f ( x ) ∈ R κ .

Original: The margin operator M : R κ ×{ 1 , . . .
Revised: The margin operator M : R κ ×{ 1 , . . .

Original: , κ } → R , ( v, y ) (cid:55)→ v y − max i ̸
Revised: , κ } → R , ( v, y ) (cid:55)→ v y − max i ̸

Original: = y
Revised: = y

Original: v i leads, with margin γ > 0 , to the ramp loss ℓ γ and the empirical ramp risk
Revised: v i leads, with margin γ > 0 , to the ramp loss ℓ γ and the empirical ramp risk

Original: ˆ R γ , defined as ℓ
Revised: ˆ R γ , defined as ℓ

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: To derive generalization bounds via classical Rademacher complexity analysis [33], without having to resort to vector-contraction inequalities [32], we consider the hypothesis class
Revised: To derive generalization bounds via classical Rademacher complexity analysis [33], without having to resort to vector-contraction inequalities [32], we consider the hypothesis class

Original: [Equation]
Revised: [Equation]

Original: Then, defining the empirical Rademacher complexity of any class H of real-valued functions as
Revised: Then, defining the empirical Rademacher complexity of any class H of real-valued functions as

Original: [Equation]
Revised: [Equation]

Original: with iid Rademacher variables σ = ( σ 1 , . . .
Revised: with iid Rademacher variables σ = ( σ 1 , . . .

Original: , σ n ) from a uniform distribution on {± 1 } , facilitates to study F γ via Eq.
Revised: , σ n ) from a uniform distribution on {± 1 } , facilitates to study F γ via Eq.

Original: The following lemma [3, Lemma 3.1] establishes the link to a margin-based multiclass generalization bound for any f ∈ F .
Revised: The following lemma [3, Lemma 3.1] establishes the link to a margin-based multiclass generalization bound for any f ∈ F .

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: Given a hypothesis class F of functions f : R d → R κ and classification margin γ > 0 , then, with probability of at least 1 − δ over the choice of S ∼ D n , for any f ∈ F , it holds that
Revised: Given a hypothesis class F of functions f : R d → R κ and classification margin γ > 0 , then, with probability of at least 1 − δ over the choice of S ∼ D n , for any f ∈ F , it holds that

Original: [Equation]
Revised: [Equation]

Original: To obtain a computable expression for the right-hand side of Lemma 3.1, we seek a bound on ˆ R S ( F γ ) tied to some measurable quantities of the network realizing F .
Revised: To obtain a computable expression for the right-hand side of Lemma 3.1, we seek a bound on ˆ R S ( F γ ) tied to some measurable quantities of the network realizing F .

Original: For our purposes, the relationship of ˆ R S ( F γ ) and the covering number of F γ turns out to be a flexible approach.
Revised: For our purposes, the relationship of ˆ R S ( F γ ) and the covering number of F γ turns out to be a flexible approach.

Original: In general, given a normed space ( G , ∥·∥ G ) , the covering number N ( G , ϵ, ∥·∥ G ) is the cardinality of the smallest ϵ -cover of G , i.e., of the smallest subset U ⊂ G such that, for any g ∈ G , there exists u ∈ U with ∥ g − u ∥ G ≤ ϵ .
Revised: In general, given a normed space ( G , ∥·∥ G ) , the covering number N ( G , ϵ, ∥·∥ G ) is the cardinality of the smallest ϵ -cover of G , i.e., of the smallest subset U ⊂ G such that, for any g ∈ G , there exists u ∈ U with ∥ g − u ∥ G ≤ ϵ .

Original: In our setting, G is a class of functions g : ( X , ∥·∥ X ) →
Revised: In our setting, G is a class of functions g : ( X , ∥·∥ X ) →

Original: ( Y , ∥·∥ Y ) between normed spaces X and Y .
Revised: ( Y , ∥·∥ Y ) between normed spaces X and Y .

Original: Given data X =
Revised: Given data X =

Original: ( x 1 , . .
Revised: ( x 1 , . .

Original: , x n ) ∈ X n , we define a data-dependent norm on G| X as
Revised: , x n ) ∈ X n , we define a data-dependent norm on G| X as

Original: [Equation]
Revised: [Equation]

Original: In other words, Eq.
Revised: In other words, Eq.

Original: (6) is the l 2 norm on the restriction of G to X .
Revised: (6) is the l 2 norm on the restriction of G to X .

Original: Specifically, we seek to bound log N ( F γ , ϵ, ∥·∥ S ) , as this facilitates to upper bound the empirical Rademacher complexity of F γ by means of Dudley’s entropy integral.
Revised: Specifically, we seek to bound log N ( F γ , ϵ, ∥·∥ S ) , as this facilitates to upper bound the empirical Rademacher complexity of F γ by means of Dudley’s entropy integral.

Original: Typically, such covering numbers depend on the norm of the data itself,
Revised: Typically, such covering numbers depend on the norm of the data itself,

Original: i.e., ∥ X ∥ = (cid:112) ∥
Revised: i.e., ∥ X ∥ = (cid:112) ∥

Original: x 1 ∥ 2 X + · · · ∥ x n ∥ 2 X .
Revised: x 1 ∥ 2 X + · · · ∥ x n ∥ 2 X .

Original: 2 Covering number bounds for convolutions
Revised: 2 Covering number bounds for convolutions

Original: We consider 2D convolutions, acting on images with c in channels of width w and height h , i.e., x ∈ R c in × h × w .
Revised: We consider 2D convolutions, acting on images with c in channels of width w and height h , i.e., x ∈ R c in × h × w .

Original: For readability only, we discuss convolutions of stride 1 and input-size preserving padding; this is not an assumption required for Theorem 3.2.
Revised: For readability only, we discuss convolutions of stride 1 and input-size preserving padding; this is not an assumption required for Theorem 3.2.

Original: Formally, a convolutional layer is a linear map ϕ K : R c in × h × w → R c out × h × w (as we omit bias parameters), where c in and c out denote the number of input and output channels.
Revised: Formally, a convolutional layer is a linear map ϕ K : R c in × h × w → R c out × h × w (as we omit bias parameters), where c in and c out denote the number of input and output channels.

Original: The map is parametrized by a tensor K ∈ R c out × c in ×
Revised: The map is parametrized by a tensor K ∈ R c out × c in ×

Original: k h × k w of spatial extension/kernel size ( k h , k w ) .
Revised: k h × k w of spatial extension/kernel size ( k h , k w ) .

Original: Since convolutions are linear maps, they can be specified by matrices which act on the (reshaped) inputs and one could invoke existing covering number bounds.
Revised: Since convolutions are linear maps, they can be specified by matrices which act on the (reshaped) inputs and one could invoke existing covering number bounds.

Original: However, this is suboptimal, as any structure specific to convolutions is ignored.
Revised: However, this is suboptimal, as any structure specific to convolutions is ignored.

Original: In particular, normbased generalization bounds agnostic to this structure incur unfavorable scaling behavior w.r.t.
Revised: In particular, normbased generalization bounds agnostic to this structure incur unfavorable scaling behavior w.r.t.

Original: the dimensionality of the input.
Revised: the dimensionality of the input.

Original: To be more specific, the weight tensor K of a convolutional layer does not directly specify the corresponding matrix; instead, it parametrizes c out filters, i.e., local linear maps, which are applied to the ( c in ×
Revised: To be more specific, the weight tensor K of a convolutional layer does not directly specify the corresponding matrix; instead, it parametrizes c out filters, i.e., local linear maps, which are applied to the ( c in ×

Original: k h × k w ) -sized pixel neighborhoods of the input.
Revised: k h × k w ) -sized pixel neighborhoods of the input.

Original: Hence, the matrix M K corresponding to the global linear map consists of many copies of the elements of this tensor, one for each of the hw patches the filters are applied to.
Revised: Hence, the matrix M K corresponding to the global linear map consists of many copies of the elements of this tensor, one for each of the hw patches the filters are applied to.

Original: Thus, the  p -norm of the matrix M K is ∥ M K ∥ p
Revised: Thus, the l p norm of the matrix M K is ∥ M K ∥ p

Original: = ( hw ) 1 /p ∥ K ∥ p
Revised: = ( hw ) 1 /p ∥ K ∥ p

Original: (see Section A.1).
Revised: (see Section A.1).

Original: We mitigate this scaling issue by tying the covering number of a convolutional layer to a variant of the (2,1) group norm on the tensor K .
Revised: We mitigate this scaling issue by tying the covering number of a convolutional layer to a variant of the (2,1) group norm on the tensor K .

Original: We define this norm as the sum over the l 2 norms taken along the input channels of K , i.e.,
Revised: We define this norm as the sum over the l 2 norms taken along the input channels of K , i.e.,

Original: [Equation]
Revised: [Equation]

Original: For the special case of inputs of size ( h, w ) = (1 , 1) and kernel size ( k h , k w ) =
Revised: For the special case of inputs of size ( h, w ) = (1 , 1) and kernel size ( k h , k w ) =

Original: (1 , 1) , convolution is just matrix multiplication along the channels.
Revised: (1 , 1) , convolution is just matrix multiplication along the channels.

Original: In this case, M K = K ·· 11 and our norm from
Revised: In this case, M K = K ·· 11 and our norm from

Original: Eq.
Revised: Eq.

Original: (7) agrees with the standard (2 , 1) group norm on M ⊤ K , i.e., ∥ K ∥ 2 , 1 = (cid:13)(cid:13)
Revised: (7) agrees with the standard (2 , 1) group norm on M ⊤ K , i.e., ∥ K ∥ 2 , 1 = (cid:13)(cid:13)

Original: M ⊤ K (cid:13)(cid:13) 2 , 1 .
Revised: M ⊤ K (cid:13)(cid:13) 2 , 1 .

Original: Theorem 3.
Revised: Theorem 3.

Original: establishes two covering number bounds for convolutions.
Revised: establishes two covering number bounds for convolutions.

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: Let b > 0 and F = { ϕ K | K ∈ R c out × c in × k h × k w , ∥ K ∥ 2 , 1 ≤ b } denote the class of 2D convolutions with c in input channels, c out output channels and kernel size k h × k w , parametrized by tensors K with W parameters.
Revised: Let b > 0 and F = { ϕ K | K ∈ R c out × c in × k h × k w , ∥ K ∥ 2 , 1 ≤ b } denote the class of 2D convolutions with c in input channels, c out output channels and kernel size k h × k w , parametrized by tensors K with W parameters.

Original: Then, for any X ∈ R n × c in × h × w and covering radius ϵ > 0 ,
Revised: Then, for any X ∈ R n × c in × h × w and covering radius ϵ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (8a) is analogous to the single-layer bound of Bartlett et al.
Revised: (8a) is analogous to the single-layer bound of Bartlett et al.

Original: [3, Lemma 3.2] for fully-connected layers, but replaces the (2 , 1) group norm constraint on matrices M with a constraint on tensors
Revised: [3, Lemma 3.2] for fully-connected layers, but replaces the (2 , 1) group norm constraint on matrices M with a constraint on tensors

Original: K .
Revised: K .

Original: This is tighter than invoking [3, Lemma 3.2] directly on M K ∈ R c in hw × c out hw , as K has only c out c in k h k w parameters and (cid:13)(cid:13)
Revised: This is tighter than invoking [3, Lemma 3.2] directly on M K ∈ R c in hw × c out hw , as K has only c out c in k h k w parameters and (cid:13)(cid:13)

Original: M ⊤ K (cid:13)(cid:13) 2 , 1 ≥ hw / √ k h k w ∥
Revised: M ⊤ K (cid:13)(cid:13) 2 , 1 ≥ hw / √ k h k w ∥

Original: K
Revised: K

Original: ∥ 2 , 1 , see Section A.1.
Revised: ∥ 2 , 1 , see Section A.1.

Original: A thorough comparison between the two bounds in Theorem 3.2 is nuanced, though, as preferring one over the other depends on the ratio between the number of parameters W and ∥ X ∥
Revised: A thorough comparison between the two bounds in Theorem 3.2 is nuanced, though, as preferring one over the other depends on the ratio between the number of parameters W and ∥ X ∥

Original: 2 b 2 /ϵ 2 .
Revised: 2 b 2 /ϵ 2 .

Original: The latter, in turn, requires to consider all covering radii ϵ .
Revised: The latter, in turn, requires to consider all covering radii ϵ .

Original: Hence, we defer this discussion to Section 3.5, where differences manifest more clearly in the overall empirical Rademacher complexity bounds.
Revised: Hence, we defer this discussion to Section 3.5, where differences manifest more clearly in the overall empirical Rademacher complexity bounds.

Original: Proof sketch.
Revised: Proof sketch.

Original: The statement of Theorem 3.2 follows from an application of Maurey’s sparsification lemma, which guarantees the existence of an ϵ -cover of F (of known cardinality) if there is a finite subset { V 1 , . .
Revised: The statement of Theorem 3.2 follows from an application of Maurey’s sparsification lemma, which guarantees the existence of an ϵ -cover of F (of known cardinality) if there is a finite subset { V 1 , . .

Original: . , V d } ⊂ F s.t.
Revised: . , V d } ⊂ F s.t.

Original: every f ∈ F is a convex combination of the V i .
Revised: every f ∈ F is a convex combination of the V i .

Original: We show that one can find such a finite subset of cardinality d = 2 c in c out k h k w = 2 W .
Revised: We show that one can find such a finite subset of cardinality d = 2 c in c out k h k w = 2 W .

Original: The cardinality of the cover is then determined by a combinatorial quantity which additionally depends on ∥ X ∥ and the norm constraint b .
Revised: The cardinality of the cover is then determined by a combinatorial quantity which additionally depends on ∥ X ∥ and the norm constraint b .

Original: Bounding this quantity, i.e., a binomial coefficient, in two different ways, establishes the bounds.
Revised: Bounding this quantity, i.e., a binomial coefficient, in two different ways, establishes the bounds.

Original: Relation to prior work.
Revised: Relation to prior work.

Original: Closely related is recent work by Ledent et al.
Revised: Closely related is recent work by Ledent et al.

Original: [26] who derive l ∞ covering number bounds for convolutional layers based on a classic result by Zhang [47].
Revised: [26] who derive l ∞ covering number bounds for convolutional layers based on a classic result by Zhang [47].

Original: Similar to Eq.
Revised: Similar to Eq.

Original: (8a), their bound depends on the square of a weight norm directly on the tensor K , the square of a data norm, as well as a logarithmic term.
Revised: (8a), their bound depends on the square of a weight norm directly on the tensor K , the square of a data norm, as well as a logarithmic term.

Original: The data norm is the maximal l 2 norm of a single patch.
Revised: The data norm is the maximal l 2 norm of a single patch.

Original: Compared to our result, this implicitly removes a factor of the spatial dimension hw , which, however, reenters via the Lipschitz constant when transitioning to multi-layer bounds in the form of the spatial dimension of the output (after subsequent pooling).
Revised: Compared to our result, this implicitly removes a factor of the spatial dimension hw , which, however, reenters via the Lipschitz constant when transitioning to multi-layer bounds in the form of the spatial dimension of the output (after subsequent pooling).

Original: Overall, the quadratic terms across both results scale similarly (with our data norm being less sensitive to outliers), but we improve on Ledent et al.
Revised: Overall, the quadratic terms across both results scale similarly (with our data norm being less sensitive to outliers), but we improve on Ledent et al.

Original: [26] in the logarithmic term.
Revised: [26] in the logarithmic term.

Original: By contrast, the use of l ∞ covers in [26] yields whole-network bounds with improved dependency on the number of classes; see Section A.6 for an in-depth comparison.
Revised: By contrast, the use of l ∞ covers in [26] yields whole-network bounds with improved dependency on the number of classes; see Section A.6 for an in-depth comparison.

Original: In other related work, Lin et al.
Revised: In other related work, Lin et al.

Original: [29] derive an l 2 covering number bound for convolutional layers similar to Eq.
Revised: [29] derive an l 2 covering number bound for convolutional layers similar to Eq.

Original: (8b), which depends linearly on the number of parameters and logarithmically on norms.
Revised: (8b), which depends linearly on the number of parameters and logarithmically on norms.

Original: In their proof, Lin et al.
Revised: In their proof, Lin et al.

Original: [29] show that every cover of a convolutional layer’s weight space (a subset of a Euclidean space) induces a cover of the corresponding function space w.r.t.
Revised: [29] show that every cover of a convolutional layer’s weight space (a subset of a Euclidean space) induces a cover of the corresponding function space w.r.t.

Original: the data dependent norm defined in Eq.
Revised: the data dependent norm defined in Eq.

Original: However, their approach incurs an additional factor inside the logarithm that corresponds to the number of how often each filter is applied, i.e., the spatial dimension of the output.
Revised: However, their approach incurs an additional factor inside the logarithm that corresponds to the number of how often each filter is applied, i.e., the spatial dimension of the output.

Original: Importantly, non-convolution specific approaches can equally mitigate undesirable scaling issues, e.g., by utilizing (1 , ∞ ) group norms on the matrices representing the linear map [14, 16, 38]; as differences primarily manifest in the resulting bounds on the Rademacher complexity, we refer to our discussion in Section 3.5.
Revised: Importantly, non-convolution specific approaches can equally mitigate undesirable scaling issues, e.g., by utilizing (1 , ∞ ) group norms on the matrices representing the linear map [14, 16, 38]; as differences primarily manifest in the resulting bounds on the Rademacher complexity, we refer to our discussion in Section 3.5.

Original: 3.3 Covering number bounds for composition & addition
Revised: 3.3 Covering number bounds for composition & addition

Original: As many neural networks are built from composition and summation of layers, we study covering numbers under these operations.
Revised: As many neural networks are built from composition and summation of layers, we study covering numbers under these operations.

Original: The key building blocks are the following, easy to verify, inequalities.
Revised: The key building blocks are the following, easy to verify, inequalities.

Original: Lemma 3.3.
Revised: Lemma 3.3.

Original: Let F 1 , F 2 be classes of functions on normed spaces ( X , ∥·∥ X )
Revised: Let F 1 , F 2 be classes of functions on normed spaces ( X , ∥·∥ X )

Original: → ( Y , ∥·∥ Y ) and let G be a class of c -Lipschitz functions ( Y , ∥·∥ Y )
Revised: → ( Y , ∥·∥ Y ) and let G be a class of c -Lipschitz functions ( Y , ∥·∥ Y )

Original: → ( Z , ∥·∥ Z ) .
Revised: → ( Z , ∥·∥ Z ) .

Original: Then, for any X ∈ X n and ϵ F 1 , ϵ F 2 , ϵ G > 0 , it holds that
Revised: Then, for any X ∈ X n and ϵ F 1 , ϵ F 2 , ϵ G > 0 , it holds that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: To establish these inequalities, one chooses minimal covers of the original function spaces and links their elements via the considered operation, i.e., addition or composition.
Revised: To establish these inequalities, one chooses minimal covers of the original function spaces and links their elements via the considered operation, i.e., addition or composition.

Original: The resulting functions correspond to tuples of elements of the original covers.
Revised: The resulting functions correspond to tuples of elements of the original covers.

Original: Hence, the right-hand side of the inequalities is a product of covering numbers.
Revised: Hence, the right-hand side of the inequalities is a product of covering numbers.

Original: The crucial step is to determine a preferably small radius ϵ such that these functions form an ϵ -cover.
Revised: The crucial step is to determine a preferably small radius ϵ such that these functions form an ϵ -cover.

Original: In Lemma 3.3, this is achieved via standard properties of norms.
Revised: In Lemma 3.3, this is achieved via standard properties of norms.

Original: Notably, iterative application of Lemma 3.3 allows bounding the covering numbers of any function class built from compositions and additions of simpler classes.
Revised: Notably, iterative application of Lemma 3.3 allows bounding the covering numbers of any function class built from compositions and additions of simpler classes.

Original: In Section C.3, we apply Lemma 3.3 on two examples, i.e., (1) f ∈ F = { f L ◦ · · · ◦ f 1 } and (2) h ∈ H
Revised: In Section C.3, we apply Lemma 3.3 on two examples, i.e., (1) f ∈ F = { f L ◦ · · · ◦ f 1 } and (2) h ∈ H

Original: = { g + h L ◦ · · · ◦ h 1 } .
Revised: = { g + h L ◦ · · · ◦ h 1 } .

Original: Instantiating the first example for f i = σ i ◦ ϕ
Revised: Instantiating the first example for f i = σ i ◦ ϕ

Original: i , with σ i fixed and ϕ i from a family of linear maps, yields covering number bounds for networks as in Eq.
Revised: i , with σ i fixed and ϕ i from a family of linear maps, yields covering number bounds for networks as in Eq.

Original: As the second example corresponds to residual blocks (with g possibly the identity map), the combination of (1) and (2) yields covering number bounds for residual networks; see Example C.3.
Revised: As the second example corresponds to residual blocks (with g possibly the identity map), the combination of (1) and (2) yields covering number bounds for residual networks; see Example C.3.

Original: Overall, this strategy not only allows to derive covering number bounds for a broad range of architectures, but also facilitates integrating linkings between function spaces in a modular manner.
Revised: Overall, this strategy not only allows to derive covering number bounds for a broad range of architectures, but also facilitates integrating linkings between function spaces in a modular manner.

Original: For instance, Lemma C.14 provides a variant of Lemma 3.3 for concatenation , used in DenseNets [22].
Revised: For instance, Lemma C.14 provides a variant of Lemma 3.3 for concatenation , used in DenseNets [22].

Original: Relation to prior work.
Revised: Relation to prior work.

Original: He et al.
Revised: He et al.

Original: [19] investigate covering number bounds for function spaces as considered above.
Revised: [19] investigate covering number bounds for function spaces as considered above.

Original: They present covering number bounds for residual networks and show that the covering number N ( F , ϵ, ∥·∥ X ) of such models with layers F α is bounded by the product (cid:81)
Revised: They present covering number bounds for residual networks and show that the covering number N ( F , ϵ, ∥·∥ X ) of such models with layers F α is bounded by the product (cid:81)

Original: α
Revised: α

Original: [Equation]
Revised: [Equation]

Original: N ( F α , ϵ F α , ∥·∥ ϕ ( X ) ) for appropriately defined function spaces G α .
Revised: N ( F α , ϵ F α , ∥·∥ ϕ ( X ) ) for appropriately defined function spaces G α .

Original: Yet, the dependency of the whole-network covering radius ϵ on the single-layer covering radii ϵ F α is only derived for a very specific residual network.
Revised: Yet, the dependency of the whole-network covering radius ϵ on the single-layer covering radii ϵ F α is only derived for a very specific residual network.

Original: Our addition to the theory is a more modular and structured way of approaching the problem, which we believe to be valuable on its own.
Revised: Our addition to the theory is a more modular and structured way of approaching the problem, which we believe to be valuable on its own.

Original: 3.4 Covering number bounds for residual networks
Revised: 3.4 Covering number bounds for residual networks

Original: We next state our whole-network covering number bounds for residual networks and then present the corresponding bounds on the empirical Rademacher complexity in Section 3.5.
Revised: We next state our whole-network covering number bounds for residual networks and then present the corresponding bounds on the empirical Rademacher complexity in Section 3.5.

Original: Accompanying generalization guarantees (obtained via Lemma 3.1) are given in Section C.5.
Revised: Accompanying generalization guarantees (obtained via Lemma 3.1) are given in Section C.5.

Original: The results of this section hold for a hypothesis class F of networks implementing functions of the form f
Revised: The results of this section hold for a hypothesis class F of networks implementing functions of the form f

Original: [Equation]
Revised: [Equation]

Original: with f i ( x
Revised: with f i ( x

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: (11) i.e., a composition of L residual blocks.
Revised: (11) i.e., a composition of L residual blocks.

Original: Here, the nonlinearities σ i and σ ij are fixed and ρ i -, resp., ρ ij -Lipschitz continuous with σ i (0) = 0 and σ ij (0)
Revised: Here, the nonlinearities σ i and σ ij are fixed and ρ i -, resp., ρ ij -Lipschitz continuous with σ i (0) = 0 and σ ij (0)

Original: = 0 .
Revised: = 0 .

Original: We further fix the shortcuts to maps with g i (0) = 0 .
Revised: We further fix the shortcuts to maps with g i (0) = 0 .

Original: The map h ij identifies the j -th layer in the i -th residual block with Lipschitz constraints s ij and distance constraints b ij (w.r.t.
Revised: The map h ij identifies the j -th layer in the i -th residual block with Lipschitz constraints s ij and distance constraints b ij (w.r.t.

Original: reference weights M ij ).
Revised: reference weights M ij ).

Original: Specifically, if h ij is fully-connected , then
Revised: Specifically, if h ij is fully-connected , then

Original: [Equation]
Revised: [Equation]

Original: and, in case h ij is convolutional , then
Revised: and, in case h ij is convolutional , then

Original: [Equation]
Revised: [Equation]

Original: In terms of notation, s i
Revised: In terms of notation, s i

Original: =
Revised: =

Original: Lip( g i ) + (cid:81)
Revised: Lip( g i ) + (cid:81)

Original: L i j =1 ρ ij s ij further denotes the upper bound on the Lipschitz constant of the i -th residual block f i .
Revised: L i j =1 ρ ij s ij further denotes the upper bound on the Lipschitz constant of the i -th residual block f i .

Original: The Lipschitz constants are w.r.t.
Revised: The Lipschitz constants are w.r.t.

Original: Euclidean norms; for a fully-connected layer this coincides with the spectral norm of the weight matrix.
Revised: Euclidean norms; for a fully-connected layer this coincides with the spectral norm of the weight matrix.

Original: The covering number bounds in Theorem 3.4 below depend on three types of quantities: (1) the total number of layers
Revised: The covering number bounds in Theorem 3.4 below depend on three types of quantities: (1) the total number of layers

Original: ¯ L = (cid:80)
Revised: ¯ L = (cid:80)

Original: i L i , (2) the numbers W ij of parameters of the j -th layer in the i -th residual block, their maximum W = max ij W ij , and (3) terms C ij that quantify the part of a layer’s capacity attributed to weight and data norms.
Revised: i L i , (2) the numbers W ij of parameters of the j -th layer in the i -th residual block, their maximum W = max ij W ij , and (3) terms C ij that quantify the part of a layer’s capacity attributed to weight and data norms.

Original: With respect to the latter, we define
Revised: With respect to the latter, we define

Original: [Equation]
Revised: [Equation]

Original: and write C ij =
Revised: and write C ij =

Original: C ij ( X ) for brevity.
Revised: C ij ( X ) for brevity.

Original: Importantly, ∥ X ∥ ≤ √ n max i ∥ x i ∥ and so the C ij can be bounded independently of the sample size.
Revised: Importantly, ∥ X ∥ ≤ √ n max i ∥ x i ∥ and so the C ij can be bounded independently of the sample size.

Original: Overall, this yields the following covering number bounds for residual networks.
Revised: Overall, this yields the following covering number bounds for residual networks.

Original: Theorem 3.4.
Revised: Theorem 3.4.

Original: The covering number of the class of residual networks F as specified above, satisfies
Revised: The covering number of the class of residual networks F as specified above, satisfies

Original: [Equation]
Revised: [Equation]

Original: 3.5 Rademacher complexity bounds
Revised: 3.5 Rademacher complexity bounds

Original: In combination with Dudley’s entropy integral, Theorem 3.4 implies the empirical Rademacher complexity bounds in Theorem 3.5.
Revised: In combination with Dudley’s entropy integral, Theorem 3.4 implies the empirical Rademacher complexity bounds in Theorem 3.5.

Original: These bounds equally hold for non-residual networks as in
Revised: These bounds equally hold for non-residual networks as in

Original: Eq.
Revised: Eq.

Original: (1), i.e., the special case of setting the shortcuts g i to the zero map (with L = 1 block).
Revised: (1), i.e., the special case of setting the shortcuts g i to the zero map (with L = 1 block).

Original: Theorem 3.5.
Revised: Theorem 3.5.

Original: Let γ > 0 and define ˜ C ij = 2 C ij /γ .
Revised: Let γ > 0 and define ˜ C ij = 2 C ij /γ .

Original: Further, let H n − 1 = (cid:80) n − 1 m =1 1 / m =
Revised: Further, let H n − 1 = (cid:80) n − 1 m =1 1 / m =

Original: O (log( n )) denote the ( n − 1)
Revised: O (log( n )) denote the ( n − 1)

Original: -th harmonic number.
Revised: -th harmonic number.

Original: Then, the empirical Rademacher complexity of F γ satisfies
Revised: Then, the empirical Rademacher complexity of F γ satisfies

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where ψ is a monotonically increasing function, satisfying ψ (0)
Revised: where ψ is a monotonically increasing function, satisfying ψ (0)

Original: = 

Revised: =

Original: and ∀ x : ψ ( x ) < 2 .
Revised: and ∀ x : ψ ( x ) < 2 .

Original: The theorem considers the function class F γ as defined in Eq.
Revised: The theorem considers the function class F γ as defined in Eq.

Original: As a consequence, the bounds depend on the quotients ˜ C ij = 2 C ij /γ , which measure a layer’s capacity (with respect to weight and data norms) relative to a classification margin parameter γ .
Revised: As a consequence, the bounds depend on the quotients ˜ C ij = 2 C ij /γ , which measure a layer’s capacity (with respect to weight and data norms) relative to a classification margin parameter γ .

Original: As we will see in the experiments, constraining the layers’ Lipschitz constants and weight norms, allows to substantially reduce the quantities C ij while the margin parameter γ decreases only moderately.
Revised: As we will see in the experiments, constraining the layers’ Lipschitz constants and weight norms, allows to substantially reduce the quantities C ij while the margin parameter γ decreases only moderately.

Original: Theorem 3.5 also immediately implies generalization bounds for F γ via Lemma 3.1.
Revised: Theorem 3.5 also immediately implies generalization bounds for F γ via Lemma 3.1.

Original: In a subsequent step one can gradually decrease the constraint strengths and invoke a union bound argument over the corresponding generalization bounds, as for example done in [3, Lemma A.9].
Revised: In a subsequent step one can gradually decrease the constraint strengths and invoke a union bound argument over the corresponding generalization bounds, as for example done in [3, Lemma A.9].

Original: This yields a generalization bound which does not depend on a priori defined constraint strengths, but on the actual
Revised: This yields a generalization bound which does not depend on a priori defined constraint strengths, but on the actual

Original: Lipschitz constants and group norms computed from a neural network’s weights.
Revised: Lipschitz constants and group norms computed from a neural network’s weights.

Original: Interpretation.
Revised: Interpretation.

Original: To facilitate a clean comparison between the bounds in Theorem 3.5, we disregard the ceiling function and apply Jensen’s inequality to the first bound ( ♣ ), yielding
Revised: To facilitate a clean comparison between the bounds in Theorem 3.5, we disregard the ceiling function and apply Jensen’s inequality to the first bound ( ♣ ), yielding

Original: [Equation]
Revised: [Equation]

Original: Denoting ˜ C = max ij ˜ C ij , Eq.
Revised: Denoting ˜ C = max ij ˜ C ij , Eq.

Original: (16) reveals that the bounds essentially differ only in that ( ♣ ) depends on (log(2 W )
Revised: (16) reveals that the bounds essentially differ only in that ( ♣ ) depends on (log(2 W )

Original: ¯ L 2 ˜ C 2 ) 1 / 2 and ( ♠ ) depends on (2 W log(1 + ¯ L 2 ˜ C 2 )) 1 / 2 .
Revised: ¯ L 2 ˜ C 2 ) 1 / 2 and ( ♠ ) depends on (2 W log(1 + ¯ L 2 ˜ C 2 )) 1 / 2 .

Original: Thus, the question of which one is tighter , hinges on the ratio of 2 W and ¯ L 2 ˜ C 2 , i.e., a tradeoff between the number of parameters and the weight norms.
Revised: Thus, the question of which one is tighter , hinges on the ratio of 2 W and ¯ L 2 ˜ C 2 , i.e., a tradeoff between the number of parameters and the weight norms.

Original: As we see in Fig.
Revised: As we see in Fig.

Original: 2, for simple, unconstrained networks, our second bound ( ♠ ) is much tighter.
Revised: 2, for simple, unconstrained networks, our second bound ( ♠ ) is much tighter.

Original: However, due to the logarithmic dependency on ˜ C , it is less affected by constraining the distance to initialization b and the Lipschitz constants s .
Revised: However, due to the logarithmic dependency on ˜ C , it is less affected by constraining the distance to initialization b and the Lipschitz constants s .

Original: In Section 4, we show that this effect causes ( ♣ ) to be a more faithful measure of excess capacity.
Revised: In Section 4, we show that this effect causes ( ♣ ) to be a more faithful measure of excess capacity.

Original: As ˜ C depends exponentially on the network depth via the product of Lipschitz constants, another perspective on the bounds is that
Revised: As ˜ C depends exponentially on the network depth via the product of Lipschitz constants, another perspective on the bounds is that

Original: Eq.
Revised: Eq.

Original: ( ♣ ) favors shallow architectures whereas Eq.
Revised: ( ♣ ) favors shallow architectures whereas Eq.

Original: ( ♠ ) favors narrow architectures.
Revised: ( ♠ ) favors narrow architectures.

Original: Notably, replacing the function class F γ with a class of networks composed with a Lipschitz augmented loss function [43] facilitates deriving Rademacher complexity- and generalization bounds, which do not suffer from the exponential depth dependency via the product of Lipschitz constants.
Revised: Notably, replacing the function class F γ with a class of networks composed with a Lipschitz augmented loss function [43] facilitates deriving Rademacher complexity- and generalization bounds, which do not suffer from the exponential depth dependency via the product of Lipschitz constants.

Original: Instead, such bounds depend on data dependent empirical estimates thereof, which are typically much smaller.
Revised: Instead, such bounds depend on data dependent empirical estimates thereof, which are typically much smaller.

Original: Relation to prior work.
Revised: Relation to prior work.

Original: Prior works [14, 16, 38] that tie generalization to (1 , ∞ ) group norms of matrices of fully-connected layers are equally applicable to convolutional networks without unfavorable scaling w.r.t.
Revised: Prior works [14, 16, 38] that tie generalization to (1 , ∞ ) group norms of matrices of fully-connected layers are equally applicable to convolutional networks without unfavorable scaling w.r.t.

Original: input size.
Revised: input size.

Original: In particular, for (1 , ∞ ) group norms of M K , we have ∥ M K ∥ 1 , ∞ =
Revised: In particular, for (1 , ∞ ) group norms of M K , we have ∥ M K ∥ 1 , ∞ =

Original: max o ∥ K o ··· ∥ 1 , i.e., the maximum 1-norm over each (input channel, width, height) slice of K .
Revised: max o ∥ K o ··· ∥ 1 , i.e., the maximum l 1 norm over each (input channel, width, height) slice of K .

Original: Yet, due to the layer-peeling strategy common to these works, the bounds scale with the product of matrix group norms vs. the product of Lipschitz constants (as in the ˜ C ij in Theorem 3.5) for covering number based strategies.
Revised: Yet, due to the layer-peeling strategy common to these works, the bounds scale with the product of matrix group norms vs. the product of Lipschitz constants (as in the ˜ C ij in Theorem 3.5) for covering number based strategies.

Original: While one can construct settings where the product of (1 , ∞ ) group norms is smaller than the product of Lipschitz constants, this is typically not observed empirically, cf.
Revised: While one can construct settings where the product of (1 , ∞ ) group norms is smaller than the product of Lipschitz constants, this is typically not observed empirically, cf.

Original: Fig.
Revised: Fig.

Original: Alternatively, Long & Sedghi [31] derive a generalization bound which does not depend on  p -norms or group norms, but only on the distance to initialization with respect to the spectral norm.
Revised: Alternatively, Long & Sedghi [31] derive a generalization bound which does not depend on l p norms or group norms, but only on the distance to initialization with respect to the spectral norm.

Original: Notably, an intermediate result in this reference yields a generalization bound of similar form as ( ♠ ), scaling with the logarithm of the product of Lipschitz constants and with the square root of the number of parameters, see Section A.5.
Revised: Notably, an intermediate result in this reference yields a generalization bound of similar form as ( ♠ ), scaling with the logarithm of the product of Lipschitz constants and with the square root of the number of parameters, see Section A.5.

Original: The distance to initialization then enters the main result [31, Theorem 3.1] at the cost of a Lipschitz constraint on the initialization.
Revised: The distance to initialization then enters the main result [31, Theorem 3.1] at the cost of a Lipschitz constraint on the initialization.

Original: We argue that ( ♠ ) incorporates the distance to initialization more naturally, as it comes without constraints on the initialization itself.
Revised: We argue that ( ♠ ) incorporates the distance to initialization more naturally, as it comes without constraints on the initialization itself.

Original: Further, it holds for any sample size n and the numerical constants are explicit.
Revised: Further, it holds for any sample size n and the numerical constants are explicit.

Original: Last, in the special case of fully-connected layers and no skip connections, ( ♣ ) reduces to the Rademacher complexity bound from [3].
Revised: Last, in the special case of fully-connected layers and no skip connections, ( ♣ ) reduces to the Rademacher complexity bound from [3].

Original: Yet, there are three differences to this result: (i) a different numerical constant, (ii) the logarithm is replaced with a harmonic number, and (iii) there are no ceiling functions.
Revised: Yet, there are three differences to this result: (i) a different numerical constant, (ii) the logarithm is replaced with a harmonic number, and (iii) there are no ceiling functions.

Original: From our understanding, these modifications are equally necessary when proving the special case directly.
Revised: From our understanding, these modifications are equally necessary when proving the special case directly.

Original: Nevertheless, these differences are only of minor importance, as they do not affect the asymptotic behavior of the bound.
Revised: Nevertheless, these differences are only of minor importance, as they do not affect the asymptotic behavior of the bound.

Original: For more details, see Section A.4.
Revised: For more details, see Section A.4.

Original:  Empirical Evaluation
Revised:  Empirical Evaluation

Original: To assess the excess capacity of a neural network trained via a standard protocol on some dataset, we seek a hypothesis class that contains a network of the same architecture with comparable testing error but smaller capacity.
Revised: To assess the excess capacity of a neural network trained via a standard protocol on some dataset, we seek a hypothesis class that contains a network of the same architecture with comparable testing error but smaller capacity.

Original: Controlling capacity via the bounds in Theorem 3.5 requires simultaneously constraining the Lipschitz constants per layer and the (2,1) group norm distance of each layer’s weight to its initialization.
Revised: Controlling capacity via the bounds in Theorem 3.5 requires simultaneously constraining the Lipschitz constants per layer and the (2,1) group norm distance of each layer’s weight to its initialization.

Original: We first discuss how to enforce the constraints.
Revised: We first discuss how to enforce the constraints.

Original: Then, we fix a residual network architecture and train on datasets of increasing difficulty while varying the constraint strengths.
Revised: Then, we fix a residual network architecture and train on datasets of increasing difficulty while varying the constraint strengths.

Original: Capacity reduction.
Revised: Capacity reduction.

Original: Controlling hypothesis class capacity necessitates ensuring that optimization yields a network parametrization that satisfies the desired constraints.
Revised: Controlling hypothesis class capacity necessitates ensuring that optimization yields a network parametrization that satisfies the desired constraints.

Original: To this end, we implement a variant of projected stochastic gradient descent (SGD) where, after a certain number of update steps, we project onto the intersection of the corresponding constraint sets C 1 and C 2 .
Revised: To this end, we implement a variant of projected stochastic gradient descent (SGD) where, after a certain number of update steps, we project onto the intersection of the corresponding constraint sets C 1 and C 2 .

Original: For convolutional layers, parametrized by tensors K , these are the convex sets C 

Revised: For convolutional layers, parametrized by tensors K , these are the convex sets C

Original: =
Revised: =

Original: { K : ∥ K − K 0 ∥ 2 , 1 ≤ b } and
Revised: { K : ∥ K − K 0 ∥ 2 , 1 ≤ b } and

Original: C 

Revised: C

Original: = { K : Lip( ϕ K ) ≤ s } .
Revised: = { K : Lip( ϕ K ) ≤ s } .

Original: Hence, jointly satisfying the constraints is a convex feasibility problem of finding a point in C =
Revised: Hence, jointly satisfying the constraints is a convex feasibility problem of finding a point in C =

Original: C 1 ∩ C 2 .
Revised: C 1 ∩ C 2 .

Original: To ensure C ̸ = ∅ , we initially (prior to optimization) scale each layer’s weight K 

Revised: To ensure C ̸ = ∅ , we initially (prior to optimization) scale each layer’s weight K

Original: so that Lip( ϕ K 0 ) = s .
Revised: so that Lip( ϕ K 0 ) = s .

Original: This starting point (per layer) resides in C by construction.
Revised: This starting point (per layer) resides in C by construction.

Original: To project onto C , we rely on alternating orthogonal projections which map K to a tensor in C 1 , resp.
Revised: To project onto C , we rely on alternating orthogonal projections which map K to a tensor in C 1 , resp.

Original: C 2 with minimal distance to K .
Revised: C 2 with minimal distance to K .

Original: Repeated application of these projections converges to a point in the intersection C [5].
Revised: Repeated application of these projections converges to a point in the intersection C [5].

Original: To implement the orthogonal projections onto C 1 and C 2 , we rely on work by Liu et al.
Revised: To implement the orthogonal projections onto C 1 and C 2 , we rely on work by Liu et al.

Original: [30] and Sedghi et al.
Revised: [30] and Sedghi et al.

Original: [40], respectively.
Revised: [40], respectively.

Original: The latter requires certain architectural prerequisites, and in consequence, we need to use convolutions of stride 1 (though our bounds equally hold for strides > 1 ) and to reduce spatial dimensionality only via max-pooling.
Revised: The latter requires certain architectural prerequisites, and in consequence, we need to use convolutions of stride 1 (though our bounds equally hold for strides > 1 ) and to reduce spatial dimensionality only via max-pooling.

Original: Further, we use circular padding and kernel sizes not larger than the input dimensionalities.
Revised: Further, we use circular padding and kernel sizes not larger than the input dimensionalities.

Original: For details on the projection algorithm and a comparison to alternating radial projections, see Section B.2.
Revised: For details on the projection algorithm and a comparison to alternating radial projections, see Section B.2.

Original: Architecture.
Revised: Architecture.

Original: We use a slightly modified (pre-activation) ResNet18 [21].
Revised: We use a slightly modified (pre-activation) ResNet18 [21].

Original: Modifications include: (1) the removal of batch normalization and biases; (2) skip connections for residual blocks where the number of channels doubles and spatial dimensionality is halved are implemented via a fixed map.
Revised: Modifications include: (1) the removal of batch normalization and biases; (2) skip connections for residual blocks where the number of channels doubles and spatial dimensionality is halved are implemented via a fixed map.

Original: Each half of the resulting channels is obtained via 2 × 2 spatial max-pooling (shifted by one pixel).
Revised: Each half of the resulting channels is obtained via 2 × 2 spatial max-pooling (shifted by one pixel).

Original: This map has Lipschitz constant √ 2 and is similar to the shortcut variant (A) of [20]; finally, (3) we fix the weight vectors of the last (classification) layer at the vertices of a κ − 1 unit simplex.
Revised: This map has Lipschitz constant √ 2 and is similar to the shortcut variant (A) of [20]; finally, (3) we fix the weight vectors of the last (classification) layer at the vertices of a κ − 1 unit simplex.

Original: Fixing the classifier is motivated by [18] and the simplex configuration is inspired by [17, 48] who show that this configuration corresponds to the geometric weight arrangement one would obtain at minimal cross-entropy loss.
Revised: Fixing the classifier is motivated by [18] and the simplex configuration is inspired by [17, 48] who show that this configuration corresponds to the geometric weight arrangement one would obtain at minimal cross-entropy loss.

Original: By construction, this classifier has Lipschitz constant (cid:112) κ / κ − 1 .
Revised: By construction, this classifier has Lipschitz constant (cid:112) κ / κ − 1 .

Original: Notably, modifications (2) and (3) do not harm performance, with empirical testing errors on a par with a standard ResNet18 without batch normalization.
Revised: Notably, modifications (2) and (3) do not harm performance, with empirical testing errors on a par with a standard ResNet18 without batch normalization.

Original: Modification (1), i.e., the omission of normalization layers, was done to ensure that the experiments are in the setting of Eq.
Revised: Modification (1), i.e., the omission of normalization layers, was done to ensure that the experiments are in the setting of Eq.

Original: (11) and therefore that ( ♣ , ♠ ) are faithful capacity measures.
Revised: (11) and therefore that ( ♣ , ♠ ) are faithful capacity measures.

Original: However, it is accompanied by a noticeable increase in testing error.
Revised: However, it is accompanied by a noticeable increase in testing error.

Original: In principle, our theory could handle batch normalization, as, during evaluation, the latter is just an affine map parametrized by the running mean and variance learned during training.
Revised: In principle, our theory could handle batch normalization, as, during evaluation, the latter is just an affine map parametrized by the running mean and variance learned during training.

Original: However, including normalization in our empirical evaluation is problematic, as normalizing batches of small variance requires the normalization layers to have a large Lipschitz constant.
Revised: However, including normalization in our empirical evaluation is problematic, as normalizing batches of small variance requires the normalization layers to have a large Lipschitz constant.

Original: Consequently, considering normalization layers as affine maps and enforcing Lipschitz constraints on them could prevent proper normalization of the data.
Revised: Consequently, considering normalization layers as affine maps and enforcing Lipschitz constraints on them could prevent proper normalization of the data.

Original: Another strategy would be to consider normalization layers as fixed nonlinearities which normalize each batch to zero mean and unit variance.
Revised: Another strategy would be to consider normalization layers as fixed nonlinearities which normalize each batch to zero mean and unit variance.

Original: However, this map is not Lipschitz continuous, and again, modifications could hinder normalization (which
Revised: However, this map is not Lipschitz continuous, and again, modifications could hinder normalization (which

Original: [Figure]
Revised: [Figure]

Original: CIFAR10 (task difficulty: easy)
Revised: CIFAR10 (task difficulty: easy)

Original: CIFAR100 (task difficulty: medium)
Revised: CIFAR100 (task difficulty: medium)

Original: Tiny-ImageNet-200 (task difficulty: hard)
Revised: Tiny-ImageNet-200 (task difficulty: hard)

Original: [Figure]
Revised: [Figure]

Original:  a i n i n g acc
Revised: r a i n i n g acc

Original: [Figure]
Revised: [Figure]

Original: ac y ( 2 , 1 ) g r o upn o r m d
Revised: ac y ( 2 , 1 ) g r o upn o r m d

Original: i s t a n ce c on s t r a i n t
Revised: i s t a n ce c on s t r a i n t

Original: [Figure]
Revised: [Figure]

Original: /oconstraints [Figure] w/oconstraints 0.5 0. [Figure]
Revised: Operatingpoint

Original: [Figure]
Revised: [Figure]

Original: 0.9 1.0 Lip.

Original: Operatingpoint
Revised: Operatingpoint

Original: Acc.(Tst) Acc.(Tst)
Revised: Acc.(Tst) Acc.(Tst)

Original: Acc.(Tst)
Revised: Acc.(Tst)

Original: O p e r a ti ngpo i n t O p e r a ti ngpo i n t
Revised: O p e r a ti ngpo i n t O p e r a ti ngpo i n t

Original: O p e r a ti ngpo i n t defeats the very purpose of these layers).
Revised: O p e r a ti ngpo i n t defeats the very purpose of these layers).

Original: Hence, we decided to remove normalization layers in our empirical evaluation.
Revised: Hence, we decided to remove normalization layers in our empirical evaluation.

Original: Presumably, however, there is a middle ground where capacity is reduced, and normalization is still possible.
Revised: Presumably, however, there is a middle ground where capacity is reduced, and normalization is still possible.

Original: If so, excess capacity could be assessed for very deep architectures, which are difficult to train without normalization layers.
Revised: If so, excess capacity could be assessed for very deep architectures, which are difficult to train without normalization layers.

Original: Datasets & Training.
Revised: Datasets & Training.

Original: We test on three benchmark datasets: CIFAR10/100 [25], and Tiny-ImageNet200 [24], listed in order of increasing task difficulty.
Revised: We test on three benchmark datasets: CIFAR10/100 [25], and Tiny-ImageNet200 [24], listed in order of increasing task difficulty.

Original: We minimize the cross-entropy loss using SGD with momentum (0.9) and small weight decay (1e-4) for 200 epochs with batch size 256 and follow a CIFAR-typical stepwise learning rate schedule, decaying the initial learning rate (of 3e-3) by a factor of 5 × at epochs 60, 120 & 160 [10].
Revised: We minimize the cross-entropy loss using SGD with momentum (0.9) and small weight decay (1e-4) for 200 epochs with batch size 256 and follow a CIFAR-typical stepwise learning rate schedule, decaying the initial learning rate (of 3e-3) by a factor of 5 × at epochs 60, 120 & 160 [10].

Original: No data augmentation is used .
Revised: No data augmentation is used .

Original: When projecting onto the constraint sets, we found one alternating projection step every 15th SGD update to be sufficient to remain close to C .
Revised: When projecting onto the constraint sets, we found one alternating projection step every 15th SGD update to be sufficient to remain close to C .

Original: To ensure that a trained model is within the capacity-constrained class, we perform 15 additional alternating projection steps after the final SGD update.
Revised: To ensure that a trained model is within the capacity-constrained class, we perform 15 additional alternating projection steps after the final SGD update.

Original: For consistency, all experiments are run with the same hyperparameters.
Revised: For consistency, all experiments are run with the same hyperparameters.

Original: Consequently, hyperparameters are chosen so that training converges for the strongest constraints we assess.
Revised: Consequently, hyperparameters are chosen so that training converges for the strongest constraints we assess.

Original: In particular, we train for 200 epochs even though unconstrained and weakly constrained models can be trained much faster.
Revised: In particular, we train for 200 epochs even though unconstrained and weakly constrained models can be trained much faster.

Original: Importantly, this affects the assessment of excess capacity only marginally, as we observe that the testing error does not deteriorate in case of more update steps.
Revised: Importantly, this affects the assessment of excess capacity only marginally, as we observe that the testing error does not deteriorate in case of more update steps.

Original: Similarly, the Lipschitz constants and the distance to initialization stay almost constant once the close-to-zero training error regime is reached, which may happen way before 200 epochs are completed.
Revised: Similarly, the Lipschitz constants and the distance to initialization stay almost constant once the close-to-zero training error regime is reached, which may happen way before 200 epochs are completed.

Original: 4.1 Results
Revised: 4.1 Results

Original: First, we assess the capacity-driving quantities in our bounds for models trained without constraints 1 .
Revised: First, we assess the capacity-driving quantities in our bounds for models trained without constraints 1 .

Original: Table 1 (top) lists a comparison across datasets, along with the capacity measures ( ♣ , ♠ ), the training/testing error, and the empirical generalization gap (i.e., the difference between testing and training error).
Revised: Table 1 (top) lists a comparison across datasets, along with the capacity measures ( ♣ , ♠ ), the training/testing error, and the empirical generalization gap (i.e., the difference between testing and training error).

Original: In accordance with our motivating figure from Section 1 (Fig. 1), we observe an overall increase in both capacity-driving quantities as a function of task difficulty.
Revised: In accordance with our motivating figure from Section 1 (Fig. 1), we observe an overall increase in both capacity-driving quantities as a function of task difficulty.

Original: To assess excess capacity in the context of questions ( Q1 ) and ( Q2 ), we first identify, per dataset, the most restrictive constraint combination where the testing error 2 is as close as possible to the unconstrained regime.
Revised: To assess excess capacity in the context of questions ( Q1 ) and ( Q2 ), we first identify, per dataset, the most restrictive constraint combination where the testing error 2 is as close as possible to the unconstrained regime.

Original: We refer to this setting as the operating point for the constraints, characterizing the function class F that serves as a reference to measure excess capacity.
Revised: We refer to this setting as the operating point for the constraints, characterizing the function class F that serves as a reference to measure excess capacity.

Original: The operating points per dataset, as well as the corresponding results are listed in Table 1 (bottom).
Revised: The operating points per dataset, as well as the corresponding results are listed in Table 1 (bottom).

Original: With respect to ( Q1 ) we find that networks can indeed maintain, or even improve, testing error at substantially lowered capacity (see performance comparison in Fig.
Revised: With respect to ( Q1 ) we find that networks can indeed maintain, or even improve, testing error at substantially lowered capacity (see performance comparison in Fig.

Original: 3 relative to f ).
Revised: 3 relative to f ).

Original: Furthermore, the observation that the capacity of the constrained models (surprisingly) remains in the same order of magnitude across tasks of varying difficulty, strongly suggests a negative answer to question ( Q2 ).
Revised: Furthermore, the observation that the capacity of the constrained models (surprisingly) remains in the same order of magnitude across tasks of varying difficulty, strongly suggests a negative answer to question ( Q2 ).

Original: [Table]
Revised: [Table]

Original: A reduced-capacity model from an easy task can perform well on a difficult task.
Revised: A reduced-capacity model from an easy task can perform well on a difficult task.

Original: Nevertheless, when comparing the top vs. bottom part of Table 1 with respect to column ( ♣ ), we do see that task difficulty primarily manifests as excess capacity.
Revised: Nevertheless, when comparing the top vs. bottom part of Table 1 with respect to column ( ♣ ), we do see that task difficulty primarily manifests as excess capacity.

Original: Another manifestation of task difficulty is evident from the more detailed analysis in Fig.
Revised: Another manifestation of task difficulty is evident from the more detailed analysis in Fig.

Original: 3 (bottom), where we see that tightening both constraints beyond the identified operating point leads to a more rapid deterioration of the testing error as the task difficulty increases.
Revised: 3 (bottom), where we see that tightening both constraints beyond the identified operating point leads to a more rapid deterioration of the testing error as the task difficulty increases.

Original: Interestingly, at the operating point, the constrained models do not only share similar capacity across datasets, but also similar empirical generalization gaps , primarily due to leaving the ubiquitous zero-training-error regime.
Revised: Interestingly, at the operating point, the constrained models do not only share similar capacity across datasets, but also similar empirical generalization gaps , primarily due to leaving the ubiquitous zero-training-error regime.

Original: The latter is particularly noteworthy, as strong regularization (e.g., via weight decay) can equally enforce this behavior, but typically at the cost of a large increase in testing error (which we do not observe).
Revised: The latter is particularly noteworthy, as strong regularization (e.g., via weight decay) can equally enforce this behavior, but typically at the cost of a large increase in testing error (which we do not observe).

Original: Finally, the parameter-counting variant of our bound (see
Revised: Finally, the parameter-counting variant of our bound (see

Original: Table 1, column ♠ ) is, by construction, much less affected by the constraints and apparently fails to capture the observations above.
Revised: Table 1, column ♠ ) is, by construction, much less affected by the constraints and apparently fails to capture the observations above.

Original: This highlights the relevance of tying capacity to weight norms and underscores their utility in our context.
Revised: This highlights the relevance of tying capacity to weight norms and underscores their utility in our context.

Original:  Discussion
Revised:  Discussion

Original: Studying the capacity of neural networks hinges crucially on the measure that is used to quantify it.
Revised: Studying the capacity of neural networks hinges crucially on the measure that is used to quantify it.

Original: In our case, capacity rests upon two bounds on the empirical Rademacher complexity, both depending on weight norms and the number of parameters, but to different extents.
Revised: In our case, capacity rests upon two bounds on the empirical Rademacher complexity, both depending on weight norms and the number of parameters, but to different extents.

Original: Hence, exerting control over the weight norms manifests in different ways: in case of the more weight norm dependent capacity measure, our results show substantial task-dependent excess capacity , while, when relying more on parameter counting, this effect is less pronounced.
Revised: Hence, exerting control over the weight norms manifests in different ways: in case of the more weight norm dependent capacity measure, our results show substantial task-dependent excess capacity , while, when relying more on parameter counting, this effect is less pronounced.

Original: Although the latter measure yields tighter bounds, its utility in terms of explaining the observed empirical behavior is limited: in fact, capacity tied to weight norms not only better correlates with observed generalization gaps (both with and without constraints), but the amount of tolerable capacity reduction also reflects the smaller generalization gaps in the constrained regime.
Revised: Although the latter measure yields tighter bounds, its utility in terms of explaining the observed empirical behavior is limited: in fact, capacity tied to weight norms not only better correlates with observed generalization gaps (both with and without constraints), but the amount of tolerable capacity reduction also reflects the smaller generalization gaps in the constrained regime.

Original: Note that our results rest upon carefully implementing constraint enforcement during optimization.
Revised: Note that our results rest upon carefully implementing constraint enforcement during optimization.

Original: Hence, numerical schemes to better account for this setting might potentially reveal an even more pronounced excess capacity effect.
Revised: Hence, numerical schemes to better account for this setting might potentially reveal an even more pronounced excess capacity effect.

Original: In summary, our experiments, guided by the theoretical bounds, strongly suggest a notion of compressibility of networks with respect to weight norms.
Revised: In summary, our experiments, guided by the theoretical bounds, strongly suggest a notion of compressibility of networks with respect to weight norms.

Original: This compressibility only moderately reduces with task difficulty.
Revised: This compressibility only moderately reduces with task difficulty.

Original: We believe these observations to be particularly relevant and we foresee them sparking future work along this direction.
Revised: We believe these observations to be particularly relevant and we foresee them sparking future work along this direction.

Original: Acknowledgements
Revised: Acknowledgements

Original: This work was supported by the Austrian Science Fund (FWF) under project FWF P31799-N38 and the Land Salzburg under projects 0102-F1901166-KZP and 20204-WISS/225/197-2019.
Revised: This work was supported by the Austrian Science Fund (FWF) under project FWF P31799-N38 and the Land Salzburg under projects 0102-F1901166-KZP and 20204-WISS/225/197-2019.

Original: Appendix
Revised: Appendix

Original: (Supplementary Material
Revised: (Supplementary Material) –

Original: Overview A Comparison with prior work on Rademacher complexity bounds
Revised: Overview A Comparison with prior work on Rademacher complexity bounds

Original: A.1 Analysis of matrices corresponding to convolutions . . . . . . . . . . . . . . . . .
Revised: A.1 Analysis of matrices corresponding to convolutions . . . . . . . . . . . . . . . . .

Original: A.2 Comparison of bounds for convolutional networks . . . . . . . . . . . . . . . . . .
Revised: A.2 Comparison of bounds for convolutional networks . . . . . . . . . . . . . . . . . .

Original: A.3 Details for the numerical comparison in Fig.
Revised: A.3 Details for the numerical comparison in Fig.

Original: A.4 Comparison with Bartlett et al.
Revised: A.4 Comparison with Bartlett et al.

Original: A.5 Comparison with Long & Sedghi [31] . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.5 Comparison with Long & Sedghi [31] . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.6 Comparison with Ledent et al.
Revised: A.6 Comparison with Ledent et al.

Original: A.6.1 Comparison for two-layer networks . . . . . . . . . . . . . . . . . . . . .

Original: B Additional Experiments
Revised: B Additional Experiments

Original: B.1 Excess capacity in non-residual networks . . . .
Revised: B.1 Excess capacity in non-residual networks . . . .

Original: B.2 Projection method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: B.2 Projection method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.3 Hardware resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: B.3 Hardware resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.4 Comparison between constrained and unconstrained models beyond testing error .
Revised: B.4 Comparison between constrained and unconstrained models beyond testing error .

Original: B.4.
Revised: B.4.

Original: Biases to particular classes . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Biases to particular classes . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.4.
Revised: B.4.

Original: Susceptibility to adversarial attacks . . . . . . . . . . . . . . . . . . . . .
Revised: Susceptibility to adversarial attacks . . . . . . . . . . . . . . . . . . . . .

Original: B.4.
Revised: B.4.

Original: Compressibility via weight pruning 
Revised: Compressibility via weight pruning . . . . . . . . . . . . . . . . . . . . .

Original: C Proofs
Revised: C Proofs

Original: C.1 Preliminaries
Revised: C.1 Preliminaries

Original: C.2 Single-layer covering number bounds
Revised: C.2 Single-layer covering number bounds

Original: C.3 Whole-network covering number bounds (general form)
Revised: C.3 Whole-network covering number bounds (general form

Original: C.3.
Revised: C.3.

Revised: Covering

Original: Covering number bounds for compositions and summations
Revised: number bounds for compositions and summations

Original: C
Revised: C.

Original: Generalstrategy for bounding the covering numbers of
Revised: General

Original:  complex classes . .
Revised: strategy for bounding the covering numbers of complex classes . .

Original: C.
Revised: C.3.

Original: Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.3.
Revised: C.

Original: Covering number bounds for concatenations
Revised: Covering number bounds for concatenations

Original: C.4 Whole-network covering number bounds (convolutional & fully-connected)
Revised: C.4 Whole-network covering number bounds (convolutional & fully-connected)

Original: C
Revised: C.4.

Original: Bounds for residual networks
Revised: Bounds for residual networks

Original: C.4.2 Application to specific architectures
Revised: C.4.2 Application to specific architectures

Original: C.5 Rademacher complexity & Generalization bounds
Revised: C.5 Rademacher complexity & Generalization bounds

Original: C.6 Calculations
Revised: C.6 Calculations

Original: In the appendix, we present (1) a more detailed comparison to prior work (Section A), (2) additional experiments (Section B) and (3) list all proofs which are left-out in the main manuscript (Section C).
Revised: In the appendix, we present (1) a more detailed comparison to prior work (Section A), (2) additional experiments (Section B) and (3) list all proofs which are left-out in the main manuscript (Section C).

Original: In particular, in Section C.2, we derive our single-layer covering number bounds from
Revised: In particular, in Section C.2, we derive our single-layer covering number bounds from

Original: Theorem 3.2; Section C.3 presents the modularized strategy from Section 3.3 to obtain whole-network covering number bounds.
Revised: Theorem 3.2; Section C.3 presents the modularized strategy from Section 3.3 to obtain whole-network covering number bounds.

Original: This includes several examples (e.g., residual networks) and an extension to accommodate concatenation .
Revised: This includes several examples (e.g., residual networks) and an extension to accommodate concatenation .

Original: Section C.4 then tailors our empirical Rademacher complexity bounds to networks with fully-connected and convolutional layers, and Section C.5 finally lists the accompanying generalization bounds.
Revised: Section C.4 then tailors our empirical Rademacher complexity bounds to networks with fully-connected and convolutional layers, and Section C.5 finally lists the accompanying generalization bounds.

Original: A Comparison with prior work on Rademacher complexity bounds A.1 Analysis of matrices corresponding to convolutions
Revised: A Comparison with prior work on Rademacher complexity bounds A.1 Analysis of matrices corresponding to convolutions

Original: We compare the norms of the matrices M K corresponding to the linear map realized by a convolutional layer with the norm of the corresponding weight tensor K .
Revised: We compare the norms of the matrices M K corresponding to the linear map realized by a convolutional layer with the norm of the corresponding weight tensor K .

Original: This facilitates studying the Rademacher complexity of convolutional layers via norm-based bounds for fully-connected layers.
Revised: This facilitates studying the Rademacher complexity of convolutional layers via norm-based bounds for fully-connected layers.

Original: In accordance with the definition of the (2 , 1) group norm in Eq.
Revised: In accordance with the definition of the (2 , 1) group norm in Eq.

Original: (7), we define the ( p, q ) group norm of a weight tensor K as
Revised: (7), we define the ( p, q ) group norm of a weight tensor K as

Original: [Equation]
Revised: [Equation]

Original: For simplicity, we (i) consider only circular, input-size preserving paddings, (ii) assume that the spatial input dimension h =
Revised: For simplicity, we (i) consider only circular, input-size preserving paddings, (ii) assume that the spatial input dimension h =

Original: w = d is a multiple of the convolution stride t , and (iii) assume that the kernel size k h
Revised: w = d is a multiple of the convolution stride t , and (iii) assume that the kernel size k h

Original: = k w = k ≤ d .
Revised: = k w = k ≤ d .

Original: In this setting, convolution corresponds to the application of a local map R c in × k × k
Revised: In this setting, convolution corresponds to the application of a local map R c in × k × k

Original: → R c out , specified by the weight tensor K ∈ R c out × c in × k × k , to all ( d/t ) 2 patches.
Revised: → R c out , specified by the weight tensor K ∈ R c out × c in × k × k , to all ( d/t ) 2 patches.

Original: Each row [ M K ] a · of M K ∈ R
Revised: Each row [ M K ] a · of M K ∈ R

Original: c out ( d/t ) 2 × c in d 2 has as non-zero elements only the entries of the tensor
Revised: c out ( d/t ) 2 × c in d 2 has as non-zero elements only the entries of the tensor

Original: K o a ··· for some o a = 1 , . . .
Revised: K o a ··· for some o a = 1 , . . .

Original: , c out .
Revised: , c out .

Original: Thus, the ( p, q ) group norm of M ⊤ K is
Revised: Thus, the ( p, q ) group norm of M ⊤ K is

Original: [Equation]
Revised: [Equation]

Original: In particular,
Revised: In particular,

Original: [Equation]
Revised: [Equation]

Original: Note the benefit of the (1 , ∞ ) group norm, which does not scale with the input dimension d .
Revised: Note the benefit of the (1 , ∞ ) group norm, which does not scale with the input dimension d .

Original: We point out that for p > q (Hölder inequality for p/q )
Revised: We point out that for p > q (Hölder inequality for p/q )

Original: [Equation]
Revised: [Equation]

Original: and so
Revised: and so

Original: [Equation]
Revised: [Equation]

Original: This inequality quantifies the disadvantage of applying generalization bounds for fully-connected layers directly on the matrices that parametrize the linear maps corresponding to convolutions.
Revised: This inequality quantifies the disadvantage of applying generalization bounds for fully-connected layers directly on the matrices that parametrize the linear maps corresponding to convolutions.

Original: A.2 Comparison of bounds for convolutional networks
Revised: A.2 Comparison of bounds for convolutional networks

Original: Table 3 lists various upper bounds on the empirical Rademacher complexity of convolutional networks as specified in the paragraph below, in a common notation (see Table 2).
Revised: Table 3 lists various upper bounds on the empirical Rademacher complexity of convolutional networks as specified in the paragraph below, in a common notation (see Table 2).

Original: As in Section A.1, we
Revised: As in Section A.1, we

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Table]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: consider input-size preserving circular padding
Revised: consider input-size preserving circular padding

Original: [Table]
Revised: [Table]

Original: convolutions with stride of t
Revised: convolutions with stride of t

Original: [Table]
Revised: [Table]

Original: and assume that (i) the spatial dimensions h
Revised: and assume that (i) the spatial dimensions h

Original: = w = d are a multiple of t and (ii) the kernel size is k h
Revised: = w = d are a multiple of t and (ii) the kernel size is k h

Original: = k w = k ≤ d .
Revised: = k w = k ≤ d .

Original: The empirical Rademacher complexity bounds in Table 3 are formulated in dependence of norms of the weights K i .
Revised: The empirical Rademacher complexity bounds in Table 3 are formulated in dependence of norms of the weights K i .

Original: This is not entirely accurate, as the bounds typically refer to neural networks whose weights satisfy a priori defined norm constraints .
Revised: This is not entirely accurate, as the bounds typically refer to neural networks whose weights satisfy a priori defined norm constraints .

Original: We choose this abuse of notation so that we do not need to introduce additional variables for each norm constraint.
Revised: We choose this abuse of notation so that we do not need to introduce additional variables for each norm constraint.

Original: The listed bounds are used in the numerical comparison for unconstrained networks in Fig.
Revised: The listed bounds are used in the numerical comparison for unconstrained networks in Fig.

Original: More specifically, we consider a hypothesis class F represented by a neural network of the form f
Revised: More specifically, we consider a hypothesis class F represented by a neural network of the form f

Original: = σ L ◦ f L ◦ · · · ◦ σ 1 ◦ f 1 , where σ i : x (cid:55)→ max( x, 0) denotes the ReLU activation function and f i identifies a convolutional layer.
Revised: = σ L ◦ f L ◦ · · · ◦ σ 1 ◦ f 1 , where σ i : x (cid:55)→ max( x, 0) denotes the ReLU activation function and f i identifies a convolutional layer.

Original: Note that fully-connected layers, e.g., a linear classifier at the last layer, can be handled by setting the spatial input dimension d
Revised: Note that fully-connected layers, e.g., a linear classifier at the last layer, can be handled by setting the spatial input dimension d

Original: i , the kernel size k i and the stride t i all equal to 1 .
Revised: i , the kernel size k i and the stride t i all equal to 1 .

Original: Bounds designed for fully-connected networks are applied to the matrices M K that correspond to the weight tensor K which parametrizes the convolution.
Revised: Bounds designed for fully-connected networks are applied to the matrices M K that correspond to the weight tensor K which parametrizes the convolution.

Original: To handle the multiclass regime, the covering number based bounds will be applied to F γ , see Eq.
Revised: To handle the multiclass regime, the covering number based bounds will be applied to F γ , see Eq.

Original: Layer-peeling based bounds, originally presented for binary classification, are multiplied by the number of classes κ (according to [32]) as done in [16].
Revised: Layer-peeling based bounds, originally presented for binary classification, are multiplied by the number of classes κ (according to [32]) as done in [16].

Original: A.3 Details for the numerical comparison in Fig.
Revised: A.3 Details for the numerical comparison in Fig.

Original: In Fig.
Revised: In Fig.

Original: 2, we evaluated several existing upper bounds (see Table 3) on the empirical Rademacher complexity of convolutional networks for two specific architectures.
Revised: 2, we evaluated several existing upper bounds (see Table 3) on the empirical Rademacher complexity of convolutional networks for two specific architectures.

Original: The first architecture (a 6-layer network) consists of 5 convolutional layers with stride 2, kernel size 3, padding of 1 and 256 filters / output channels, so that a (3 × 32 × 32) -dimensional input image is mapped to a (256 × 1 × 1) -dimensional representation.
Revised: The first architecture (a 6-layer network) consists of 5 convolutional layers with stride 2, kernel size 3, padding of 1 and 256 filters / output channels, so that a (3 × 32 × 32) -dimensional input image is mapped to a (256 × 1 × 1) -dimensional representation.

Original: The subsequent linear classifier is a convolutional layer with kernel size 1 and no padding.
Revised: The subsequent linear classifier is a convolutional layer with kernel size 1 and no padding.

Original: Its number of filters equals the number of classes of the classification problem.
Revised: Its number of filters equals the number of classes of the classification problem.

Original: This classification layer is equivalent to applying a fullyconnected layer to the flattened representations.
Revised: This classification layer is equivalent to applying a fullyconnected layer to the flattened representations.

Original: The second architecture (an 11-layer network) only differs in that each convolutional layer with stride 2 is followed by an additional convolutional layer with stride 1 (kernel size 3, padding of 1 and 256 filters).
Revised: The second architecture (an 11-layer network) only differs in that each convolutional layer with stride 2 is followed by an additional convolutional layer with stride 1 (kernel size 3, padding of 1 and 256 filters).

Original: All activation functions are ReLUs.
Revised: All activation functions are ReLUs.

Original: We trained both networks on the CIFAR10 dataset, minimizing the cross-entropy loss using stochastic gradient descent (SGD) with batch size 256, weight decay (1e-4), and momentum (0.9).
Revised: We trained both networks on the CIFAR10 dataset, minimizing the cross-entropy loss using stochastic gradient descent (SGD) with batch size 256, weight decay (1e-4), and momentum (0.9).

Original: During the 100 training epochs, the learning rate is gradually reduced following a cosine annealing schedule, starting with an initial learning rate of 1e-4.
Revised: During the 100 training epochs, the learning rate is gradually reduced following a cosine annealing schedule, starting with an initial learning rate of 1e-4.

Original: Notably, we do not use any data augmentation .
Revised: Notably, we do not use any data augmentation .

Original: Both networks fit the training data, achieving an accuracy of 72.5% (6-layer), resp.
Revised: Both networks fit the training data, achieving an accuracy of 72.5% (6-layer), resp.

Original: 77.9% (11-layer), on the test data.
Revised: 77.9% (11-layer), on the test data.

Original: To assess the different empirical Rademacher complexity bounds, we measured weight norms of the networks’ layers and inserted them into the bounds from Table 3.
Revised: To assess the different empirical Rademacher complexity bounds, we measured weight norms of the networks’ layers and inserted them into the bounds from Table 3.

Original: Fig.
Revised: Fig.

Original: 4 illustrates the results.
Revised: 4 illustrates the results.

Original: Note that, following our discussion in Section A.2, Rademacher complexity bounds are typically formulated for networks with a priori specified weight norm constraints, whereas here, we train unconstrained networks and merely measure the weight norms at the end of training.
Revised: Note that, following our discussion in Section A.2, Rademacher complexity bounds are typically formulated for networks with a priori specified weight norm constraints, whereas here, we train unconstrained networks and merely measure the weight norms at the end of training.

Original: Fig.
Revised: Fig.

Original: 4 highlights several aspects of the studied bounds.
Revised: 4 highlights several aspects of the studied bounds.

Original: First of all, all bounds are vacuous , as they are larger than 1 .
Revised: First of all, all bounds are vacuous , as they are larger than 1 .

Original: Somewhat surprisingly, the bounds mainly driven by the number of parameters, i.e.,  [29] and Theorem 3.5 ( ♠ ), are clearly the smallest.
Revised: Somewhat surprisingly, the bounds mainly driven by the number of parameters are clearly the smallest, i.e., the ones from Lin et al. [29] and Theorem 3.5 ( ♠ ).

Original: As expected, our bound from Theorem 3.5 ( ♣ ) is smaller than [3], as it accounts for the structure of convolutions.
Revised: As expected, our bound from Theorem 3.5 ( ♣ ) is smaller than [3], as it accounts for the structure of convolutions.

Original: Furthermore, we see the benign scaling of the product of Lipschitz constants with the network depth compared to the product of l 2 -norms, resp., l 1 , ∞ -norms.
Revised: Furthermore, we see the benign scaling of the product of Lipschitz constants with the network depth compared to the product of l 2 norms, resp., l 1 , ∞ norms.

Original: This is mirrored in the benign scaling of the covering number based bounds compared to the layer-peeling based ones.
Revised: This is mirrored in the benign scaling of the covering number based bounds compared to the layer-peeling based ones.

Original: A.4 Comparison with Bartlett et al. [3]
Revised: A.4 Comparison with Bartlett et al. [3]

Original: Our Rademacher complexity bounds are based on the proof strategy of Bartlett et al.
Revised: Our Rademacher complexity bounds are based on the proof strategy of Bartlett et al.

Original: That is, we first derive single layer covering number bounds for convolutional layers.
Revised: That is, we first derive single layer covering number bounds for convolutional layers.

Original: In a second step, we derive covering numbers for entire residual networks.
Revised: In a second step, we derive covering numbers for entire residual networks.

Original: Last, a combination of Dudley’s entropy integral and [3, Lemma 3.1] implies the Rademacher complexity bounds ( ♣ ) and ( ♠ ).
Revised: Last, a combination of Dudley’s entropy integral and [3, Lemma 3.1] implies the Rademacher complexity bounds ( ♣ ) and ( ♠ ).

Original: As already discussed in Section 3.2, our single-layer covering number bound for convolutional layers includes the single-layer covering number bound for fully-connected layers from [3, Lemma 3.2].
Revised: As already discussed in Section 3.2, our single-layer covering number bound for convolutional layers includes the single-layer covering number bound for fully-connected layers from [3, Lemma 3.2].

Original: Consequently, in the special case of fully-connected layers and no skip connections, our main result ( ♣ ) reduces to
Revised: Consequently, in the special case of fully-connected layers and no skip connections, our main result ( ♣ ) reduces to

Original: [Figure]
Revised: [Figure]

Original: the Rademacher complexity bound from [3].
Revised: the Rademacher complexity bound from [3].

Original: To be more specific, we show that
Revised: To be more specific, we show that

Original: [Equation]
Revised: [Equation]

Original: and Bartlett et al.
Revised: and Bartlett et al.

Original: [3] prove
Revised: [3] prove

Original: [Equation]
Revised: [Equation]

Original: Here, L denotes the depth of the network and ˜ C
Revised: Here, L denotes the depth of the network and ˜ C

Original: i is the part of the capacity of the i -th layer due to weight and data norms, i.e.,
Revised: i is the part of the capacity of the i -th layer due to weight and data norms, i.e.,

Original: [Equation]
Revised: [Equation]

Original: 26) with s i the layers’ Lipschitz constraints, b i the layers’ (2,1) norm constraints and ρ i the Lipschitz constants of the nonlinearities.
Revised: 26) with s i the layers’ Lipschitz constraints, b i the layers’ (2,1) norm constraints and ρ i the Lipschitz constants of the nonlinearities.

Original: A closer look reveals that there are three differences between the results: (i) a different numerical constant, (ii) the logarithm is replaced with a harmonic number, and (iii) Eq.
Revised: A closer look reveals that there are three differences between the results: (i) a different numerical constant, (ii) the logarithm is replaced with a harmonic number, and (iii) Eq.

Original: (26) contains no ceiling functions.
Revised: (26) contains no ceiling functions.

Original: Of course, the differences do not affect the asymptotic behavior of the bound and are thus only of minor importance.
Revised: Of course, the differences do not affect the asymptotic behavior of the bound and are thus only of minor importance.

Original: From our understanding, the differences are rooted in a lapse in the inequality chains of [3, Eq.
Revised: From our understanding, the differences are rooted in a lapse in the inequality chains of [3, Eq.

Original: (A.3)].
Revised: (A.3)].

Original: • The difference in the numerical constant appears, because proving the entire network covering number bound requires transitioning to external covering numbers, which manifests as an additional factor of 2 in the final result.
Revised: • The difference in the numerical constant appears, because proving the entire network covering number bound requires transitioning to external covering numbers, which manifests as an additional factor of 2 in the final result.

Original: This is because the single layer covering number bounds from Theorem 3.2, resp.
Revised: This is because the single layer covering number bounds from Theorem 3.2, resp.

Original: [3, Lemma 3.2], hold for layers with only a (2,1) group norm constraint, which form a superset of the layers with a (2,1) group norm constraint and a Lipschitz constraint as considered in Theorem 3.5, resp [3, Theorem 3.3].
Revised: [3, Lemma 3.2], hold for layers with only a (2,1) group norm constraint, which form a superset of the layers with a (2,1) group norm constraint and a Lipschitz constraint as considered in Theorem 3.5, resp [3, Theorem 3.3].

Original: On the other hand, the parameter α in the proof of [3, Lemma A.8] can be chosen as α = 1 / √ n , which improves the bound by a factor of 3 / 2 (see proof of Theorem C.19).
Revised: On the other hand, the parameter α in the proof of [3, Lemma A.8] can be chosen as α = 1 / √ n , which improves the bound by a factor of 3 / 2 (see proof of Theorem C.19).

Original: Overall, both effects lead to a factor of 4 / 3 , which is precisely the quotient of the numerical constants in Eq.
Revised: Overall, both effects lead to a factor of 4 / 3 , which is precisely the quotient of the numerical constants in Eq.

Original: (25) and Eq.
Revised: (25) and Eq.

Original: • Our result in Eq.
Revised: • Our result in Eq.

Original: (25) contains ceiling functions and a harmonic number, which is a direct consequence of the ceiling function appearing in the single layer covering number bound of Theorem 3.2, resp.
Revised: (25) contains ceiling functions and a harmonic number, which is a direct consequence of the ceiling function appearing in the single layer covering number bound of Theorem 3.2, resp.

Original: [3, Lemma 3.2].
Revised: [3, Lemma 3.2].

Original: In the chain of inequalities [3, Eq.
Revised: In the chain of inequalities [3, Eq.

Original: (A.3)] in the proof of [3, Theorem 3.3], the single layer bound is inserted without the ceiling function.
Revised: (A.3)] in the proof of [3, Theorem 3.3], the single layer bound is inserted without the ceiling function.

Original: A.5 Comparison with Long & Sedghi [31]
Revised: A.5 Comparison with Long & Sedghi [31]

Original: Long & Sedghi [31] study generalization bounds for the class F of convolutional networks that realize functions of the form
Revised: Long & Sedghi [31] study generalization bounds for the class F of convolutional networks that realize functions of the form

Original: [Equation]
Revised: [Equation]

Original: with Lipschitz/spectral-norm constraints, i.e., they assume that the initializations ϕ K (
Revised: with Lipschitz/spectral-norm constraints, i.e., they assume that the initializations ϕ K (

Original: [Equation]
Revised: [Equation]

Original: i per layer are (1 + ν ) -Lipschitz and that the distances
Revised: i per layer are (1 + ν ) -Lipschitz and that the distances

Original: [Equation]
Revised: [Equation]

Original: to initialization satisfy (cid:80) i β i ≤ β for some given constant β > 0 .
Revised: to initialization satisfy (cid:80) i β i ≤ β for some given constant β > 0 .

Original: They show [31, Theorem 3.1] that for λ -Lipschitz loss functions ℓ , the generalization gap is (with probability 1 − δ ) uniformly bounded over the class F ℓ = { ( x, y )
Revised: They show [31, Theorem 3.1] that for λ -Lipschitz loss functions ℓ , the generalization gap is (with probability 1 − δ ) uniformly bounded over the class F ℓ = { ( x, y )

Original: (cid:55)→ ℓ ( f ( x ) , y ) | f ∈ F} by
Revised: (cid:55)→ ℓ ( f ( x ) , y ) | f ∈ F} by

Original: [Equation]
Revised: [Equation]

Original: assuming that λβχ (1 + ν + β/L ) L ≥ 5 and n large enough.
Revised: assuming that λβχ (1 + ν + β/L ) L ≥ 5 and n large enough.

Original: Here, C denotes an unspecified constant and M the maximum of the loss function ℓ .
Revised: Here, C denotes an unspecified constant and M the maximum of the loss function ℓ .

Original: Further, ¯ W = (cid:80) i
Revised: Further, ¯ W = (cid:80) i

Original: W i is the total number of network parameters and ∥ x ∥ 2 ≤
Revised: W i is the total number of network parameters and ∥ x ∥ 2 ≤

Original: χ
Revised: χ

Original: is an upper bound on the Euclidean norm of the data.
Revised: is an upper bound on the Euclidean norm of the data.

Original: As can be seen, this bound depends on the square root of parameters and the distance β to initialization.
Revised: As can be seen, this bound depends on the square root of parameters and the distance β to initialization.

Original: In contrast to other results (e.g., [3, 26, ( ♣ ), ( ♠ )]), it also depends on a Lipschitz constraint (1 + ν ) directly on the initialization.
Revised: In contrast to other results (e.g., [3, 26, ( ♣ ), ( ♠ )]), it also depends on a Lipschitz constraint (1 + ν ) directly on the initialization.

Original: Eq.
Revised: Eq.

Original: (27) is based on [31, Lemma 2.3], which requires the class F ℓ to be ( B, d ) - Lipschitz parametrized , i.e., that there exists d ∈ N and a norm ∥·∥ on R d , together with a B -Lipschitz continuous and surjective map ϕ : B (1 , ∥·∥ )
Revised: (27) is based on [31, Lemma 2.3], which requires the class F ℓ to be ( B, d ) - Lipschitz parametrized , i.e., that there exists d ∈ N and a norm ∥·∥ on R d , together with a B -Lipschitz continuous and surjective map ϕ : B (1 , ∥·∥ )

Original: → F ℓ from the ∥·∥ -unit ball in R d onto F ℓ , which is B -Lipschitz.
Revised: → F ℓ from the ∥·∥ -unit ball in R d onto F ℓ , which is B -Lipschitz.

Original: The latter means that for every θ, θ ′ ∈ B (1 , ∥·∥ ) , it holds that ∥ ϕ ( θ ) − ϕ ( θ ′ )
Revised: The latter means that for every θ, θ ′ ∈ B (1 , ∥·∥ ) , it holds that ∥ ϕ ( θ ) − ϕ ( θ ′ )

Original: ∥ ∞ ≤ B ∥
Revised: ∥ ∞ ≤ B ∥

Original: θ − θ ′ ∥ .
Revised: θ − θ ′ ∥ .

Original: In this situation, the generalization gap is bounded by
Revised: In this situation, the generalization gap is bounded by

Original: [Equation]
Revised: [Equation]

Original: In a series of lemmas [31, Lemma 3.2–3.4], the authors show that F ℓ is indeed ( B, d ) -Lipschitz parametrized with d = ¯ W and B = λχβ
Revised: In a series of lemmas [31, Lemma 3.2–3.4], the authors show that F ℓ is indeed ( B, d ) -Lipschitz parametrized with d = ¯ W and B = λχβ

Original: (1 + ν + β/L ) L .
Revised: (1 + ν + β/L ) L .

Original: We will repeat the argument and show that it implies an intermediate result which scales similarly to our result ( ♠ ) from Theorem 3.5, i.e., with the square root of
Revised: We will repeat the argument and show that it implies an intermediate result which scales similarly to our result ( ♠ ) from Theorem 3.5, i.e., with the square root of

Original: [Equation]
Revised: [Equation]

Original: where s i denote Lipschitz constraints on the layers
Revised: where s i denote Lipschitz constraints on the layers

Original: ϕ K
Revised: ϕ K

Original: i .
Revised: i .

Original: Let K =
Revised: Let K =

Original: ( K 1 , . .
Revised: ( K 1 , . .

Original: . , K L ) and ˜ K
Revised: . , K L ) and ˜ K

Original: = ( ˜ K 1 , . . .
Revised: = ( ˜ K 1 , . . .

Original: , ˜ K L ) be tuples of weight tensors and denote the corresponding networks by f K , resp.
Revised: , ˜ K L ) be tuples of weight tensors and denote the corresponding networks by f K , resp.

Original: f ˜K .
Revised: f ˜K .

Original: If K and ˜K differ in only one layer, say K j ̸ =
Revised: If K and ˜K differ in only one layer, say K j ̸ =

Original: ˜ K j , then for all x ∈ X (see proof of [31, Lemma 3.2]),
Revised: ˜ K j , then for all x ∈ X (see proof of [31, Lemma 3.2]),

Original: [Equation]
Revised: [Equation]

Original: Consequently, if K and ˜K differ in all layers, it holds that
Revised: Consequently, if K and ˜K differ in all layers, it holds that

Original: [Equation]
Revised: [Equation]

Original: As (cid:80)
Revised: As (cid:80)

Original: Lj =1 Lip( ϕ K j ) defines a norm ∥·∥ on R ¯ W (in [31] this norm is denoted ∥·∥ σ ), the inequality above implies that the surjective map
Revised: Lj =1 Lip( ϕ K j ) defines a norm ∥·∥ on R ¯ W (in [31] this norm is denoted ∥·∥ σ ), the inequality above implies that the surjective map

Original: [Equation]
Revised: [Equation]

Original: is (cid:16) χ
Revised: is (cid:16) χ

Original: (cid:16)(cid:81)
Revised: (cid:16)(cid:81)

Original: Li
Revised: Li

Original: =
Revised: =

Original: 1 s i (cid:17) (cid:80) i s i (cid:17) -Lipschitz, i.e., the class F ℓ = { ( x, y ) (cid:55)→ ℓ ( f ( x ) , y ) | f ∈ F} is (cid:16) λχ (cid:16)(cid:81)
Revised: 1 s i (cid:17) (cid:80) i s i (cid:17) -Lipschitz, i.e., the class F ℓ = { ( x, y ) (cid:55)→ ℓ ( f ( x ) , y ) | f ∈ F} is (cid:16) λχ (cid:16)(cid:81)

Original: Li
Revised: Li

Original: =
Revised: =

Original: 1 s i (cid:17) (cid:80) i s i , ¯ W (cid:17)
Revised: 1 s i (cid:17) (cid:80) i s i , ¯ W (cid:17)

Original: -Lipschitz parametrized.
Revised: -Lipschitz parametrized.

Original: Thus, Eq.
Revised: Thus, Eq.

Original: (28) implies a generalization bound of the form
Revised: (28) implies a generalization bound of the form

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: Similarly to our result ( ♠ ) from Theorem 3.5, this bound scales with the square root of the number of parameters and with the logarithm of the product of Lipschitz constants.
Revised: Similarly to our result ( ♠ ) from Theorem 3.5, this bound scales with the square root of the number of parameters and with the logarithm of the product of Lipschitz constants.

Original: However, as Eq.
Revised: However, as Eq.

Original: (31) and Eq.
Revised: (31) and Eq.

Original: (27) are proven via an asymptotic bound from Giné and Guillou [13], the constant C and the minimal sample size n required for Eq.
Revised: (27) are proven via an asymptotic bound from Giné and Guillou [13], the constant C and the minimal sample size n required for Eq.

Original: (31) and Eq.
Revised: (31) and Eq.

Original: (27) to hold are not readily available.
Revised: (27) to hold are not readily available.

Original: This makes further comparisons difficult.
Revised: This makes further comparisons difficult.

Original: Eq.
Revised: Eq.

Original: (31) differs from the main result in [31], i.e., Eq.
Revised: (31) differs from the main result in [31], i.e., Eq.

Original: (27), as, instead of constraints on the layers’
Revised: (27), as, instead of constraints on the layers’

Original: Lipschitz constants Lip( ϕ K i ) ≤ s i , Long & Sedghi consider constraints on the Lipschitz constants of the initialization Lip( ϕ K (0) i ) ≤ 1 + ν and on the distance to initialization Lip( ϕ K i − ϕ K (0) i )
Revised: Lipschitz constants Lip( ϕ K i ) ≤ s i , Long & Sedghi consider constraints on the Lipschitz constants of the initialization Lip( ϕ K (0) i ) ≤ 1 + ν and on the distance to initialization Lip( ϕ K i − ϕ K (0) i )

Original: ≤ β i with (cid:80) i β i = β .
Revised: ≤ β i with (cid:80) i β i = β .

Original: Starting from Eq.
Revised: Starting from Eq.

Original: (29), these constraints enter via the triangle inequality, i.e.,
Revised: (29), these constraints enter via the triangle inequality, i.e.,

Original: Lip( ϕ K
Revised: Lip( ϕ K

Original: [Equation]
Revised: [Equation]

Original: Maximizing (cid:81) i (1 + ν + β i ) subject to (cid:80) i β i = β , yields
Revised: Maximizing (cid:81) i (1 + ν + β i ) subject to (cid:80) i β i = β , yields

Original: [Equation]
Revised: [Equation]

Original: [31, Lemma 3.3 & 3.4] then imply that F ℓ is ( B, d ) -Lipschitz parametrized with d = ¯ W and
Revised: [31, Lemma 3.3 & 3.4] then imply that F ℓ is ( B, d ) -Lipschitz parametrized with d = ¯ W and

Original: B = λχβ exp ( νL + β ) , which in turn implies Eq.
Revised: B = λχβ exp ( νL + β ) , which in turn implies Eq.

Original: Obviously, every bound that depends on weight norms can be transferred to a bound that depends on the norm of the initialization and the distance to it, simply by application of the triangle inequality.
Revised: Obviously, every bound that depends on weight norms can be transferred to a bound that depends on the norm of the initialization and the distance to it, simply by application of the triangle inequality.

Original: We argue, that utilizing the translation invariance of covering numbers, as done in, e.g., ( ♣ ), ( ♠ ), as well as in [3, 26], is a more natural way of incorporating the distance to initialization, as it allows for bounds which do not depend on norm constraints on the initialization.
Revised: We argue, that utilizing the translation invariance of covering numbers, as done in, e.g., ( ♣ ), ( ♠ ), as well as in [3, 26], is a more natural way of incorporating the distance to initialization, as it allows for bounds which do not depend on norm constraints on the initialization.

Original: A.6 Comparison with Ledent et al. [26]
Revised: A.6 Comparison with Ledent et al. [26]

Original: In [26], Ledent et al.
Revised: In [26], Ledent et al.

Original: derive generalization/
Revised: derive generalization/

Original: Rademacher complexity bounds via l ∞ -coverings of convolutional networks.
Revised: Rademacher complexity bounds via l ∞ coverings of convolutional networks.

Original: These bounds incorporate weight sharing and thus directly depend on the norms of the weight tensors, instead of depending on the norms of the matrix parametrizing the linear (convolutional) map.
Revised: These bounds incorporate weight sharing and thus directly depend on the norms of the weight tensors, instead of depending on the norms of the matrix that parametrizes the linear (convolutional) map.

Original: This results in an improved scaling with respect to the spatial input width.
Revised: This results in an improved scaling with respect to the spatial input width.

Original: In general, the bounds in [26] scale similarly to our bound ( ♣ ) from Theorem 3.5 in that they depend on the square root of the product of Lipschitz constants (or empirical estimates thereof).
Revised: In general, the bounds in [26] scale similarly to our bound ( ♣ ) from Theorem 3.5 in that they depend on the square root of the product of Lipschitz constants (or empirical estimates thereof).

Original: In particular, just as our result ( ♣ ), [26, Theorem 16] is based on Rademacher complexity bounds for function classes F γ , i.e., the composition of Lipschitz- and distance-constrained convolutional networks with the ramp loss at margin γ > 0 .
Revised: In particular, just as our result ( ♣ ), [26, Theorem 16] is based on Rademacher complexity bounds for function classes F γ , i.e., the composition of Lipschitz- and distance-constrained convolutional networks with the ramp loss at margin γ > 0 .

Original: The main result [26, Theorem 3], as well as [26, Theorem 20], adapts techniques from [43] and [34] to replace the product of Lipschitz constants with empirical equivalents, which are typically much smaller.
Revised: The main result [26, Theorem 3], as well as [26, Theorem 20], adapts techniques from [43] and [35] to replace the product of Lipschitz constants with empirical equivalents, which are typically much smaller.

Original: To this end, they study the composition of convolutional networks with an augmented loss function, see for example [26, Eq.
Revised: To this end, they study the composition of convolutional networks with an augmented loss function, see for example [26, Eq.

Original: In this part of the appendix, we compare our norm-driven bound ( ♣ ) with the main results in [26].
Revised: In this part of the appendix, we compare our norm-driven bound ( ♣ ) with the main results in [26].

Original: As mentioned in Section 3.2, we find that both results exhibit similar scaling behavior, but we improve in the logarithmic term and in that our dependency on data norms is less sensitive to outliers .
Revised: As mentioned in Section 3.2, we find that both results exhibit similar scaling behavior, but we improve in the logarithmic term and in that our dependency on data norms is less sensitive to outliers .

Original: On the other hand, the main bounds in [26] exhibit an improved dependency on the number of classes.
Revised: On the other hand, the main bounds in [26] exhibit an improved dependency on the number of classes.

Original: The latter pays off, e.g., for shallow networks or in extreme multiclass problems with a large number of classes.
Revised: The latter pays off, e.g., for shallow networks or in extreme multiclass problems with a large number of classes.

Original: All three effects are due to the use of l 2- vs. l ∞ -covering numbers.
Revised: All three effects are due to the use of l 2 vs. l ∞ covering numbers.

Original: Central to all Rademacher complexity bounds [26] is the single-layer l ∞ -covering number bound restated in the proposition below.
Revised: Central to all Rademacher complexity bounds [26] is the single-layer l ∞ covering number bound restated in the proposition below.

Original: Proposition A.1 ([26, Proposition 6]) .
Revised: Proposition A.1 ([26, Proposition 6]) .

Original: Let positive reals ( a, b, ϵ ) and positive integer m be given.
Revised: Let positive reals ( a, b, ϵ ) and positive integer m be given.

Original: Let the tensor X ∈ R n × U ×
Revised: Let the tensor X ∈ R n × U ×

Original: d
Revised: d

Original: ′ be given with ∀ i ∈ { 1 , . .
Revised: ′ be given with ∀ i ∈ { 1 , . .

Original: . , n } , ∀ u ∈ { 1 , . .
Revised: . , n } , ∀ u ∈ { 1 , . .

Original: , U } , ∥ X iu · ∥ 2 ≤ b .
Revised: , U } , ∥ X iu · ∥ 2 ≤ b .

Original: For any fixed M :
Revised: For any fixed M :

Original: [Equation]
Revised: [Equation]

Original: log(cid:20)(cid:18)
Revised: log(cid:20)(cid:18)

Original: [Equation]
Revised: [Equation]

Original: ab ϵ +(cid:19) mnU (cid:21) (34) with the norm ∥·∥ ∗ over the space R n × U × m defined by ∥ Y ∥ ∗ = max i ≤ n max j ≤ U ( (cid:80) mk =1 Y 2 ijk ) 1 / 2 .
Revised: ab ϵ +(cid:19) mnU (cid:21) (34) with the norm ∥·∥ ∗ over the space R n × U × m defined by ∥ Y ∥ ∗ = max i ≤ n max j ≤ U ( (cid:80) mk =1 Y 2 ijk ) 1 / 2 .

Original: Some remarks regarding the notation.
Revised: Some remarks regarding the notation.

Original: Here, X does not denote the input data ( x 1 , . . .
Revised: Here, X does not denote the input data ( x 1 , . . .

Original: , x n ) , but the nU -tuple of all d ′ -sized convolutional patches of the input data.
Revised: , x n ) , but the nU -tuple of all d ′ -sized convolutional patches of the input data.

Original: Thus, d ′ =
Revised: Thus, d ′ =

Original: k 2 c in is the square of the kernel size times the number of input channels and U =
Revised: k 2 c in is the square of the kernel size times the number of input channels and U =

Original: ⌈ d/t ⌉ 2 is the number of patches per image, which is computed as the square of the spatial width divided by the stride.
Revised: ⌈ d/t ⌉ 2 is the number of patches per image, which is computed as the square of the spatial width divided by the stride.

Original: The matrix
Revised: The matrix

Original: A ∈ R k 2 c in × c out then is the local linear map acting on the convolutional patches, i.e., A is a reshaping of the weight tensor K and XA is the output of the convolutional layer, i.e., n images with c out channels with U pixels each.
Revised: A ∈ R k 2 c in × c out then is the local linear map acting on the convolutional patches, i.e., A is a reshaping of the weight tensor K and XA is the output of the convolutional layer, i.e., n images with c out channels with U pixels each.

Original: Further, ∥ A ∥ 2 , 1 is the standard matrix (2 , 1) group norm which differs from ∥ K ∥ 2 , 1 defined in Eq.
Revised: Further, ∥ A ∥ 2 , 1 is the standard matrix (2 , 1) group norm which differs from ∥ K ∥ 2 , 1 defined in Eq.

Original: As the single-layer bound in Eq.
Revised: As the single-layer bound in Eq.

Original: (34) and our single-layer bound in Eq.
Revised: (34) and our single-layer bound in Eq.

Original: (54) are the fundamental building blocks of all inferred results (and we did not study augmented loss functions), we will focus on them for the comparison.
Revised: (54) are the fundamental building blocks of all inferred results (and we did not study augmented loss functions), we will focus on them for the comparison.

Original: For ease of reference, we restate the relevant part of Theorem 3.2.
Revised: For ease of reference, we restate the relevant part of Theorem 3.2.

Original: Theorem A.2.
Revised: Theorem A.2.

Original: Let b > 0 and F = { ϕ K | K ∈ R c out × c in × k × k , ∥ K ∥ 2 , 1 ≤ b } denote the class of 2D convolutions with c in input channels, c out output channels and kernel size k × k , parametrized by tensors K with W parameters.
Revised: Let b > 0 and F = { ϕ K | K ∈ R c out × c in × k × k , ∥ K ∥ 2 , 1 ≤ b } denote the class of 2D convolutions with c in input channels, c out output channels and kernel size k × k , parametrized by tensors K with W parameters.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: There is a clear similarity between Eq.
Revised: There is a clear similarity between Eq.

Original: (34) and Eq.
Revised: (34) and Eq.

Original: Both depend quadratically on weight and data norms divided by the covering radius ϵ , as well as on a logarithmic term.
Revised: Both depend quadratically on weight and data norms divided by the covering radius ϵ , as well as on a logarithmic term.

Original: Consequently, differences between both bounds are nuanced and, ignoring the constant in Eq.
Revised: Consequently, differences between both bounds are nuanced and, ignoring the constant in Eq.

Original: (34), it is a priori not clear which bound is preferable.
Revised: (34), it is a priori not clear which bound is preferable.

Original: We will discuss these nuances theoretically and provide a empirical comparison in Fig.
Revised: We will discuss these nuances theoretically and provide a empirical comparison in Fig.

Original: (Diff-1) Data norms.
Revised: (Diff-1) Data norms.

Original: Our work assumes a bound on the l 2 norm of the whole input x (a c in d 2 -tuple), whereas [26] only assumes a bound on the l 2 norm of every single patch p ( c in k 2 -tuples).
Revised: Our work assumes a bound on the l 2 norm of the whole input x (a c in d 2 -tuple), whereas [26] only assumes a bound on the l 2 norm of every single patch p ( c in k 2 -tuples).

Original: This potentially improves Eq.
Revised: This potentially improves Eq.

Original: (34) over Eq.
Revised: (34) over Eq.

Original: (35) by a factor of ( k/d ) 2 , as
Revised: (35) by a factor of ( k/d ) 2 , as

Original: [Equation]
Revised: [Equation]

Original: The left inequality is obvious.
Revised: The left inequality is obvious.

Original: The right inequality follows from considering the sum of all patch norms.
Revised: The right inequality follows from considering the sum of all patch norms.

Original: As every pixel x ijk appears in at least ⌊ k/s ⌋ 2 patches
Revised: As every pixel x ijk appears in at least ⌊ k/s ⌋ 2 patches

Original: and there are at most ⌈ d/s ⌉ 2 patches in total, it holds that
Revised: and there are at most ⌈ d/s ⌉ 2 patches in total, it holds that

Original: [Equation]
Revised: [Equation]

Original: and ⌊ k/s ⌋ / ⌈ d/s ⌉ ≈ k/d .
Revised: and ⌊ k/s ⌋ / ⌈ d/s ⌉ ≈ k/d .

Original: Notably, the maximum in
Revised: Notably, the maximum in

Original: Eq.
Revised: Eq.

Original: (34) is over the patches on all of the input data, which is quite sensitive to outliers .
Revised: (34) is over the patches on all of the input data, which is quite sensitive to outliers .

Original: Hence, the improvement over Eq.
Revised: Hence, the improvement over Eq.

Original: (35) is typically smaller than d/k , especially at hidden layers, see top row of Fig.
Revised: (35) is typically smaller than d/k , especially at hidden layers, see top row of Fig.

Original: 5. (Diff-2)
Revised: 5. (Diff-2)

Original: Weight norms.
Revised: Weight norms.

Original: The (2 , 1) group norms on the weights are applied differently, i.e., we compute a (2 , 1) norm via Eq.
Revised: The (2 , 1) group norms on the weights are applied differently, i.e., we compute a (2 , 1) norm via Eq.

Original: (7), whereas [26] computes the (2 , 1) group norm of the matrix corresponding to the local linear map, which is applied to each patch, i.e., (cid:80) i
Revised: (7), whereas [26] computes the (2 , 1) group norm of the matrix corresponding to the local linear map, which is applied to each patch, i.e., (cid:80) i

Original: ∥ K i ··· ∥ 2 .
Revised: ∥ K i ··· ∥ 2 .

Original: As
Revised: As

Original: [Equation]
Revised: [Equation]

Original: this potentially improves Eq.
Revised: this potentially improves Eq.

Original: (34) over Eq.
Revised: (34) over Eq.

Original: (35) by a factor of 1 /k 2 .
Revised: (35) by a factor of 1 /k 2 .

Original: Empirically, we observe that ∥ K ∥ 2 , 1 ≈ k (cid:80) i ∥ K i ··· ∥ 2 , see bottom row of Fig.
Revised: Empirically, we observe that ∥ K ∥ 2 , 1 ≈ k (cid:80) i ∥ K i ··· ∥ 2 , see bottom row of Fig.

Original: Thus, considering norm constraints only, i.e., (Diff-1) and (Diff-2), we find that Eq.
Revised: Thus, considering norm constraints only, i.e., (Diff-1) and (Diff-2), we find that Eq.

Original: (34) is potentially better by a factor ( k/d ) 2 · (1 /k ) 2 = 1 /d 2 , i.e., the reciprocal of the squared height/width of the input images.
Revised: (34) is potentially better by a factor ( k/d ) 2 · (1 /k ) 2 = 1 /d 2 , i.e., the reciprocal of the squared height/width of the input images.

Original: However, the comparison is more intricate, as the coverings are with respect to different ( lvs.
Revised: However, the comparison is more intricate, as the coverings are with respect to different ( lvs.

Original: l ∞ ) norms and, more importantly, the considered function classes differ.
Revised: l ∞ ) norms and, more importantly, the considered function classes differ.

Original: As, ultimately, we want to get Rademacher complexity bounds for whole networks, we need to consider effects that appear when transitioning to whole-network bounds.
Revised: As, ultimately, we want to get Rademacher complexity bounds for whole networks, we need to consider effects that appear when transitioning to whole-network bounds.

Original: (Diff-3) Lipschitz constants.
Revised: (Diff-3) Lipschitz constants.

Original: In whole-network bounds, contributions of all layers are summed.
Revised: In whole-network bounds, contributions of all layers are summed.

Original: These contributions are the (logarithmic) single-layer bounds, scaled by a factor corresponding to the Lipschitz constant of the remainder of a network after the layer.
Revised: These contributions are the (logarithmic) single-layer bounds, scaled by a factor corresponding to the Lipschitz constant of the remainder of a network after the layer.

Original: Typically, the
Revised: Typically, the

Original: Lipschitz constant of the part before a layer additionally enters as an estimate of the norm of the layer’s input.
Revised: Lipschitz constant of the part before a layer additionally enters as an estimate of the norm of the layer’s input.

Original: Notably, in [26], the Lipschitz constant of the network’s remaining layers incurs an additional factor ⌈ d/t ⌉ 2 , i.e., the spatial dimensionality of the output (denoted by w l in the reference).
Revised: Notably, in [26], the Lipschitz constant of the network’s remaining layers incurs an additional factor ⌈ d/t ⌉ 2 , i.e., the spatial dimensionality of the output (denoted by w l in the reference).

Original: This counterbalances the improvements by (Diff-1) and (Diff-2).
Revised: This counterbalances the improvements by (Diff-1) and (Diff-2).

Original: Specifically, in [26], the Lipschitz constants are with respect to the norms ∥·∥ ∞ ,r on the domain and | · | s on the codomain, see, e.g., the definition of ρ A l 1 → l 2 in the statement of [26,
Revised: Specifically, in [26], the Lipschitz constants are with respect to the norms ∥·∥ ∞ ,r on the domain and | · | s on the codomain, see, e.g., the definition of ρ A l 1 → l 2 in the statement of [26,

Original: Proposition 10].
Revised: Proposition 10].

Original: There,
Revised: There,

Original: [Equation]
Revised: [Equation]

Original: (cid:118)(cid:117)(cid:117)(cid:116)
Revised: (cid:118)(cid:117)(cid:117)(cid:116)

Original: [Equation]
Revised: [Equation]

Original: is the maximum l 2 norm of a slice of the image x along the channels, i.e., at fixed spatial position.
Revised: is the maximum l 2 norm of a slice of the image x along the channels, i.e., at fixed spatial position.

Original: The norm | y | s = max p ∈ patches ∥
Revised: The norm | y | s = max p ∈ patches ∥

Original: p ∥ is the maximal norm of a convolutional patch on y .
Revised: p ∥ is the maximal norm of a convolutional patch on y .

Original: Transitioning to Lipschitz constants with respect to l 2 norms, i.e., spectral norms, as done for the main result in [26, Theorem 3], incurs an additional factor
Revised: Transitioning to Lipschitz constants with respect to l 2 norms, i.e., spectral norms, as done for the main result in [26, Theorem 3], incurs an additional factor

Original: d (the spatial dimension of x ), since
Revised: d (the spatial dimension of x ), since

Original: [Equation]
Revised: [Equation]

Original: In this inequality, x denotes the output of the considered layer and so d is its spatial width.
Revised: In this inequality, x denotes the output of the considered layer and so d is its spatial width.

Original: Notably, in [26]
Revised: Notably, in [26]

Original: d can actually be reduced to the output’s spatial width after a subsequent max-pooling operation.
Revised: d can actually be reduced to the output’s spatial width after a subsequent max-pooling operation.

Original: In our whole-network bound, the Lipschitz constant is already with respect to l 2 norms and thus no additional factors appear.
Revised: In our whole-network bound, the Lipschitz constant is already with respect to l 2 norms and thus no additional factors appear.

Original: (Diff-4) Dependency on number of classes.
Revised: (Diff-4) Dependency on number of classes.

Original: The use of l ∞ -covering numbers in [26] improves the dependency on the number of classes for whole-network bounds.
Revised: The use of l ∞ covering numbers in [26] improves the dependency on the number of classes for whole-network bounds.

Original: This is because the weights of the classification layer do not enter via a (2 , 1) group norm constraint, but a Frobenius norm constraint.
Revised: This is because the weights of the classification layer do not enter via a (2 , 1) group norm constraint, but a Frobenius norm constraint.

Original: This implicitly improves the covering number of this layer by a factor of the number of classes.
Revised: This implicitly improves the covering number of this layer by a factor of the number of classes.

Original: Since, for whole-network bounds, the contribution of all layers are summed, we expect this effect to be significant if the contribution of the classification is a substantial fraction of the whole-network bound.
Revised: Since, for whole-network bounds, the contribution of all layers are summed, we expect this effect to be significant if the contribution of the classification is a substantial fraction of the whole-network bound.

Original: This would be the case, e.g., for shallow networks or in extreme multiclass settings.
Revised: This would be the case, e.g., for shallow networks or in extreme multiclass settings.

Original: Finally, we discuss the logarithmic terms and constants.
Revised: Finally, we discuss the logarithmic terms and constants.

Original: (Diff-5) Logarithmic terms.
Revised: (Diff-5) Logarithmic terms.

Original: Our bound in Eq.
Revised: Our bound in Eq.

Original: (35) depends logarithmically on the number of parameters, denoted by W .
Revised: (35) depends logarithmically on the number of parameters, denoted by W .

Original: By contrast, Eq.
Revised: By contrast, Eq.

Original: (34) depends on log 2 (cid:2)(cid:0) 8 abϵ + 7 (cid:1) mnU (cid:3) .
Revised: (34) depends on log 2 (cid:2)(cid:0) 8 abϵ + 7 (cid:1) mnU (cid:3) .

Original: When transitioning to Rademacher complexity bounds via Dudley’s entropy integral (cf.
Revised: When transitioning to Rademacher complexity bounds via Dudley’s entropy integral (cf.

Original: [26, Eq.
Revised: [26, Eq.

Original: (29)]), the covering radius ϵ in the log 2 term is replaced by 1 n .
Revised: (29)]), the covering radius ϵ in the log 2 term is replaced by 1 n .

Original: So, considering the definitions of U and m , we need to compare 8 abmUn 2 = 8 abc out d 2 n 2 /t 

Revised: So, considering the definitions of U and m , we need to compare 8 abmUn 2 = 8 abc out d 2 n 2 /t

Original: (Ledent et al.) with 2 W = 2 c in c out k 2 (Ours).
Revised: (Ledent et al.) with 2 W = 2 c in c out k 2 (Ours).

Original: As, typically, k ≤ d/t , we improve over [26] in the logarithmic term.
Revised: As, typically, k ≤ d/t , we improve over [26] in the logarithmic term.

Original: (Diff-6) Multiplicative constants.
Revised: (Diff-6) Multiplicative constants.

Original: The single-layer bound by Ledent et al.
Revised: The single-layer bound by Ledent et al.

Original: Eq.
Revised: Eq.

Original: (34) has a rather large multiplicative constant 64 (compared to 1 in Eq.
Revised: (34) has a rather large multiplicative constant 64 (compared to 1 in Eq.

Original: This constant enters mainly via a previous theorem by Zhang [47, Theorem 4].
Revised: This constant enters mainly via a previous theorem by Zhang [47, Theorem 4].

Original: Notably, a remark in [47] highlights that
Revised: Notably, a remark in [47] highlights that

Original: [Figure]
Revised: [Figure]

Original: ·· ∥
Revised: ·· ∥

Original: [Figure]
Revised: [Figure]

Original: in
Revised: in

Original: [Figure]
Revised: [Figure]

Original: r i
Revised: r i

Original: [Figure]
Revised: [Figure]

Original: t ∥
Revised: t ∥

Original: [Figure]
Revised: [Figure]

Original: c
Revised: c

Original: [Figure]
Revised: [Figure]

Original: C o m p a r i s on w r t .
Revised: C o m p a r i s on w r t .

Original: d a t a n o r m s
Revised: d a t a n o r m s

Original: C o m p a r i s on w r t .
Revised: C o m p a r i s on w r t .

Original: w e i g h t n o r m s
Revised: w e i g h t n o r m s

Original: [Figure]
Revised: [Figure]

Original: √ s p a ti a l ou t pu t
Revised: √ s p a ti a l ou t pu t

Original: w i d t h the constants in this theorem are not optimized.
Revised: w i d t h the constants in this theorem are not optimized.

Original: Thus, improving Eq.
Revised: Thus, improving Eq.

Original: (34) in this regard might be possible, and the difference in numerical constants might be less pronounced than it appears at first sight.
Revised: (34) in this regard might be possible, and the difference in numerical constants might be less pronounced than it appears at first sight.

Original: Yet, from our understanding, some constants are unavoidable, e.g., the factor 2 which enters the proof sketch of [26, Proposition 6].
Revised: Yet, from our understanding, some constants are unavoidable, e.g., the factor 2 which enters the proof sketch of [26, Proposition 6].

Original: Overall, (Diff-1) – (Diff-3) lead to several effects, which can potentially compensate each other, especially if, for each layer, the coordinates of its input and of the weights have roughly equal norm.
Revised: Overall, (Diff-1) – (Diff-3) lead to several effects, which can potentially compensate each other, especially if, for each layer, the coordinates of its input and of the weights have roughly equal norm.

Original: Notably, in this situation, our single-layer bound can be improved by a factor of 1 /t 2 , with t denoting the stride of the convolution, see Remark C.9.
Revised: Notably, in this situation, our single-layer bound can be improved by a factor of 1 /t 2 , with t denoting the stride of the convolution, see Remark C.9.

Original: Thus, in the absence of pooling (e.g., when downsampling is handled directly by the stride of the convolutional layer), the scaling is precisely the same.
Revised: Thus, in the absence of pooling (e.g., when downsampling is handled directly by the stride of the convolutional layer), the scaling is precisely the same.

Original: As Eq.
Revised: As Eq.

Original: (34) depends on the maximum norm of a patch over all of the input data (i.e., a quantity which is sensitive to outliers), we do not expect (Diff-1) – (Diff-3) to fully compensate each other, but rather expect an advantage of our bound from Theorem A.2.
Revised: (34) depends on the maximum norm of a patch over all of the input data (i.e., a quantity which is sensitive to outliers), we do not expect (Diff-1) – (Diff-3) to fully compensate each other, but rather expect an advantage of our bound from Theorem A.2.

Original: A detailed investigation is shown in Fig.
Revised: A detailed investigation is shown in Fig.

Original: 5, which highlights weight and data norms for layers of a trained network (an 11-layer convolutional network as used for Fig. 4).
Revised: 5, which highlights weight and data norms for layers of a trained network (an 11-layer convolutional network as used for Fig. 4).

Original: To incorporate the effects (Diff-1) – (Diff-3) and to allow for a cleaner comparison, we multiply the l 2 norms of the patches by the square root of spatial dimensionality of the output and shift a factor of the kernel size from our weight norms to our data norms.
Revised: To incorporate the effects (Diff-1) – (Diff-3) and to allow for a cleaner comparison, we multiply the l 2 norms of the patches by the square root of spatial dimensionality of the output and shift a factor of the kernel size from our weight norms to our data norms.

Original: We see that the (rescaled) weight norms across all layers are essentially the same, whereas, due to the maximum being sensitive to outliers, our weight norms are substantially smaller at the hidden layers.
Revised: We see that the (rescaled) weight norms across all layers are essentially the same, whereas, due to the maximum being sensitive to outliers, our weight norms are substantially smaller at the hidden layers.

Original: [Figure]
Revised: [Figure]

Original: As a last comparison, we illustrate the magnitudes of all factors appearing in the bounds of Eqs.
Revised: As a last comparison, we illustrate the magnitudes of all factors appearing in the bounds of Eqs.

Original: (34) and (35) and of the spatial dimensionality of the output, see (Diff-3) in the enumeration above.
Revised: (34) and (35) and of the spatial dimensionality of the output, see (Diff-3) in the enumeration above.

Original: In both bounds, we discard the denominator ϵ 2 and, in
Revised: In both bounds, we discard the denominator ϵ 2 and, in

Original: Eq.
Revised: Eq.

Original: (34), we replace the factor 1 /ϵ in the logarithm by n , just as it enters the Rademacher complexity bounds.
Revised: (34), we replace the factor 1 /ϵ in the logarithm by n , just as it enters the Rademacher complexity bounds.

Original: As can be seen from the figure to the right (with our single-layer bound marked by ■ ), our improvement in the quadratic terms is due to data norms.
Revised: As can be seen from the figure to the right (with our single-layer bound marked by ■ ), our improvement in the quadratic terms is due to data norms.

Original: To be specific, one needs to compare the data norms in our case, to the combination of data norms and spatial dimensions in the bound of Ledent et al.
Revised: To be specific, one needs to compare the data norms in our case, to the combination of data norms and spatial dimensions in the bound of Ledent et al.

Original: We also improve in the logarithmic terms and constants.
Revised: We also improve in the logarithmic terms and constants.

Original: A.6.1 Comparison for two-layer networks
Revised: Comparison for two-layer networks

Original: As discussed in (Diff-4), the l ∞ covering approach in [26] allows for a favorable treatment of the last (classification) layer.
Revised: As discussed in (Diff-4), the l ∞ covering approach in [26] allows for a favorable treatment of the last (classification) layer.

Original: If the contribution of this last layer to the respective Rademacher complexity bounds is substantial, then the bound in [26] is superior.
Revised: If the contribution of this last layer to the respective Rademacher complexity bounds is substantial, then the bound in [26] is superior.

Original: We evaluate this effect on two-layer networks, where it is most pronounced.
Revised: We evaluate this effect on two-layer networks, where it is most pronounced.

Original: The comparison considers network of the following architecture.
Revised: The comparison considers networks of the following architecture.

Original: The first layer is convolutional and parametrized by a tensor K ∈ R c × 3 × k × k .
Revised: The first layer is convolutional and parametrized by a tensor K ∈ R c × 3 × k × k .

Original: Here c denotes the number of filters (channels of the output) and k the kernel size, which is chosen equal to the stride and the spatial dimensionality of the input, e.g., 32 for images from CIFAR100.
Revised: Here c denotes the number of filters (channels of the output) and k the kernel size, which is chosen equal to the stride and the spatial dimensionality of the input, e.g., 32 for images from CIFAR100.

Original: Consequently, the spatial dimensionality of the output is 1.
Revised: Consequently, the spatial dimensionality of the output is 1.

Original: This convolutional layer is followed by an activation function with Lipschitz constant 1 (e.g.,
Revised: This convolutional layer is followed by an activation function with Lipschitz constant 1 (e.g.,

Original: ReLU) and a linear map W ∈ R κ × c , with κ denoting the number of classes.
Revised: ReLU) and a linear map W ∈ R κ × c , with κ denoting the number of classes.

Original: Since the quantities and norms appearing in the respective bounds differ, we make the following simplifications, which are motivated by corresponding inequalities and verified empirically.
Revised: Since the quantities and norms appearing in the respective bounds differ, we make the following simplifications, which are motivated by corresponding inequalities and verified empirically.

Original: (1) ∥ K ∥ 2 , 1 ≈ k (cid:80) ci =

Revised: (1) ∥ K ∥ 2 , 1 ≈ k (cid:80) ci =

Original: ∥ K i ··· ∥ 2 , (2) (cid:13)(cid:13) W ⊤ (cid:13)(cid:13) 2 , 1 ≈ √ c
Revised: ∥ K i ··· ∥ 2 , (2) (cid:13)(cid:13) W ⊤ (cid:13)(cid:13) 2 , 1 ≈ √ c

Original: ∥ W ∥ 2 , (3) max j ≤ κ ∥ W j · ∥ 2 ≈ Lip( W ) , (4)
Revised: ∥ W ∥ 2 , (3) max j ≤ κ ∥ W j · ∥ 2 ≈ Lip( W ) , (4)

Original: ∥
Revised: ∥

Original: X ∥ 2 √ n ≈ max i ≤ n ∥ x i ∥ 2 = maximal norm of convolutional patches from the data , (5) H n − 1 ≈ log n .
Revised: X ∥ 2 √ n ≈ max i ≤ n ∥ x i ∥ 2 = maximal norm of convolutional patches from the data , (5) H n − 1 ≈ log n .

Original: Furthermore, just as the single-layer bound in [26] depends on the maximal norm of a patch of the data, ours actually depends only on the maximal norm of particular slices of the data, which we here denote as | X | s .
Revised: Furthermore, just as the single-layer bound in [26] depends on the maximal norm of a patch of the data, ours actually depends only on the maximal norm of particular slices of the data, which we here denote as | X | s .

Original: In the special case of the stride being equal to the spatial dimensionality, these slices are over the channels at fixed local position (see Remark C.9 and the last chain of inequalities in the proof of Theorem C.7).
Revised: In the special case of the stride being equal to the spatial dimensionality, these slices are over the channels at fixed local position (see Remark C.9 and the last chain of inequalities in the proof of Theorem C.7).

Original: Thus, for the contribution of the first layer, we can use (
Revised: Thus, for the contribution of the first layer, we can use (

Original: [Equation]
Revised: [Equation]

Original: | X | s ≈ ∥ X ∥ k .

Original: Last, we empirically evaluate a maximum operator which appears in the quantity R in the two layer bound (Theorem 2) in [26], i
Revised: Last, we empirically evaluate a maximum operator which appears in the quantity R in the two layer bound (Theorem 2) in [26], i

Original: [Equation]
Revised: [Equation]

Original: e.,
Revised: e.,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: With these simplifications
Revised: With these simplifications

Original: [Equation]
Revised: [Equation]

Original: our bound ( ♣ ) becomes
Revised: our bound ( ♣ ) becomes

Original: [Equation]
Revised: [Equation]

Original: and the bound from Theorem 2 in [26] becomes
Revised: and the bound from Theorem 2 in [26] becomes

Original: [Equation]
Revised: [Equation]

Original: with D = max
Revised: with D = max

Original: [Equation]
Revised: [Equation]

Original:  As expected, ignoring constants and log terms, the bound from [26] is better by a factor of √ κ (square root of number of classes) in the summand corresponding to the last layer.
Revised: (cid:33) . As expected, ignoring constants and log terms, the bound from [26] is better by a factor of √ κ (square root of number of classes) in the summand corresponding to the last layer.

Original: Empirically , we evaluate the bounds for networks of varying width c ∈ { 32 , 1024 , 8192 } trained on CIFAR100.
Revised: Empirically , we evaluate the bounds for networks of varying width c ∈ { 32 , 1024 , 8192 } trained on CIFAR100.

Original: Here, we compute exact values and do not use the simplifications (1) - (7).
Revised: Here, we compute exact values and do not use the simplifications (1) - (7).

Original: As the models do not fit the training data, we use a margin parameter of γ = 1 for simplicity.
Revised: As the models do not fit the training data, we use a margin parameter of γ = 1 for simplicity.

Original: Overall, the models performed rather poorly (as expected) with testing accuracies of 21%, 29%, 31% and training accuracies 32%, 99.8%, 99.9%.
Revised: Overall, the models performed rather poorly (as expected) with testing accuracies of 21%, 29%, 31% and training accuracies 32%, 99.8%, 99.9%.

Original: Table 4 lists the computed values of the bounds and how they distribute over the respective factors (weight & data norms, logarithmic term, numerical constant, sample size dependency).
Revised: Table 4 lists the computed values of the bounds and how they distribute over the respective factors (weight & data norms, logarithmic term, numerical constant, sample size dependency).

Original: Results are presented on a logarithmic scale with base 10.
Revised: Results are presented on a logarithmic scale with base 10.

Original: Overall, we observe the following effects:
Revised: Overall, we observe the following effects:

Original: Relatively, the contribution of the second layer in [26] is improved by a factor of 10.
Revised: Relatively, the contribution of the second layer in [26] is improved by a factor of 10.

Original: This is expected, as 10 is the square root of the number of classes in the CIFAR100 dataset.
Revised: This is expected, as 10 is the square root of the number of classes in the CIFAR100 dataset.

Original: The wider the network, the more dominant the term corresponding to the first layer becomes.
Revised: The wider the network, the more dominant the term corresponding to the first layer becomes.

Original: At width 32, the factor from weight and data norms of [26] is clearly superior.
Revised: At width 32, the factor from weight and data norms of [26] is clearly superior.

Original: This is due to the improved class dependency.
Revised: This is due to the improved class dependency.

Original: However, for a width of 1024, this effect is already negligible.
Revised: However, for a width of 1024, this effect is already negligible.

Original: For the wider networks ( c ∈ { 1024 m 8192 } ), we improve over [26] by a factor of approximately 10 1 . 5 ≈ 30 .
Revised: For the wider networks ( c ∈ { 1024 , 8192 } ), we improve over [26] by a factor of approximately 10 1 . 5 ≈ 30 .

Original: Ignoring numerical constants, we improve by a factor 10 0 .
Revised: Ignoring numerical constants, we improve by a factor 10 0 .

Original: 3 ≈ 2 , which is due to an improvement in the logarithmic term.
Revised: 3 ≈ 2 , which is due to an improvement in the logarithmic term.

Original: For fixed width, the bounds and factors do not vary over the random initializations.
Revised: For fixed width, the bounds and factors do not vary over the random initializations.

Original: For the models with widths 1024 and 8192, the standard deviation of the base 10 logarithms are < 0 .
Revised: For the models with widths 1024 and 8192, the standard deviation of the base 10 logarithms are < 0 .

Original: 005 , which corresponds to a geometric standard deviation of less than a factor 1 .
Revised: 005 , which corresponds to a geometric standard deviation of less than a factor 1 .

Original: 01 , i.e., 1%.
Revised: 01 , i.e., 1%.

Original: Last, we consider a network whose first layer has kernel size 3 and stride 1.
Revised: Last, we consider a network whose first layer has kernel size 3 and stride 1.

Original: Reduction of the spatial dimensionality is achieved by a subsequent max pooling layer of window size 32 × 32 .
Revised: Reduction of the spatial dimensionality is achieved by a subsequent max pooling layer of window size 32 × 32 .

Original: This setting favors [26] as this work can better account for the pooling layer.
Revised: This setting favors [26] as this work can better account for the pooling layer.

Original: In our simplified bound of Eq.
Revised: In our simplified bound of Eq.

Original: (38), the factor 1 /k in the first summand disappears, since k = 1 ; their result improves due to the now smaller convolutional patches.
Revised: (38), the factor 1 /k in the first summand disappears, since k = 1 ; their result improves due to the now smaller convolutional patches.

Original: In this setting the contribution of the first layer’s weight and data norms is clearly larger in our bound.
Revised: In this setting the contribution of the first layer’s weight and data norms is clearly larger in our bound.

Original: Yet, our bound is still smaller, but only due to numerical constants.
Revised: Yet, our bound is still smaller, but only due to numerical constants.

Original: [Table]
Revised: [Table]

Original: B Additional Experiments B.1 Excess capacity in non-residual networks
Revised: B Additional Experiments B.1 Excess capacity in non-residual networks

Original: In addition to the experiments presented in Section 4 of the main manuscript, we performed the same excess capacity experiments on a non-residual convolutional network.
Revised: In addition to the experiments presented in Section 4 of the main manuscript, we performed the same excess capacity experiments on a non-residual convolutional network.

Original: Architecture.
Revised: Architecture.

Original: Essentially, we rely on the same 11-layer convolutional network with ReLU activations as described in Section A.3, only that we substitute each convolutional layer with stride 2 by a convolutional layer with stride 1 followed by a max-pooling layer with kernel size 3 and stride 2.
Revised: Essentially, we rely on the same 11-layer convolutional network with ReLU activations as described in Section A.3, only that we substitute each convolutional layer with stride 2 by a convolutional layer with stride 1 followed by a max-pooling layer with kernel size 3 and stride 2.

Original: This is done so that we can enforce the constraints on the capacity-driving quantities via the approach described in Section B.2.
Revised: This is done so that we can enforce the constraints on the capacity-driving quantities via the approach described in Section B.2.

Original: Consistent with our ResNet18 experiments, the linear classifier is fixed with weights set to the vertices of a ( #classes − 1) unit simplex in the output space of the network and kernel sizes of the convolutional layers are not larger than the width of their input.
Revised: Consistent with our ResNet18 experiments, the linear classifier is fixed with weights set to the vertices of a ( #classes − 1) unit simplex in the output space of the network and kernel sizes of the convolutional layers are not larger than the width of their input.

Original: Datasets & Training.
Revised: Datasets & Training.

Original: Experiments are performed on the CIFAR10 and CIFAR100 benchmark datasets [25].
Revised: Experiments are performed on the CIFAR10 and CIFAR100 benchmark datasets [25].

Original: We minimize the cross-entropy loss using SGD with momentum (0.9) for 200 epochs with batch size 256 and decay the initial learning rate (of 3e-2) with a cosine annealing scheduler after each epoch.
Revised: We minimize the cross-entropy loss using SGD with momentum (0.9) for 200 epochs with batch size 256 and decay the initial learning rate (of 3e-2) with a cosine annealing scheduler after each epoch.

Original: No data augmentation is used .
Revised: No data augmentation is used .

Original: For projecting onto the constraint sets, we perform one alternating projection step every 10th SGD update.
Revised: For projecting onto the constraint sets, we perform one alternating projection step every 10th SGD update.

Original: After the final SGD update, we additionally do 15 alternating projection steps to ensure that the trained model is within the capacity-constrained class.
Revised: After the final SGD update, we additionally do 15 alternating projection steps to ensure that the trained model is within the capacity-constrained class.

Original: Results.
Revised: Results.

Original: We observe similar phenomena as for the residual (ResNet18) network studied in Section 4.
Revised: We observe similar phenomena as for the residual (ResNet18) network studied in Section 4.

Original: When comparing models trained with and without constraint, we see a substantial amount of excess capacity, and this excess capacity increases with task difficulty.
Revised: When comparing models trained with and without constraint, we see a substantial amount of excess capacity, and this excess capacity increases with task difficulty.

Original: In fact, compared to our results with the residual network architecture, this effect is even more pronounced as the capacity-driving quantities in the unconstrained setting are surprisingly large.
Revised: In fact, compared to our results with the residual network architecture, this effect is even more pronounced as the capacity-driving quantities in the unconstrained setting are surprisingly large.

Original: For instance, the median Lipschitz constant of the model trained on CIFAR100 is 11.53 (cf.
Revised: For instance, the median Lipschitz constant of the model trained on CIFAR100 is 11.53 (cf.

Original: Table 5), compared to 2.17 for the ResNet results in Table 1.
Revised: Table 5), compared to 2.17 for the ResNet results in Table 1.

Original: Notably, the capacity-driving quantities can be drastically reduced without a loss of testing accuracy and the constraints can be chosen equally across datasets.
Revised: Notably, the capacity-driving quantities can be drastically reduced without a loss of testing accuracy and the constraints can be chosen equally across datasets.

Original: This is similar to
Revised: This is similar to

Original: Section 4 where constraints are not precisely equal, but within a small range.
Revised: Section 4 where constraints are not precisely equal, but within a small range.

Original: We also observe another manifestation of task difficulty: tightening both constraints beyond the identified operating point leads to a more rapid deterioration of the testing error as task difficulty increases (Fig. 6, middle).
Revised: We also observe another manifestation of task difficulty: tightening both constraints beyond the identified operating point leads to a more rapid deterioration of the testing error as task difficulty increases (Fig. 6, middle).

Original: Different to Section 4, the constrained models (almost) fit the training data.
Revised: Different to Section 4, the constrained models (almost) fit the training data.

Original: However, under slightly stronger constraints, we can still find models with testing accuracy comparable (but slightly worse) to the unconstrained setting, but with noticeably less generalization gap (Fig. 6, bottom).
Revised: However, under slightly stronger constraints, we can still find models with testing accuracy comparable (but slightly worse) to the unconstrained setting, but with noticeably less generalization gap (Fig. 6, bottom).

Original: Again, this is primarily due to leaving the zero-training-error regime.
Revised: Again, this is primarily due to leaving the zero-training-error regime.

Original: We suspect that the constraints could be much stronger, but enforcing the constraints appears to more heavily influence optimization for networks without skip connections.
Revised: We suspect that the constraints could be much stronger, but enforcing the constraints appears to more heavily influence optimization for networks without skip connections.

Original: In this context, it is also worth pointing out that the constraints are quite strong for the non-residual network (proportionally much stronger than for the ResNet model in Section 4).
Revised: In this context, it is also worth pointing out that the constraints are quite strong for the non-residual network (proportionally much stronger than for the ResNet model in Section 4).

Original: During training, we projected after every 10th SGD step, which was actually not enough to enforce the constraints throughout the whole training procedure.
Revised: During training, we projected after every 10th SGD step, which was actually not enough to enforce the constraints throughout the whole training procedure.

Original: Only towards the end of training, when the learning rate is already small, do the constraints become satisfied.
Revised: Only towards the end of training, when the learning rate is already small, do the constraints become satisfied.

Original: Increasing the projection frequency might thus allow for even stronger constraints.
Revised: Increasing the projection frequency might thus allow for even stronger constraints.

Original: B.2 Projection method
Revised: B.2 Projection method

Original: A key aspect of the experiments in Section 4 is to obtain, for each pair of Lipschitz constant and (2,1) group norm constraints, a model with testing accuracy as high as possible.
Revised: A key aspect of the experiments in Section 4 is to obtain, for each pair of Lipschitz constant and (2,1) group norm constraints, a model with testing accuracy as high as possible.

Original: The quality of such a model depends, to a large extent, on the way the constraints are enforced.
Revised: The quality of such a model depends, to a large extent, on the way the constraints are enforced.

Original: This section specifies the projection method used for the experiments and provides additional background information.
Revised: This section specifies the projection method used for the experiments and provides additional background information.

Original: As mentioned in Section 4, we utilize orthogonal projections .
Revised: As mentioned in Section 4, we utilize orthogonal projections .

Original: Given x 0 ∈ R d and a nonempty closed convex set A ⊂ R d , the orthogonal projection of x 0 onto A is defined as the unique
Revised: Given x 0 ∈ R d and a nonempty closed convex set A ⊂ R d , the orthogonal projection of x 0 onto A is defined as the unique

Original: [Equation]
Revised: [Equation]

Original: Orthogonal projections have several beneficial properties.
Revised: Orthogonal projections have several beneficial properties.

Original: First, if f : R d → R is a strictly convex function, then, for appropriately chosen step sizes, projected gradient descent (i.e., gradient
Revised: First, if f : R d → R is a strictly convex function, then, for appropriately chosen step sizes, projected gradient descent (i.e., gradient

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: r

Original: [Table]
Revised: [Table]

Original: r o upn o r m d i s t a n ce c on s t r a i n t ( 2 , 1 ) g r o upn
Revised: upn o r m d i s t a n ce c on s t r a i n t ( 2 , 1 ) g r o upn

Original: o r m d
Revised: o r m d

Original: i s t a n ce c on s t r a i n t ( 2 , 1 ) g
Revised: i s t a n ce c on s t r a i n t ( 2 , 1 ) g

Original: r o upn
Revised: r o upn

Original: o r m d
Revised: o r m d

Original: i s t
Revised: i s t

Original: [Figure]
Revised: [Figure]

Original: c
Revised: c

Original: [Figure]
Revised: [Figure]

Original: t r a i n
Revised: t r [Figure] i n

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ) g r o upn o r m d
Revised: ) g r o upn o r m d

Original: i s t a n ce c on
Revised: i s t a n ce c on

Original: [Figure]
Revised: [Figure]

Original:  a i nt
Revised: t r a i n

Original: [Figure]
Revised: [Figure]

Original: O p e r a ti ng
Revised: r a ti ng

Original: po i n t descent with a subsequent orthogonal projection onto A after each step) converges to the minimizer arg min x ∈
Revised: po i n t descent with a subsequent orthogonal projection onto A after each step) converges to the minimizer arg min x ∈

Original: A f ( x ) [7].
Revised: A f ( x ) [7].

Original: Second, for a tuple ( C 1 , . . .
Revised: Second, for a tuple ( C 1 , . . .

Original: C N ) of closed convex sets C j ⊂ R d with orthogonal projections P C j , alternating orthogonal projections, i.e., the sequence x i +1 =
Revised: C N ) of closed convex sets C j ⊂ R d with orthogonal projections P C j , alternating orthogonal projections, i.e., the sequence x i +1 =

Original: P C N ◦ · · · ◦ P C 1 ( x i ) converges [5] to a point in the intersection C 1 ∩ · · · ∩ C N (if it is non-empty).
Revised: P C N ◦ · · · ◦ P C 1 ( x i ) converges [5] to a point in the intersection C 1 ∩ · · · ∩ C N (if it is non-empty).

Original: Notably, there are variants of alternating orthogonal projections, e.g., Dykstra’s algorithm [12], which converge to the orthogonal projection P (cid:84)
Revised: Notably, there are variants of alternating orthogonal projections, e.g., Dykstra’s algorithm [12], which converge to the orthogonal projection P (cid:84)

Original: i C i onto the intersection (cid:84)
Revised: i C i onto the intersection (cid:84)

Original: i C i .
Revised: i C i .

Original: However, there is a key disadvantage of orthogonal projections.
Revised: However, there is a key disadvantage of orthogonal projections.

Original: Being defined by
Revised: Being defined by

Original: the optimization
Revised: the optimization

Original: problem x orth = arg
Revised: problem x orth = arg

Original: min x ∈ A ∥ x − x 0 ∥of Eq.
Revised: min x ∈ A ∥ x − x 0 ∥of Eq.

Original: (40), they often can only be computed numerically and might require a large compute budget.
Revised: (40), they often can only be computed numerically and might require a large compute budget.

Original: In Section 4, the convex set is
Revised: In Section 4, the convex set is

Original: [Equation]
Revised: [Equation]

Original: The orthogonal projection onto C is unknown, but the alternating orthogonal projection onto the sets
Revised: The orthogonal projection onto C is unknown, but the alternating orthogonal projection onto the sets

Original: [Equation]
Revised: [Equation]

Original: still defines a projection onto C ⊂ R c in × c out × k h × k w considered as subset of R c in × c out × h × w .
Revised: still defines a projection onto C ⊂ R c in × c out × k h × k w considered as subset of R c in × c out × h × w .

Original: Importantly, all three orthogonal projections are known.
Revised: Importantly, all three orthogonal projections are known.

Original: The projection onto C 1 is due to [30].
Revised: The projection onto C 1 is due to [30].

Original: The projection onto
Revised: The projection onto

Original: C 2 requires a singular value decomposition of M K , i.e., the hwc in × hwc out matrix corresponding to the linear map ϕ K .
Revised: C 2 requires a singular value decomposition of M K , i.e., the hwc in × hwc out matrix corresponding to the linear map ϕ K .

Original: As this matrix can be quite large, this is infeasible in practice.
Revised: As this matrix can be quite large, this is infeasible in practice.

Original: However, [40] show that for strides 1 4 , due to the particular structure of convolutions, it suffices to compute the singular value decomposition of hw matrices of size c in × c out .
Revised: However, [40] show that for strides 1 4 , due to the particular structure of convolutions, it suffices to compute the singular value decomposition of hw matrices of size c in × c out .

Original: Still, the computation of the projection onto C 2 is the bottleneck of the training procedure in Section 4.
Revised: Still, the computation of the projection onto C 2 is the bottleneck of the training procedure in Section 4.

Original: The orthogonal projection onto C 3 , which is a plane, is realized by setting the corresponding coordinates to zero.
Revised: The orthogonal projection onto C 3 , which is a plane, is realized by setting the corresponding coordinates to zero.

Original: Another approach is to use radial projections .
Revised: Another approach is to use radial projections .

Original: Given x 0 ∈ R d and a norm ∥·∥ , the radial projection of x 0 onto the ∥·∥ -ball B ( r, y, ∥·∥ ) of radius r centered at y , is defined as
Revised: Given x 0 ∈ R d and a norm ∥·∥ , the radial projection of x 0 onto the ∥·∥ -ball B ( r, y, ∥·∥ ) of radius r centered at y , is defined as

Original: [Equation]
Revised: [Equation]

Original: Such a projection is called radial, as it translates the point x 0 in radial direction w.r.t.
Revised: Such a projection is called radial, as it translates the point x 0 in radial direction w.r.t.

Original: the ball
Revised: the ball

Original: B ( r, y, ∥·∥ ) such that it lands on the boundary (if it is not already in B ( r, y, ∥·∥ ) ).
Revised: B ( r, y, ∥·∥ ) such that it lands on the boundary (if it is not already in B ( r, y, ∥·∥ ) ).

Original: Notably, C 1 and C 2 are both balls, one with respect to the (2,1) group norm, the other with respect to the spectral norm of the matrix M K associated to K .
Revised: Notably, C 1 and C 2 are both balls, one with respect to the (2,1) group norm, the other with respect to the spectral norm of the matrix M K associated to K .

Original: Importantly, the spectral norm can be easily estimated by the power method for convolutional layers [15, 27], so radial projections have far less computational overhead.
Revised: Importantly, the spectral norm can be easily estimated by the power method for convolutional layers [15, 27], so radial projections have far less computational overhead.

Original: However, alternating radial projections are not guaranteed 5 to converge to a point in the intersection C .
Revised: However, alternating radial projections are not guaranteed 5 to converge to a point in the intersection C .

Original: Furthermore, by definition, we have ∥ x − x rad ∥ 2 ≥ ∥ x − x orth ∥ 2 , so we expect radial projections to yield inferior results (w.r.t.
Revised: Furthermore, by definition, we have ∥ x − x rad ∥ 2 ≥ ∥ x − x orth ∥ 2 , so we expect radial projections to yield inferior results (w.r.t.

Original: the constraint strengths that can be enforced).
Revised: the constraint strengths that can be enforced).

Original: We evaluated three different approaches to obtain models with constrained capacity: (1) training with a variant of projected gradient descent, where we perform one alternating orthogonal projection step after every 15th SGD update; (2) performing one alternating radial projection step after every
Revised: We evaluated three different approaches to obtain models with constrained capacity: (1) training with a variant of projected gradient descent, where we perform one alternating orthogonal projection step after every 15th SGD update; (2) performing one alternating radial projection step after every

Original: SGD update 6 ; (3) orthogonal projection onto C of an already trained unconstrained model, usingiterations of Dykstra’s algorithm.
Revised: SGD update 6 ; (3) orthogonal projection onto C of an already trained unconstrained model, usingiterations of Dykstra’s algorithm.

Original: Our findings are summarized in Fig.
Revised: Our findings are summarized in Fig.

Original: We see that, alternating orthogonal projections during training allow for the strongest constraints, without a drop in the testing accuracy.
Revised: We see that, alternating orthogonal projections during training allow for the strongest constraints, without a drop in the testing accuracy.

Original: This is expected, because they divert the weights less from the training trajectory than radial projections.
Revised: This is expected, because they divert the weights less from the training trajectory than radial projections.

Original: By the same logic it is obvious that projecting only at the end of training is not feasible, as the weights of the trained network are already too far away from the constraint set.
Revised: By the same logic it is obvious that projecting only at the end of training is not feasible, as the weights of the trained network are already too far away from the constraint set.

Original: We conclude that alternating orthogonal projections allow for the best estimate of excess capacity.
Revised: We conclude that alternating orthogonal projections allow for the best estimate of excess capacity.

Original: B.3 Hardware resources
Revised: B.3 Hardware resources

Original: All experiments were run on an Ubuntu Linux 20.04.4 LTS system with 128 GB of main memory, an
Revised: All experiments were run on an Ubuntu Linux 20.04.4 LTS system with 128 GB of main memory, an

Original: Intel®Core™ i9-10980XE processor and two NVIDIA GeForce RTX 3090 graphics cards (24 GB memory, CUDA 11.4, driver version 470.129.06).
Revised: Intel®Core™ i9-10980XE processor and two NVIDIA GeForce RTX 3090 graphics cards (24 GB memory, CUDA 11.4, driver version 470.129.06).

Original: All models are implemented in Pytorch (v1.10).
Revised: All models are implemented in Pytorch (v1.10).

Original: B.4 Comparison between constrained and unconstrained models beyond testing error
Revised: B.4 Comparison between constrained and unconstrained models beyond testing error

Original: So far, we have analyzed to which extent the weights of neural networks can be constrained without a loss of testing accuracy.
Revised: So far, we have analyzed to which extent the weights of neural networks can be constrained without a loss of testing accuracy.

Original: In particular, we have identified the maximal constraint strength (i.e., the operating point) such that the testing error of the constrained models is on a par with the one of unconstrained models.
Revised: In particular, we have identified the maximal constraint strength (i.e., the operating point) such that the testing error of the constrained models is on a par with the one of unconstrained models.

Original: However, this does not imply that constrained models and unconstrained ones can be used interchangeably, as they might differ in other aspects.
Revised: However, this does not imply that constrained models and unconstrained ones can be used interchangeably, as they might differ in other aspects.

Original: In this section, we will study how
Revised: In this section, we will study how

Original: [Figure]
Revised: [Figure]

Revised: proj.

Revised: at

Original: [Figure]
Revised: [Figure]

Original: T r a i n i n g acc u r ac
Revised: T r a i n i n g acc u r ac

Original: y
Revised: y

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original:  r o upn o r m d
Revised: ) g r o upn o r m d

Original: i s t a n ce c
Revised: i s t a n ce c

Original: [Figure]
Revised: [Figure]

Original: s t
Revised: s t

Original: r a i n t
Revised: r a i n t

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: [Figure]

Revised: [Figure]

Original: Acc.(Tst)
Revised: [Figure] [Figure] [Figure] ti

Original: [Figure]
Revised: [Figure]

Original:  pronounced such differences are with respect to (i) biases to particular classes, (ii) susceptibility to adversarial attacks, and (iii) compressibility in terms of the number of weights (via weight pruning).
Revised: n c y pronounced such differences are with respect to (i) biases to particular classes, (ii) susceptibility to adversarial attacks, and (iii) compressibility in terms of the number of weights (via weight pruning).

Original: For the evaluation, we use 25 unconstrained and 25 constrained models trained on CIFAR100, with the same architecture and optimization hyperparameters as listed in the main text, i.e., Section 4.
Revised: For the evaluation, we use 25 unconstrained and 25 constrained models trained on CIFAR100, with the same architecture and optimization hyperparameters as listed in the main text, i.e., Section 4.

Original: As constraint strength, we choose a layer-wise Lipschitz constant of 0.8 and a distance constraint of 70.
Revised: As constraint strength, we choose a layer-wise Lipschitz constant of 0.8 and a distance constraint of 70.

Original: B.4.1 Biases to particular classes
Revised: B.4.1 Biases to particular classes

Original: For each of the 50 models, we counted how often each class is predicted on the testing data, which consists of 100 images per class.
Revised: For each of the 50 models, we counted how often each class is predicted on the testing data, which consists of 100 images per class.

Original: If there are no biases to particular classes, the counts should be distributed around this value with preferably small spread.
Revised: If there are no biases to particular classes, the counts should be distributed around this value with preferably small spread.

Original: Fig.
Revised: Fig.

Original: 8 visualizes the results.
Revised: 8 visualizes the results.

Original: We immediately see, that for the unconstrained models, the predictions per class are more uniformly distributed with the average class close to 100 and small standard deviations.
Revised: We immediately see, that for the unconstrained models, the predictions per class are more uniformly distributed with the average class close to 100 and small standard deviations.

Original: In contrast, for the constrained models, the standard deviations are much larger.
Revised: In contrast, for the constrained models, the standard deviations are much larger.

Original: Most striking is the peak at class index 21, indicating that the constrained models are indeed biased to this particular class (chimpanzee).
Revised: Most striking is the peak at class index 21, indicating that the constrained models are indeed biased to this particular class (chimpanzee).

Original: In fact, only one of the 25 models predicted this class less than 100 times.
Revised: In fact, only one of the 25 models predicted this class less than 100 times.

Original: Notably, the unconstrained models are also biased towards this class, as the error region of (mean ± standard deviation) does not contain 100.
Revised: Notably, the unconstrained models are also biased towards this class, as the error region of (mean ± standard deviation) does not contain 100.

Original: Overall, there are more favored/disfavored classes for the unconstrained models (32 vs. only 5 for the constrained models), but for the constrained models, the biases are more pronounced.
Revised: Overall, there are more favored/disfavored classes for the unconstrained models (32 vs. only 5 for the constrained models), but for the constrained models, the biases are more pronounced.

Original: B.4.2 Susceptibility to adversarial attacks
Revised: B.4.2 Susceptibility to adversarial attacks

Original: We tested several adversarial attacks (FGSM, FGM, L2PGD, LinfPGD, L2DeepFool,
Revised: We tested several adversarial attacks (FGSM, FGM, L2PGD, LinfPGD, L2DeepFool,

Original: L2AdditiveGaussianNoise, L2AdditiveUniformNoise, L2ContrastReduction, GaussianBlur) using the foolbox [39] Python package.
Revised: L2AdditiveGaussianNoise, L2AdditiveUniformNoise, L2ContrastReduction, GaussianBlur) using the foolbox [39] Python package.

Original: To compare constrained vs. unconstrained models, we extract 1024 images from the testing data, which are correctly classified by all 50 models (25 constrained models, 25 unconstrained models) on which the attacks are evaluated.
Revised: To compare constrained vs. unconstrained models, we extract 1024 images from the testing data, which are correctly classified by all 50 models (25 constrained models, 25 unconstrained models) on which the attacks are evaluated.

Original: The fraction of correctly classified images for increasing attack strengths is visualized in Fig.
Revised: The fraction of correctly classified images for increasing attack strengths is visualized in Fig.

Original: As can be seen from the figure, constrained models are less susceptible to the gradient-based attacks FGSM, FGM, L2PGD, and LinfPGD.
Revised: As can be seen from the figure, constrained models are less susceptible to the gradient-based attacks FGSM, FGM, L2PGD, and LinfPGD.

Original: For L2DeepFool, contrast reduction (L2ContrastReductionAttack) and Gaussian blur
Revised: For L2DeepFool, contrast reduction (L2ContrastReductionAttack) and Gaussian blur

Original: [Figure]
Revised: [Figure]

Original: F [Figure] ac ti ono f c o rrec t l y c l a ss i fi e d
Revised: F r ac ti ono f c o rrec t l y c l a ss i fi e d

Original: i m a g e s
Revised: i m a g e s

Original: L2PGD
Revised: [Figure] GaussianBlurAttack

Original: [Figure]
Revised: [Figure]

Original: (GaussianBlurAttack), constrained and unconstrained models are equally affected.
Revised: (GaussianBlurAttack), constrained and unconstrained models are equally affected.

Original: To our surprise, the unconstrained models are less susceptible to additive Gaussian and uniform noise.
Revised: To our surprise, the unconstrained models are less susceptible to additive Gaussian and uniform noise.

Original: B.4.3 Compressibility via weight pruning
Revised: B.4.3 Compressibility via weight pruning

Original: We measure compressibility for global (unstructured) ℓ 1 weight pruning.
Revised: We measure compressibility for global (unstructured) l 1 weight pruning.

Original: This simple pruning technique identifies a predefined fraction of a model’s parameters (i
Revised: This simple pruning technique identifies a predefined fraction of a model’s parameters (i

Original: [Figure]
Revised: [Figure]

Original: e., elements of the weight tensors/matrices) and sets them to zero.
Revised: e., elements of the weight tensors/matrices) and sets them to zero.

Original: No subsequent fine-tuning steps were performed.
Revised: No subsequent fine-tuning steps were performed.

Original: [Figure]
Revised: [Figure]

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: [Figure]
Revised: [Figure]

Original: unconstrained models are less affected by this pruning technique than constrained models.
Revised: unconstrained models are less affected by this pruning technique than constrained models.

Original: Yet, constrained models can still be pruned [Figure] and, in the range of pruning strengths 0 to 0.15, the median testing accuracy drops only marginally for both model types.
Revised: Yet, constrained models can still be pruned, and, in the range of pruning strengths 0 to 0.15, the median testing accuracy drops only marginally for both model types.

Original: Only at larger pruning strengths do differences between the model types become visible.
Revised: Only at larger pruning strengths do differences between the model types become visible.

Original: If we consider a median testing accuracy of 46%, resp.
Revised: If we consider a median testing accuracy of 46%, resp.

Original: 45%, to be acceptable, then this threshold allows for pruning 20%, resp.
Revised: 45%, to be acceptable, then this threshold allows for pruning 20%, resp.

Original: 30%, of the weights of constrained models and 30%, resp 40%, of the unconstrained models.
Revised: 30%, of the weights of constrained models and 30%, resp 40%, of the unconstrained models.

Original: Of course, the pruned models might not satisfy the constraints anymore.
Revised: Of course, the pruned models might not satisfy the constraints anymore.

Original: In particular, the distance constraint gets violated, as with increasing constraint strength, the distance to initialization converges to the norm of the initialization.
Revised: In particular, the distance constraint gets violated, as with increasing constraint strength, the distance to initialization converges to the norm of the initialization.

Original: Typically, the latter is already larger than the distance constraint (of 70).
Revised: Typically, the latter is already larger than the distance constraint (of 70).

Original: Thus, when combining norm constraints with weight pruning, constraining the distance with respect to the zero weight appears to be more sensible.
Revised: Thus, when combining norm constraints with weight pruning, constraining the distance with respect to the zero weight appears to be more sensible.

Original: This can be done, as the distance constraint in Theorem 3.5 is not required to be with respect to the initialization; it can be chosen relative to any reference weight as long as the reference weight does not depend on the training data.
Revised: This can be done, as the distance constraint in Theorem 3.5 is not required to be with respect to the initialization; it can be chosen relative to any reference weight as long as the reference weight does not depend on the training data.

Original: C Proofs
Revised: C Proofs

Original: In the following sections, we present proofs for the theoretical results listed in the manuscript as well as additional supplementary results.
Revised: In the following sections, we present proofs for the theoretical results listed in the manuscript as well as additional supplementary results.

Original: C.1 Preliminaries
Revised: C.1 Preliminaries

Original: In terms of notation, we consider spaces F of functions f : ( X , ∥·∥ X ) →
Revised: In terms of notation, we consider spaces F of functions f : ( X , ∥·∥ X ) →

Original: ( Y , ∥·∥ Y ) between normed spaces.
Revised: ( Y , ∥·∥ Y ) between normed spaces.

Original: We write
Revised: We write

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: for the Lipschitz constant of f and the supremal Lipschitz constant of F , respectively.
Revised: for the Lipschitz constant of f and the supremal Lipschitz constant of F , respectively.

Original: For the remainder of the section, all function spaces F will have bounded Lipschitz constants Lip( F ) < ∞ .
Revised: For the remainder of the section, all function spaces F will have bounded Lipschitz constants Lip( F ) < ∞ .

Original: Such function spaces are vector spaces, where addition and scalar multiplication are defined pointwise via the vector space structure on Y , i.e., ( f + g ): x (cid:55)→ f ( x ) + g ( x ) and ( αf ): x (cid:55)→ αf ( x ) .
Revised: Such function spaces are vector spaces, where addition and scalar multiplication are defined pointwise via the vector space structure on Y , i.e., ( f + g ): x (cid:55)→ f ( x ) + g ( x ) and ( αf ): x (cid:55)→ αf ( x ) .

Original: We equip F with a data-dependent norm , defined below.
Revised: We equip F with a data-dependent norm , defined below.

Original: Definition C.1.
Revised: Definition C.1.

Original: Let F be a space of functions f : ( X , ∥·∥ X ) →
Revised: Let F be a space of functions f : ( X , ∥·∥ X ) →

Original: ( Y , ∥·∥ Y ) between normed spaces.
Revised: ( Y , ∥·∥ Y ) between normed spaces.

Original: The data-dependent norm , denoted as ∥·∥ X , on F restricted to X = ( x 1 . . .
Revised: The data-dependent norm , denoted as ∥·∥ X , on F restricted to X = ( x 1 . . .

Original: , x n ) ∈ X n , i.e., F| X , is defined as
Revised: , x n ) ∈ X n , i.e., F| X , is defined as

Original: [Equation]
Revised: [Equation]

Original: Remark C.2.
Revised: Remark C.2.

Original: This norm is a seminorm on F and a norm on F| X
Revised: This norm is a seminorm on F and a norm on F| X

Original: =
Revised: =

Original: { f | { x 1 ,...,x n } : f ∈ F} .
Revised: { f | { x 1 ,...,x n } : f ∈ F} .

Original: If f ∈ F has norm ∥ f ∥ X = 0 , then it holds that ∀ i : f ( x i )
Revised: If f ∈ F has norm ∥ f ∥ X = 0 , then it holds that ∀ i : f ( x i )

Original: = 0 .
Revised: = 0 .

Original: Thus, f is the zero element in F| X , but not necessarily the zero element in F , as there might exist v ∈ X \ { x 1 , . . .
Revised: Thus, f is the zero element in F| X , but not necessarily the zero element in F , as there might exist v ∈ X \ { x 1 , . . .

Original: , x n } with f ( v ) ̸ = 0 .
Revised: , x n } with f ( v ) ̸ = 0 .

Original: Two fundamental properties concerning compositions of functions are worth pointing out:
Revised: Two fundamental properties concerning compositions of functions are worth pointing out:

Original: [Equation]
Revised: [Equation]

Original: Further, we recall the definition of covering numbers .
Revised: Further, we recall the definition of covering numbers .

Original: Definition C.3.
Revised: Definition C.3.

Original: Let ( H , ∥·∥ ) be a normed space, S ⊂ H and ϵ > 0 .
Revised: Let ( H , ∥·∥ ) be a normed space, S ⊂ H and ϵ > 0 .

Original: We call any subset U ⊂ S an internal ϵ -cover of S if for every s ∈ S there exists u ∈ U such that ∥ s − u ∥ ≤ ϵ .
Revised: We call any subset U ⊂ S an internal ϵ -cover of S if for every s ∈ S there exists u ∈ U such that ∥ s − u ∥ ≤ ϵ .

Original: The internal covering number N int ( H , ϵ, ∥·∥ ) is the cardinality of the smallest internal ϵ -cover of S , i.e.,
Revised: The internal covering number N int ( H , ϵ, ∥·∥ ) is the cardinality of the smallest internal ϵ -cover of S , i.e.,

Original: [Equation]
Revised: [Equation]

Original: Dropping the requirement U ⊂ S , we analogously define external ϵ -covers U ⊂ H and external covering numbers N ext ( S, ϵ, ∥·∥ ) .
Revised: Dropping the requirement U ⊂ S , we analogously define external ϵ -covers U ⊂ H and external covering numbers N ext ( S, ϵ, ∥·∥ ) .

Original: In the manuscript, if not stated otherwise, covers will always be internal and covering numbers will be denoted as N = N int .
Revised: In the manuscript, if not stated otherwise, covers will always be internal and covering numbers will be denoted as N = N int .

Original: Internal and external covering numbers are related via the following chain of inequalities:
Revised: Internal and external covering numbers are related via the following chain of inequalities:

Original: [Equation]
Revised: [Equation]

Original: The first inequality follows directly from the definition of covering numbers; the second one follows from the triangle inequality.
Revised: The first inequality follows directly from the definition of covering numbers; the second one follows from the triangle inequality.

Original: Furthermore, for any subset T ⊂ S , it holds that
Revised: Furthermore, for any subset T ⊂ S , it holds that

Original: [Equation]
Revised: [Equation]

Original: Notably, this is not true for internal coverings.
Revised: Notably, this is not true for internal coverings.

Original: For example, the unit ball in R d defines an internal cover of itself, whereas an annulus cannot be covered internally with only one ball of radius 1 .
Revised: For example, the unit ball in R d defines an internal cover of itself, whereas an annulus cannot be covered internally with only one ball of radius 1 .

Original: C.2 Single-layer covering number bounds
Revised: C.2 Single-layer covering number bounds

Original: This section contains the covering number bounds for single convolutional layers.
Revised: This section contains the covering number bounds for single convolutional layers.

Original: For simplicity, we will first present the special case of the single-layer covering number bound for 1D convolutions with one channel, stride 1, odd kernel size and input size preserving (zero) padding.
Revised: For simplicity, we will first present the special case of the single-layer covering number bound for 1D convolutions with one channel, stride 1, odd kernel size and input size preserving (zero) padding.

Original: The proof of the general case then follows along the same line of arguments, but is more tedious, due to the additional notation and subindices.
Revised: The proof of the general case then follows along the same line of arguments, but is more tedious, due to the additional notation and subindices.

Original: Let X =
Revised: Let X =

Original: ( x 1 , . .
Revised: ( x 1 , . .

Original: , x n ) ∈ X n with x i ∈
Revised: , x n ) ∈ X n with x i ∈

Original: X
Revised: X

Original: = R h .
Revised: = R h .

Original: Further, let K ∈ R k , with k odd, be a convolutional kernel and ϕ K : R h
Revised: Further, let K ∈ R k , with k odd, be a convolutional kernel and ϕ K : R h

Original: → R h the corresponding convolutional map, which is defined coordinate-wise as
Revised: → R h the corresponding convolutional map, which is defined coordinate-wise as

Original: [Equation]
Revised: [Equation]

Original: For the norm of the data X , we write
Revised: For the norm of the data X , we write

Original: [Equation]
Revised: [Equation]

Original: Our covering number bounds hinge on the seminal Maurey sparsification lemma.
Revised: Our covering number bounds hinge on the seminal Maurey sparsification lemma.

Original: We state one variant, see [3, Lemma A.6].
Revised: We state one variant, see [3, Lemma A.6].

Original: Lemma C.4 (Maurey sparsification lemma) .
Revised: Lemma C.4 (Maurey sparsification lemma) .

Original: Fix a Hilbert space H with norm ∥·∥ .
Revised: Fix a Hilbert space H with norm ∥·∥ .

Original: Let U ∈ H be given with representation U
Revised: Let U ∈ H be given with representation U

Original: = (cid:80) di
Revised: = (cid:80) di

Original: =

Revised: =

Original: α i V i
Revised: α i V i

Original: where V i ∈ H , α ∈ R d ≥ 0 \ { 0 } and (cid:80) i | α
Revised: where V i ∈ H , α ∈ R d ≥ 0 \ { 0 } and (cid:80) i | α

Original: i | ≤ 1 .
Revised: i | ≤ 1 .

Original: Then, for any positive integer m , there exists a choice of non-negative integers ( m 1 , . . .
Revised: Then, for any positive integer m , there exists a choice of non-negative integers ( m 1 , . . .

Original: , m d ) with (cid:80) i m i
Revised: , m d ) with (cid:80) i m i

Original: = m , such that
Revised: = m , such that

Original: [Equation]
Revised: [Equation]

Original: Theorem C.5 (Single-layer covering number bound – Simple 1D variant) .
Revised: Theorem C.5 (Single-layer covering number bound – Simple 1D variant) .

Original: Let b > 0 , k odd and let F = { ϕ K | K ∈ R k , ∥ K ∥ 1 ≤ b } be the set of 1D convolutions determined by kernels K ∈ R k with ∥ K ∥ 1 ≤ b .
Revised: Let b > 0 , k odd and let F = { ϕ K | K ∈ R k , ∥ K ∥ 1 ≤ b } be the set of 1D convolutions determined by kernels K ∈ R k with ∥ K ∥ 1 ≤ b .