Original: E STIMATING L IPSCHITZ CONSTANTS OF MONOTONE DEEP EQUILIBRIUM MODELS
Revised: E STIMATING L IPSCHITZ CONSTANTS OF MONOTONE DEEP EQUILIBRIUM MODELS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Several methods have been proposed in recent years to provide bounds on the
Revised: Several methods have been proposed in recent years to provide bounds on the

Original: Lipschitz constants of deep networks, which can be used to provide robustness guarantees, generalization bounds, and characterize the smoothness of decision boundaries.
Revised: Lipschitz constants of deep networks, which can be used to provide robustness guarantees, generalization bounds, and characterize the smoothness of decision boundaries.

Original: However, existing bounds get substantially weaker with increasing depth of the network, which makes it unclear how to apply such bounds to recently proposed models such as the deep equilibrium (DEQ) model, which can be viewed as representing an inﬁnitely-deep network.
Revised: However, existing bounds get substantially weaker with increasing depth of the network, which makes it unclear how to apply such bounds to recently proposed models such as the deep equilibrium (DEQ) model, which can be viewed as representing an inﬁnitely-deep network.

Original: In this paper, we show that monotone DEQs, a recently-proposed subclass of DEQs, have Lipschitz constants that can be bounded as a simple function of the strong monotonicity parameter of the network.
Revised: In this paper, we show that monotone DEQs, a recently-proposed subclass of DEQs, have Lipschitz constants that can be bounded as a simple function of the strong monotonicity parameter of the network.

Original: We derive simple-yet-tight bounds on both the input-output mapping and the weight-output mapping deﬁned by these networks, and demonstrate that they are small relative to those for comparable standard DNNs.
Revised: We derive simple-yet-tight bounds on both the input-output mapping and the weight-output mapping deﬁned by these networks, and demonstrate that they are small relative to those for comparable standard DNNs.

Original: We show that one can use these bounds to design monotone DEQ models, even with e.g. multiscale convolutional structure, that still have constraints on the Lipschitz constant.
Revised: We show that one can use these bounds to design monotone DEQ models, even with e g. multiscale convolutional structure, that still have constraints on the Lipschitz constant.

Original: We also highlight how to use these bounds to develop PAC-Bayes generalization bounds that do not depend on any depth of the network, and which avoid the exponential depth-dependence of comparable DNN bounds.
Revised: We also highlight how to use these bounds to develop PAC-Bayes generalization bounds that do not depend on any depth of the network, and which avoid the exponential depth-dependence of comparable DNN bounds.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Measuring the sensitivity of deep neural networks (DNNs) to changes in their inputs or weights is important in a wide range of applications.
Revised: Measuring the sensitivity of deep neural networks (DNNs) to changes in their inputs or weights is important in a wide range of applications.

Original: A standard way of measuring the sensitivity of a function f is the Lipschitz constant of f , the smallest constant L ∈ R + such that (cid:107) f
Revised: A standard way of measuring the sensitivity of a function f is the Lipschitz constant of f , the smallest constant L ∈ R + such that (cid:107) f

Original: ( x ) − f ( y ) (cid:107) 2 ≤ L (cid:107) x − y (cid:107)for all inputs x and y .
Revised: ( x ) − f ( y ) (cid:107) 2 ≤ L (cid:107) x − y (cid:107)for all inputs x and y .

Original: While exact computation of the Lipschitz constant of DNNs is NP-hard (Virmaux & Scaman, 2018), bounds or estimates can be used to certify a network’s robustness to adversarial input perturbations (Weng et al., 2018), encourage robustness during training (Tsuzuku et al., 2018), or as a complexity measure of the DNN (Bartlett et al., 2017), among other applications.
Revised: While exact computation of the Lipschitz constant of DNNs is NP-hard (Virmaux & Scaman, 2018), bounds or estimates can be used to certify a network’s robustness to adversarial input perturbations (Weng et al., 2018), encourage robustness during training (Tsuzuku et al., 2018), or as a complexity measure of the DNN (Bartlett et al., 2017), among other applications.

Original: An analogous Lipschitz constant that bounds the sensitivity of f to changes in its weights can be used to derive generalization bounds for DNNs (Neyshabur et al., 2018).
Revised: An analogous Lipschitz constant that bounds the sensitivity of f to changes in its weights can be used to derive generalization bounds for DNNs (Neyshabur et al., 2018).

Original: A growing number of methods for computing bounds on the Lipschitz constant of DNNs have been proposed in recent works, primarily based on semideﬁnite programs (Fazlyab et al., 2019; Raghunathan et al., 2018) or polynomial programs (Latorre et al., 2019).
Revised: A growing number of methods for computing bounds on the Lipschitz constant of DNNs have been proposed in recent works, primarily based on semideﬁnite programs (Fazlyab et al., 2019; Raghunathan et al., 2018) or polynomial programs (Latorre et al., 2019).

Original: However, as the depth of the network increases, these bounds become either very loose or prohibitively expensive to compute.
Revised: However, as the depth of the network increases, these bounds become either very loose or prohibitively expensive to compute.

Original: Additionally, they are typically not applicable to structured DNNs such as convolutional networks which are common in everyday use.
Revised: Additionally, they are typically not applicable to structured DNNs such as convolutional networks which are common in everyday use.

Original: The deep equilibrium model (DEQ) (Bai et al., 2019) is an implicit-depth model which directly solves for the ﬁxed point of an “inﬁnitely-deep”, weight-tied network.
Revised: The deep equilibrium model (DEQ) (Bai et al., 2019) is an implicit-depth model which directly solves for the ﬁxed point of an “inﬁnitely-deep”, weight-tied network.

Original: DEQs have been shown to perform as well as DNNs in domains such as computer vision (Bai et al., 2020) and sequence modelling (Bai et al., 2019), while avoiding the large memory footprint required by DNN training in order to backpropagate through a long computation chain.
Revised: DEQs have been shown to perform as well as DNNs in domains such as computer vision (Bai et al., 2020) and sequence modelling (Bai et al., 2019), while avoiding the large memory footprint required by DNN training in order to backpropagate through a long computation chain.

Original: Given that DEQs represent inﬁnite-depth networks, however, their Lipschitz constants clearly cannot be bounded by existing methods, which are very loose even on networks of depth 10 or less.
Revised: Given that DEQs represent inﬁnite-depth networks, however, their Lipschitz constants clearly cannot be bounded by existing methods, which are very loose even on networks of depth 10 or less.

Original: In this paper we take up the question of how to bound the Lipschitz constant of DEQs.
Revised: In this paper we take up the question of how to bound the Lipschitz constant of DEQs.

Original: In particular, we focus on monotone DEQs (monDEQ) (Winston & Kolter, 2020), a recently-proposed class of
Revised: In particular, we focus on monotone DEQs (monDEQ) (Winston & Kolter, 2020), a recently-proposed class of

Original: DEQs which parameterizes the DEQ model in a way that guarantees existence of a unique ﬁxedpoint, which can be computed efﬁciently as the solution to a monotone operator splitting problem.
Revised: DEQs which parameterizes the DEQ model in a way that guarantees existence of a unique ﬁxedpoint, which can be computed efﬁciently as the solution to a monotone operator splitting problem.

Original: We show that monDEQs, despite representing inﬁnite-depth networks, have Lipschitz constants which can be bounded by a simple function of the strong-monotonicity parameter, the choice of which therefore directly inﬂuences the bound.
Revised: We show that monDEQs, despite representing inﬁnite-depth networks, have Lipschitz constants which can be bounded by a simple function of the strong-monotonicity parameter, the choice of which therefore directly inﬂuences the bound.

Original: We also derive a bound on the Lipschtiz constant w.r.t.
Revised: We also derive a bound on the Lipschtiz constant w.r.t.

Original: the weights of the monDEQ, with which we derive a deterministic PAC-Bayes generalization bound for the monDEQ by adapting the technique of (Neyshabur et al., 2018).
Revised: the weights of the monDEQ, with which we derive a deterministic PAC-Bayes generalization bound for the monDEQ by adapting the technique of (Neyshabur et al., 2018).

Original: While such generalization bounds for DNNs are plagued by exponential dependence on network depth, the corresponding monDEQ bound does not involve any depth-like term.
Revised: While such generalization bounds for DNNs are plagued by exponential dependence on network depth, the corresponding monDEQ bound does not involve any depth-like term.

Original: Empirically, we demonstrate that our  bounds on fully-connected monDEQs trained on MNIST are small relative to comparable DNNs, even for DNNs of depth only 4.
Revised: Empirically, we demonstrate that our Lipschitz bounds on fully-connected monDEQs trained on MNIST are small relative to comparable DNNs, even for DNNs of depth only 4.

Original: We show a similar trend on single- and multi-convolutional monDEQs as compared to the bounds on traditional CNNs computed by AutoLip and SeqLip (Virmaux & Scaman, 2018), the only existing methods for (even approximately) bounding CNN Lipshitz constants.
Revised: We show a similar trend on single- and multi-convolutional monDEQs as compared to the bounds on traditional CNNs computed by AutoLip and SeqLip (Virmaux & Scaman, 2018), the only existing methods for (even approximately) bounding CNN Lipshitz constants.

Original: Further, our monDEQ generalization bounds are comparable with bounds on DNNs of around depth 5, and avoid the exponential dependence on depth of those bounds.
Revised: Further, our monDEQ generalization bounds are comparable with bounds on DNNs of around depth 5, and avoid the exponential dependence on depth of those bounds.

Revised: Finally, we also validate the signiﬁcance of the small Lipschitz bounds for monDEQs by empirically demonstrating strong adversarial robustness on MNIST and CIFAR-10.

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  B ACKGROUND AND R ELATED W ORK

Original: Lipschitz constants of DNNs Lipschitz constants of DNNs were proposed as early as Szegedy et al.
Revised: Lipschitz constants of DNNs Lipschitz constants of DNNs were proposed as early as Szegedy et al.

Original: (2014) as a potential means of controlling adversarial robustness.
Revised: (2014) as a potential means of controlling adversarial robustness.

Original: The bound proposed in that work was the product of the spectral norms of the layers, which in practice is extremely loose.
Revised: The bound proposed in that work was the product of the spectral norms of the layers, which in practice is extremely loose.

Original: Virmaux & Scaman (2018) derive a tighter bound via a convex maximization problem; however the bound is typically intractable and can only be approximated.
Revised: Virmaux & Scaman (2018) derive a tighter bound via a convex maximization problem; however the bound is typically intractable and can only be approximated.

Original: Combettes & Pesquet (2019) bound the
Revised: Combettes & Pesquet (2019) bound the

Original: Lipschitz constant of DNNs by noting that the common nonlinearities employed as activation functions are averaged, nonexapansive operators; however, their method scales exponentially with depth of the network.
Revised: Lipschitz constant of DNNs by noting that the common nonlinearities employed as activation functions are averaged, nonexpansive operators; however, their method scales exponentially with depth of the network.

Original: (Zou et al., 2019) propose linear-program-based bounds speciﬁc to convolutional networks, which in practice are several orders of magnitude larger than empirical lower bounds.
Revised: (Zou et al., 2019) propose linear-program-based bounds speciﬁc to convolutional networks, which in practice are several orders of magnitude larger than empirical lower bounds.

Original: Upper bounds based on semideﬁnite programs which relax the quadratic constraints imposed by the nonlinearities are studied by Fazlyab et al.
Revised: Upper bounds based on semideﬁnite programs which relax the quadratic constraints imposed by the nonlinearities are studied by Fazlyab et al.

Original: ; Raghunathan et al.
Revised: ; Raghunathan et al.

Original: ; Jin & Lavaei (2018).
Revised: ; Jin & Lavaei (2018).

Original: The bounds can be tight in practice but expensive to compute for deep networks; as such, Fazlyab et al.
Revised: The bounds can be tight in practice but expensive to compute for deep networks; as such, Fazlyab et al.

Original: (2019) propose a sequence of SDPs which trade off computational complexity and accuracy.
Revised: (2019) propose a sequence of SDPs which trade off computational complexity and accuracy.

Original: This allows us to compare our monDEQ bounds to their SDP bounds for networks of increasing depth (see Section 5).
Revised: This allows us to compare our monDEQ bounds to their SDP bounds for networks of increasing depth (see Section 5).

Original: Latorre et al.
Revised: Latorre et al.

Original: (2019) show that the complexity of the optimization problems can be reduced by taking advantage of the typical sparsity of connections common to DNNs, but the resulting methods are still prohibitively expensive for deep networks.
Revised: (2019) show that the complexity of the optimization problems can be reduced by taking advantage of the typical sparsity of connections common to DNNs, but the resulting methods are still prohibitively expensive for deep networks.

Original: DEQs and monotone DEQs An emerging focus of deep learning research is on implicit-depth models, typiﬁed by Neural ODEs (Chen et al., 2018) and deep equilibrium models (DEQs) (Bai et al., 2019; 2020).
Revised: monotone DEQs An emerging focus of deep learning research is on implicit-depth models, typiﬁed by Neural ODEs (Chen et al., 2018) and deep equilibrium models (DEQs) (Bai et al., 2019; 2020).

Original: Unlike traditional deep networks which compute their output by sequential, layer-wise computation, implicit-depth models simulate “inﬁnite-depth” networks by specifying, and directly solving for, some analytical conditions satisﬁed by their output.
Revised: Unlike traditional deep networks which compute their output by sequential, layer-wise computation, implicit-depth models simulate “inﬁnite-depth” networks by specifying, and directly solving for, some analytical conditions satisﬁed by their output.

Original: The DEQ model directly solves for the ﬁxed-point of an inﬁnitely-deep, weight-tied and input-injected network, which would consist of the iteration
Revised: The DEQ model directly solves for the ﬁxed-point of an inﬁnitely-deep, weight-tied and input-injected network, which would consist of the iteration

Original: z
Revised: z

Original: i +1 =
Revised: i +1 =

Original: g ( z i , x ) where, g represents a nonlinear layer computation which is applied repeatedly, z i is the activation at “layer” i , and x is the network input, which is injected at each layer.
Revised: g ( z i , x ) where, g represents a nonlinear layer computation which is applied repeatedly, z i is the activation at “layer” i , and x is the network input, which is injected at each layer.

Original: Instead of iteratively applying the function g (which indeed may not converge), the inﬁnite-depth ﬁxed-point z ∗ =
Revised: Instead of iteratively applying the function g (which indeed may not converge), the inﬁnite-depth ﬁxed-point z ∗ =

Original: g ( z ∗ , x ) can be solved using a root-ﬁnding method.
Revised: g ( z ∗ , x ) can be solved using a root-ﬁnding method.

Original: A key advantage of DEQs is that backpropagation through the ﬁxed-point can be performed analytically using the implicit function theorem, and DEQ training therefore requires much less memory than DNNs, which need to store the intermediate layer activations for backpropagation.
Revised: A key advantage of DEQs is that backpropagation through the ﬁxed-point can be performed analytically using the implicit function theorem, and DEQ training therefore requires much less memory than DNNs, which need to store the intermediate layer activations for backpropagation.

Original: In standard DEQs, existence of a unique ﬁxed point is not guaranteed, nor is stable convergence to a ﬁxed-point easy to obtain in practice.
Revised: In standard DEQs, existence of a unique ﬁxed point is not guaranteed, nor is stable convergence to a ﬁxed-point easy to obtain in practice.

Original: Monotone DEQs (monDEQs) (Winston & Kolter, 2020) improve upon this aspect by parameterizing the DEQ in a mannr that guarantees the existence of a stable ﬁxed point.
Revised: Monotone DEQs (monDEQs) (Winston & Kolter, 2020) improve upon this aspect by parameterizing the DEQ in a manner that guarantees the existence of a stable ﬁxed point.

Original: Monotone operator theory provides a class of operator splitting methods which are guaranteed to converge linearly to the ﬁxed point (see Ryu & Boyd (2016) for a primer).
Revised: Monotone operator theory provides a class of operator splitting methods which are guaranteed to converge linearly to the ﬁxed point (see Ryu & Boyd (2016) for a primer).

Original: The monDEQ considers a weight-tied, input-injected network with iterations of the form
Revised: The monDEQ considers a weight-tied, input-injected network with iterations of the form

Original: [Equation]
Revised: [Equation]

Original: where x ∈ R n is the input,
Revised: where x ∈ R n is the input,

Original: U ∈ R h × n the input-injection weights, z ( i ) ∈
Revised: U ∈ R h × n the input-injection weights, z ( i ) ∈

Original: R h the hidden unit activations at “layer” i , and W ∈ R
Revised: R h the hidden unit activations at “layer” i , and W ∈ R

Original: h × h the hidden-unit weights, and b ∈ R h a bias term, and σ : R h
Revised: h × h the hidden-unit weights, and b ∈ R h a bias term, and σ : R h

Original: → R h an elementwise nonlinearity.
Revised: → R h an elementwise nonlinearity.

Original: The output of the monDEQ is deﬁned as the ﬁxed point of the iteration, a z ∗ such that
Revised: The output of the monDEQ is deﬁned as the ﬁxed point of the iteration, a z ∗ such that

Original: [Equation]
Revised: [Equation]

Original: Just as for DEQs, forward iteration of this system need not converge to z ∗ ; instead, the ﬁxed point is found as the solution to a particular operator splitting problem.
Revised: Just as for DEQs, forward iteration of this system need not converge to z ∗ ; instead, the ﬁxed point is found as the solution to a particular operator splitting problem.

Original: Various operator splitting methods can be employed here, for example forward-backward iteration, which results in a damped version of the forward iteration
Revised: Various operator splitting methods can be employed here, for example forward-backward iteration, which results in a damped version of the forward iteration

Original: [Equation]
Revised: [Equation]

Original: The operator I − α
Revised: The operator I − α

Original: ( I − W ) appearing in this iteration is contractive for any 0 < α ≤ 2 m/L 2 , and this iteration is guaranteed to converge so long as the operator I − W is Lipschitz and strongly monotone with parameters L (which is in fact the spectral norm (cid:107) I − W (cid:107) 2 ) and m (Ryu & Boyd, 2016).
Revised: ( I − W ) appearing in this iteration is contractive for any 0 < α ≤ 2 m/L 2 , and this iteration is guaranteed to converge so long as the operator I − W is Lipschitz and strongly monotone with parameters L (which is in fact the spectral norm (cid:107) I − W (cid:107) 2 ) and m (Ryu & Boyd, 2016).

Original: In
Revised: In

Original: Section 3, we will see how unrolling this iteration leads directly to a bound on the Lipschitz constant of the monDEQ.
Revised: Section 3, we will see how unrolling this iteration leads directly to a bound on the Lipschitz constant of the monDEQ.

Original: To ensure the strong monotonicity condition, that I − W (cid:23) mI , the monDEQ parameterizes W as
Revised: To ensure the strong monotonicity condition, that I − W (cid:23) mI , the monDEQ parameterizes W as

Original: [Equation]
Revised: [Equation]

Original: The strong-monotonicity parameter m will in fact ﬁgure in directly to the Lipschitz constant of the monDEQ.
Revised: The strong-monotonicity parameter m will in fact ﬁgure in directly to the Lipschitz constant of the monDEQ.

Original: Lipschitz constants for implicit-depth models A few prior works have proposed methods for bounding the Lipschitz constants of other classes of implicit depth network.
Revised: Lipschitz constants for implicit-depth models A few prior works have proposed methods for bounding the Lipschitz constants of other classes of implicit depth network.

Original: Ghaoui et al.
Revised: Ghaoui et al.

Original: (2020) deﬁne restrictive conditions for well-posedness of an implicit network which are different from those of the monDEQ.
Revised: (2020) deﬁne restrictive conditions for well-posedness of an implicit network which are different from those of the monDEQ.

Original: In particular, they require the weight matrix W to be such that forward iteration is stable (as opposed to the stability of the operator splitting methods required by monDEQ).
Revised: In particular, they require the weight matrix W to be such that forward iteration is stable (as opposed to the stability of the operator splitting methods required by monDEQ).

Original: They derive Lipschitz constants and robustness guarantees under these conditions; for example when (cid:107) W (cid:107) ∞ < 1 , then a Lipschitz bound can be derived by simply manipulating the ﬁxed-point equation as they demonstrate in equation 4.3.
Revised: They derive Lipschitz constants and robustness guarantees under these conditions; for example when (cid:107) W (cid:107) ∞ < 1 , then a Lipschitz bound can be derived by simply manipulating the ﬁxed-point equation as they demonstrate in equation 4.3.

Original: Herrera et al.
Revised: Herrera et al.

Original: (2020) propose a framework for implicit depth models which incorporate the Neural ODE (Chen et al., 2018)
Revised: (2020) propose a framework for implicit depth models which incorporate the Neural ODE (Chen et al., 2018)

Original: (which is the solution of an ODE at a given time T) but not the monDEQ (which can be cast as ﬁnding the equilibrium point of an ODE).
Revised: (which is the solution of an ODE at a given time T) but not the monDEQ (which can be cast as ﬁnding the equilibrium point of an ODE).

Original: They derive bounds on the Lipschitz constant w.r.t.
Revised: They derive bounds on the Lipschitz constant w.r.t.

Original: network weights, but their framework cannot be applied to bound the Lipschitz constant of the monDEQ.
Revised: network weights, but their framework cannot be applied to bound the Lipschitz constant of the monDEQ.

Original:  L IPSCHITZ CONSTANTS FOR MONOTONE DEQ S
Revised:  L IPSCHITZ CONSTANTS FOR MONOTONE DEQ S

Original: We now present our central methodological contributions, easily-computable bounds on the Lipschitz constants of monotone DEQs.
Revised: We now present our main methodological contributions, easily-computable bounds on the Lipschitz constants of monDEQs.

Original: We ﬁrst derive the Lipschitz bound on the input-output mapping deﬁned by the monDEQ, followed by that for the weight-output mapping.
Revised: We ﬁrst derive the Lipschitz bound on the input-output mapping deﬁned by the monDEQ, followed by that for the weight-output mapping.

Original: As we describe below, both bounds turn out to depend inversely on the strong-monotonicity parameter m of the monDEQ.
Revised: As we describe below, both bounds turn out to depend inversely on the strong-monotonicity parameter m of the monDEQ.

Original: Since m is chosen for the monDEQ at design time, this implies an analytical handle on its Lipschitz constant.
Revised: Since m is chosen for the monDEQ at design time, this implies an analytical handle on its Lipschitz constant.

Original: 3. L IPSCHITZ CONSTANTS WITH RESPECT TO INPUT
Revised: 3. L IPSCHITZ CONSTANTS WITH RESPECT TO INPUT

Original: The naive way of computing L for feedforward deep networks is by multiplying the spectral norms of the weight matrices.
Revised: The naive way of computing L for feedforward deep networks is by multiplying the spectral norms of the weight matrices.

Original: As stated above, just employing forward iterations does not lead to convergence of the monDEQ.
Revised: As stated above, just employing forward iterations does not lead to convergence of the monDEQ.

Original: Analogously, if we were to adopt the naive method and simply unroll the forward iterations of the monDEQ as described in equation 1, we would end up with an inﬁnite product of spectral norms, which would not converge unless W itself is contractive.
Revised: Analogously, if we were to adopt the naive method and simply unroll the forward iterations of the monDEQ as described in equation 1, we would end up with an inﬁnite product of spectral norms, which would not converge unless W itself is contractive.

Original: Here again, we consider unrolling the averaged operator T :
Revised: Here again, we consider unrolling the averaged operator T :

Original: = I − α
Revised: = I − α

Original: ( I − W ) employed in the forward-backward iterations, which ensures that the monDEQ converges, and will also lead to a ﬁnite bound on the
Revised: ( I − W ) employed in the forward-backward iterations, which ensures that the monDEQ converges, and will also lead to a ﬁnite bound on the

Original: Lipschitz constant.
Revised: Lipschitz constant.

Original: Notice that T appears in the forward iterations in equation 3.
Revised: Notice that T appears in the forward iterations in equation 3.

Original: In the sequel, let
Revised: In the sequel, let

Original: L [ A ] denote the Lipschitz constant of a function or operator A .
Revised: L [ A ] denote the Lipschitz constant of a function or operator A .

Original: The following proposition, which we prove in Appendix A, bounds the Lipschitz constant L [ T ] .
Revised: The following proposition, which we prove in Appendix A, bounds the Lipschitz constant L [ T ] .

Original: Proposition 1.
Revised: Proposition 1.

Original: L [ T ] ≤ (cid:112) 1 − 2 αm + α 2 L
Revised: L [ T ] ≤ (cid:112) 1 − 2 αm + α 2 L

Original: [ I − W ]
Revised: [ I − W ]

Original: This implies that for α ∈ (cid:16) 0 , 2 m L [ I − W ] 2 (cid:17) , L [ T ] < 1 .
Revised: This implies that for α ∈ (cid:16) 0 , 2 m L [ I − W ] 2 (cid:17) , L [ T ] < 1 .

Original: In our subsequent analysis, we only consider values of α in this range.
Revised: In our subsequent analysis, we only consider values of α in this range.

Original: We are now ready to state our bound for the Lipschitz constant of the 
Revised: We are now ready to state our bound for the Lipschitz constant of the monDEQ:

Original: Theorem 1 (Lipschitz constant of monDEQ) .
Revised: Theorem 1 (Lipschitz constant of monDEQ) .

Original: Let f ( x ) =
Revised: Let f ( x ) =

Original: z ∗ denote the output of the monDEQ on input x , as in equation 2.
Revised: z ∗ denote the output of the monDEQ on input x , as in equation 2.

Original: Consider any x, y ∈ R n .
Revised: Consider any x, y ∈ R n .

Original: Then, we have that
Revised: Then, we have that

Original: [Equation]
Revised: [Equation]

Original: In other words [Equation] L [ f ] ≤ (cid:107) U (cid:107) 2 m .
Revised: In other words, L [ f ] ≤ (cid:107) U (cid:107) 2 m .

Original: Proof.
Revised: Proof.

Original: Let f k ( x )
Revised: Let f k ( x )

Original: =
Revised: =

Original: z ( k ) denote the k th iterate of the forward-backward iterations as described in equation 3 (we begin with f 0 ( x ) = 0 ).
Revised: z ( k ) denote the k th iterate of the forward-backward iterations as described in equation 3 (we begin with f 0 ( x ) = 0 ).

Original: We will try and unroll these iterations in the following:
Revised: We will try and unroll these iterations in the following:

Original: [Equation]
Revised: [Equation]

Revised: L [

Original: [Equation]
Revised: [Equation]

Original: ( unrolling k times )
Revised: ( unrolling k times )

Original: [Equation]
Revised: [Equation]

Revised: i =

Revised: ( L [ T ]

Revised: [Equation]

Original: ( since f 0 ( x ) = f 0 ( y ) = 0)
Revised: ( since f 0 ( x ) = f 0 ( y ) = 0)

Original: Since the above inequality holds for all k , we can take the limit on both sides as k → ∞ , keeping α ﬁxed.
Revised: Since the above inequality holds for all k , we can take the limit on both sides as k → ∞ , keeping α ﬁxed.

Original: But notice that since the forward-backward iterations converge to the true f (which does not depend on α ), we have that
Revised: But notice that since the forward-backward iterations converge to the true f (which does not depend on α ), we have that

Original: lim k →∞ f k = f .
Revised: lim k →∞ f k = f .

Original: That is, the dependence on α disappears on the LHS once we take the limit on k .
Revised: That is, the dependence on α disappears on the LHS once we take the limit on k .

Original: Thus, by using the continuity of the l 2 norm, we have
Revised: Thus, by using the continuity of the l 2 norm, we have

Original: [Equation]
Revised: [Equation]

Original: ( since L [ T ] < 1)
Revised: ( since L [ T ] < 1)

Original: [Equation]
Revised: [Equation]

Original: ( from Proposition 1 )
Revised: ( from Proposition 1 )

Original: Now, since the above result holds for any α in the range considered, taking α → 0 , we have that
Revised: Now, since the above result holds for any α in the range considered, taking α → 0 , we have that

Original: [Equation]
Revised: [Equation]

Original: L’Hopital’s rule
Revised: rule

Original: [Equation]
Revised: [Equation]

Original: We observe here that the Lipschitz constant of the monDEQ with respect to its inputs indeed depends on only two quantities, namely (cid:107) U (cid:107) 2 and m , and doesn’t depend at all on the weight matrix W .
Revised: We observe here that the Lipschitz constant of the monDEQ with respect to its inputs indeed depends on only two quantities, namely (cid:107) U (cid:107) 2 and m , and doesn’t depend at all on the weight matrix W .

Original: Furthermore, because m is a hyperparameter chosen by the user, this illustrates that monDEQs have the notable property that one can directly control the Lipschitz parameter of the network (at least as far as the inﬂuence of W is concerned), soley by  choosing m , instead of requiring any additionl structure or regularization on W .
Revised: Furthermore, because m is a hyperparameter chosen by the user, this illustrates that monDEQs have the notable property that one can essentially control the Lipschitz parameter of the network (insofar as the inﬂuence of W is concerned) by appropriately choosing m , and not require any additional structure or regularization on W .

Original: This is in stark contrast to most existing DNN architectures, where enforcing Lipschitz bounds requires substantial additional effort.
Revised: This is in stark contrast to most existing DNN architectures, where enforcing Lipschitz bounds requires substantial additional effort.

Original: 3. L IPSCHITZ CONSTANTS WITH RESPECT TO WEIGHTS
Revised: 3. L IPSCHITZ CONSTANTS WITH RESPECT TO WEIGHTS

Original: We now turn to the question of bounding the change in the output of the monDEQ when the weights are perturbed but the input remains ﬁxed.
Revised: We now turn to the question of bounding the change in the output of the monDEQ when the weights are perturbed but the input remains ﬁxed.

Original: This computation has several important use cases, one of which is in the derivation of generalization bounds for the monDEQ.
Revised: This calculation has several important use cases, one of which is in the derivation of generalization bounds for the monDEQ.

Original: Given a bound on the change in the output on perturbing the weights of the monDEQ, we can derive bounds on the generalization error in a straight-forward manner, as detailed in Section 4 below.
Revised: Given a bound on the change in the output on perturbing the weights of the monDEQ, we can derive bounds on the generalization error in a straight-forward manner, as detailed in Section 4 below.

Original: In order to derive the perturbation bound, we ﬁrst state the following proposition (proved in Appendix B), which bounds the norm of the output after k forward-backward iterations.

Original: Proposition 2.

Original: Let f k ( W, U, b ) denote the k th iterate of the forward-backward iterations of the monDEQ parameterized by W, U, b on a ﬁxed arbitrary input x .

Original: Further, let T ( W ) =

Original: (1 − α )

Original: I + αW . Then, we have that [Equation] We can now derive a perturbation bound for the monDEQ.
Revised: The following theorem establishes a perturbation bound for the monDEQ.

Original: Theorem 2 (Perturbation bound for monDEQ) .
Revised: Theorem 2 (Perturbation bound for monDEQ) .

Original: Let I − W (cid:23) mI
Revised: Let I − W (cid:23) mI

Original: and I − ¯ W (cid:23) ¯ mI .
Revised: and I − ¯ W (cid:23) ¯ mI .

Original: The change in the output of the monDEQ on perturbing the weights and biases from W, U, b to ¯ W, ¯ U, ¯ b is bounded as follows:
Revised: The change in the output of the monDEQ on perturbing the weights and biases from W, U, b to ¯ W, ¯ U, ¯ b is bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: The proof steps for Theorem 2 parallel closely those involved in the derivation of the Lipschitz constant with respect to the inputs, and are outlined in Appendix C.
Revised: The proof steps for Theorem 2 parallel closely those involved in the derivation of the Lipschitz constant with respect to the inputs, and are outlined in Appendix B.

Original: We highlight here again that the bound depends inversely on m , a design parameter in our control.
Revised: We highlight here again that the bound depends inversely on m , a design parameter in our control.

Original: Further, when compared to a similar perturbation bound derived in Neyshabur et al.
Revised: Further, when compared to a similar perturbation bound derived in Neyshabur et al.

Original: , we note that our perturbation bound for the monDEQ does not involve a depth-dependent product of spectral norms of weights.
Revised: , we note that our perturbation bound for the monDEQ does not involve a depth-dependent product of spectral norms of weights.

Original: Furthermore, although we state the theorem in terms of a perturbation of W (which can thus lead to a different strong monotonicity parameter ¯ m ), the bound can also be adapted to perturbations on A and
Revised: In addition, although we state the theorem in terms of a perturbation of W (which can thus lead to a different strong monotonicity parameter ¯ m ), the bound can also be adapted to perturbations on A and

Original: B in the typical monDEQ parameterization, which leads to a perturbed network that will necessarily still have the same monotonicity parameter m as the original (indeed, we take this approach in the next section, when deriving the generalization bound).
Revised: B in the typical monDEQ parameterization, which leads to a perturbed network that will necessarily still have the same monotonicity parameter m as the original (indeed, we take this approach in the next section, when deriving the generalization bound).

Original:  G ENERALIZATION BOUND FOR MON DEQ
Revised:  G ENERALIZATION BOUND FOR MON DEQ

Original: In this section, we demonstrate how the perturbation bound derived in Section 3.2 leads directly to a deterministic PAC-Bayes, margin-based bound on the monDEQ generalization error, following the analysis for DNNs of Neyshabur et al.
Revised: In this section, we demonstrate how the perturbation bound derived in Section 3.2 leads directly to a deterministic PAC-Bayes, margin-based bound on the monDEQ generalization error, following the analysis for DNNs of Neyshabur et al.

Original: A key difference from our work, however, is that the perturbation bound they derive involves the product of spectral norms of all the weight matrices in the DNN.
Revised: A key difference from our work, however, is that the perturbation bound they derive involves the product of spectral norms of all the weight matrices in the DNN.

Original: Thus, as the network gets deeper, their bound grows exponentially looser.
Revised: Thus, as the network gets deeper, their bound grows exponentially looser.

Original: As in Neyshabur et al.
Revised: As in Neyshabur et al.

Original: (2018), our generalization bound is based on two key ingredients.
Revised: (2018), our generalization bound is based on two key ingredients.

Original: The ﬁrst is their deterministic PAC-Bayes margin bound (Lemma 1 in the Appendix D), which adapts traditional
Revised: The ﬁrst is their deterministic PAC-Bayes margin bound (Lemma 1 in the Appendix C), which adapts traditional

Original: PAC-Bayes bounds to bound the the expected risk of a parameterized, deterministic classiﬁer in terms of its empirical margin loss.
Revised: PAC-Bayes bounds to bound the the expected risk of a parameterized, deterministic classiﬁer in terms of its empirical margin loss.

Original: The second is the perturbation bound on monDEQ with respect to weights as derived in Section 3.2 above.
Revised: The second is the perturbation bound on monDEQ with respect to weights as derived in Section 3.2 above.

Original: Crucially, since our perturbation bound does not explicitly involve a product of spectral norms of weights (which in the case of the monDEQ, would be an inﬁnite product), our ﬁnal generalization bound does not either.
Revised: Crucially, since our perturbation bound does not explicitly involve a product of spectral norms of weights (which in the case of the monDEQ, would be an inﬁnite product), our ﬁnal generalization bound does not either.

Original: The monDEQ model we consider here consists of a fully connected layer at the end that maps f to the output, so that f o ( x ) =
Revised: The monDEQ model we consider here consists of a fully connected layer at the end that maps f to the output, so that f o ( x ) =

Original: W o f ( x ) + b o , where W o and b o are the weights and bias in the output layer; these parameters are important to include here since they contribute directly to the perturbation bound.
Revised: W o f ( x ) + b o , where W o and b o are the weights and bias in the output layer; these parameters are important to include here since they contribute directly to the perturbation bound.

Original: We also restrict the input x to the monDEQ to lie in an l 2 norm ball of radius B .
Revised: We also restrict the input x to the monDEQ to lie in an l 2 norm ball of radius B .

Original: Let h denote the hidden dimension of the monDEQ, and M the size of the training set, and deﬁne β := max {(cid:107) U (cid:107) 2 , (cid:107) A (cid:107) 2 , (cid:107) b (cid:107) 2 , (cid:107) W o (cid:107) 2 } .
Revised: Let h denote the hidden dimension of the monDEQ, and M the size of the training set, and deﬁne β := max {(cid:107) U (cid:107) 2 , (cid:107) A (cid:107) 2 , (cid:107) b (cid:107) 2 , (cid:107) W o (cid:107) 2 } .

Original: Let L γ ( f o ) denote the expected margin loss at margin γ of the monDEQ on the data distribution D , where
Revised: Let L γ ( f o ) denote the expected margin loss at margin γ of the monDEQ on the data distribution D , where

Original: [Equation]
Revised: [Equation]

Original: and ˆ L γ ( f o ) denote the corresponding empirical margin loss on the training dataset.
Revised: and ˆ L γ ( f o ) denote the corresponding empirical margin loss on the training dataset.

Original: We are now ready to state our generalization bound for the monDEQ:
Revised: We are now ready to state our generalization bound for the monDEQ:

Original: Theorem 3 (Generalization bound for monDEQ) .
Revised: Theorem 3 (Generalization bound for monDEQ) .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: For any δ, γ > 0 , with probability at least 1 − δ over the training set of size M , we have that
Revised: For any δ, γ > 0 , with probability at least 1 − δ over the training set of size M , we have that

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Note that our bound above does not involve any depth-like term that scales exponentially, like the term that involves the product of spectral norms of the weight matrices in Neyshabur et al.
Revised: Note that our bound above does not involve any depth-like term that scales exponentially, like the term that involves the product of spectral norms of the weight matrices in Neyshabur et al.

Revised: (2018) (while still having the same dependence on h , which is √ h ln h ).

Original: To the best of our knowledge
Revised: To the best of our knowledge

Original: [Figure]
Revised: [Figure]

Original: this is the ﬁrst generalization bound for an implicit-layer model having effectively inﬁnite depth.
Revised: this is the ﬁrst generalization bound for an implicit-layer model having effectively inﬁnite depth.

Original: The proof of Theorem 3 is given in Appendix D.
Revised: The proof of Theorem 3 is given in Appendix C.

Original:  E XPERIMENTAL R ESULTS 5. L IPSCHITZ CONSTANTS
Revised:  E XPERIMENTAL R ESULTS5. L IPSCHITZ CONSTANTS

Original: In this section, we empirically verify the tightness of the Lipschitz constant of the monDEQ with respect to inputs.
Revised: In this section, we empirically verify the tightness of the Lipschitz constant of the monDEQ with respect to inputs.

Original: We conduct all our experiments on MNIST and CIFAR-10, for which several benchmarks exist on computing the Lipschitz constant.
Revised: We conduct all our experiments on MNIST and CIFAR-10, for which several benchmarks exist for computing the Lipschitz constant.

Original: We conducta variety of experiments for different monDEQ architectures (fully connected as well as convolutional) with varying parameters (strong-monotonicity parameter m and width h ), which we compare to DNNs with different depths and widths.
Revised: We conduct experiments for different monDEQ architectures (fully connected/convolutional) with varying parameters (strong-monotonicity parameter m and width h ), which we compare to DNNs with different depths and widths.

Original: We compute empirical lower bounds by maximizing the norm of the gradient at 50k randomly sampled points.
Revised: We compute empirical lower bounds by maximizing the norm of the gradient at 10k randomly sampled points.

Original: A naive upper bound can be computed as (cid:81) di =1 (cid:107) W i (cid:107) 2 .
Revised: A naive upper bound can be computed as (cid:81) di =1 (cid:107) W i (cid:107) 2 .

Original: We include these bounds wherever applicable.
Revised: We include these bounds wherever applicable.

Original: MNIST Here, we train DNNs for various depths from d = 3 , 4 , . . .
Revised: MNIST Here, we train DNNs for various depths from d = 3 , 4 , . . .

Original: , 14 for a ﬁxed hidden layer width h = 40 , and plot (Figure 1a) the bound on the Lipschitz constant given by the SDP-based method of Fazlyab et al.
Revised: , 14 for a ﬁxed hidden layer width h = 40 , and plot (Figure 1a) the bound on the Lipschitz constant given by the SDP-based method of Fazlyab et al.

Original: (2019) on these DNNs.
Revised: (2019) on these DNNs.

Original: We can observe that all estimates of the Lipschitz constant increase exponentially with depth.
Revised: We can observe that all estimates of the Lipschitz constant increase exponentially with depth.

Original: For comparison, in Figure 1b we plot our Lipschitz constant bounds for monDEQs with ﬁxed h = 40 , 60 , for a range of strong-monotonicity parameters m .
Revised: For comparison, in Figure 1b we plot our Lipschitz constant bounds for monDEQs with ﬁxed h = 40 , 60 , for a range of strong-monotonicity parameters m .

Original: We note that the DNNs all have test error of around 3%, while the monDEQ test error ranges from2.4%-4.3%, increasing with m (see Figure 5 in Appendix G for details). We see that the Lipschitz
Revised: We note that the DNNs all have test error of around 3%, while the monDEQ test error ranges from

Original: [Figure]
Revised: [Figure]

Original: [Figure][Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: 2.4%-4.3%, increasing with m (see Figure 5 in Appendix F for details).

Original:  constant of the monDEQ is much smaller, and that on increasing m , the Lipschitz constant of the monDEQ decreases, outlining how we can exercise control on the Lipschitz constant.
Revised: We see that the Lipschitz constant of the monDEQ is much smaller, and that on increasing m , the Lipschitz constant of the monDEQ decreases, outlining how we can exercise control on the Lipschitz constant.

Original: We also compare the Lipschitz constants of monDEQs and DNNs having the same width, for a ﬁxed depth d = 5 .
Revised: We also compare the Lipschitz constants of monDEQs and DNNs having the same width, for a ﬁxed depth d = 5 .

Original: The results are shown in Figure 6 in Appendix G. The DNN numbers are derived from
Revised: The results are shown in Figure 6 in Appendix F. The DNN numbers are derived from

Original: Figure 2(a) in (Fazlyab et al., 2019).
Revised: Figure 2(a) in (Fazlyab et al., 2019).

Original: We observe that the Lipschitz constant of the monDEQ for the same width (and essentially inﬁnite depth) is much lower than the bounds for regular DNNs.
Revised: We observe that the Lipschitz constant of the monDEQ for the same width (and essentially inﬁnite depth) is much lower than the bounds for regular DNNs.

Original: Next, using the bound derived in Section 3.1, we compute the Lipschitz constant of convolutional monDEQ architectures, namely single convolutional and multi-tier convolutional monDEQs.
Revised: Next, using the bound derived in Section 3.1, we compute the Lipschitz constant of convolutional monDEQ architectures, namely single convolutional and multi-tier convolutional monDEQs.

Original: We compare to the numbers in Figure 5 in Virmaux & Scaman (2018), which reports the Lipschitz constants computed by various methods for different CNNs with increasing depth.
Revised: We compare to the numbers in Figure 5 in Virmaux & Scaman (2018), which reports the Lipschitz constants computed by various methods for different CNNs with increasing depth.

Original: For our estimate on the single convolutional monDEQ, we use a single convolutional layer with 128 channels, whereas for the multi-tier convolutional monDEQ, we use 3 convolutional layers with 32, 64 and 128 channels.
Revised: For our estimate on the single convolutional monDEQ, we use a single convolutional layer with 128 channels, whereas for the multi-tier convolutional monDEQ, we use 3 convolutional layers with 32, 64 and 128 channels.

Original: In Figure 1c, we can observe that as for DNNs, the CNN Lipschitz constants estimated by existing methods also suffer with depth.
Revised: In Figure 1c, we can observe that as for DNNs, the CNN Lipschitz constants estimated by existing methods also suffer with depth.

Original: However, we can observe in Figure 1d that the Lipschitz bounds for convolutional monDEQs are much smaller.
Revised: However, we can observe in Figure 1d that the Lipschitz bounds for convolutional monDEQs are much smaller.

Original: Also, on increasing m , we can control the
Revised: Also, on increasing m , we can control the

Original: Lipschitz constant of both single as well as multi-tier convolutional monDEQs.
Revised: Lipschitz constant of both single as well as multi-tier convolutional monDEQs.

Revised: The test error for the convolutional monDEQs is 0.65%-3.22%, increasing with m (see Figure 5 in Appendix F), but is not reported for the CNNs in Virmaux & Scaman (2018).

Original: CIFAR-10 To demonstrate that the Lipschitz bounds scale to higher-dimensional datasets we conduct similar experiments on CIFAR-10.
Revised: CIFAR-10 To demonstrate that the Lipschitz bounds scale to larger datasets, we run similar experiments on CIFAR-10.

Original: Figure 2a showsthe our bound (solid blue lines) for singleconvolutional monDEQs (128 channels) with a range of m values, together with empirical lower bounds (dashed blue lines).
Revised: Figure 2a shows our bound (solid blue lines) for single-convolutional monDEQs (128 channels) with a range of m values, together with empirical lower bounds (dashed blue lines).

Original: Also shown are upper bounds (solid lines) and empirical lower bounds (dashed lines) for three standard CNN models (CNN sm, med, and lg, having 2, 4, and 6 convolutions respectively, detailed in Appendix G).
Revised: Also shown are upper bounds (solid lines) and empirical lower bounds (dashed lines) for three standard CNN models (CNN sm, med, and lg, having 2, 4, and 6 convolutions respectively, detailed in Appendix F).

Original: The upper bounds are computed using the Greedy SeqLip method of Virmaux & Scaman (2018).
Revised: The upper bounds are computed using the Greedy SeqLip method of Virmaux & Scaman (2018).

Original: We see that a) the monDEQ Lipschitz bounds decrease with m , and b) the upper bounds (and gap between upper and lower bounds) for the medium and large CNNs arevery large by comparison.
Revised: We see that a) the monDEQ Lipschitz bounds decrease with m , and b) the upper bounds (and gap between upper and lower bounds) for the medium and large CNNs are large by comparison.

Original: In Figure 2b we see that test error of the monDEQ increases with m fromabout 26% to 33%, and that, despite their much higher Lipschitz bounds, the three CNNs havetest error in a similar range.
Revised: In Figure 2b we see that test error of the monDEQ increases with m from 26% to 33%, and that, despite their much higher Lipschitz bounds, the three CNNs have similar test error.

Original: Finally, on CIFAR-10 with data augmentation, we also trained a larger multi-tier convolutional monDEQ with three convolutional layers with 64, 128, and 128 channels.
Revised: Finally, on CIFAR-10 with data augmentation, we also trained a larger multi-tier convolutional monDEQ with three convolutional layers with 64, 128, and 128 channels.

Original: This model obtains 10.25% test error and has a Lipschitzupper bound of 1996.86, which is on par with the upper bound of the medium CNN (plotted in brown; with L = 1554.01, test error =30.6%).
Revised: This model obtains 10.25% test error and has a Lipschitz

Original: Unrolling monDEQs
Revised: Unrolling monDEQs

Original: In this experiment, we study if unrolling monDEQs up to a ﬁnite depth and constructing an equivalent DNN with this depth leads to a tight estimate of[-the Lipschitz constant of the monDEQ. Concretely, we do this for two operator splitting methods in the monDEQ: Forwardbackward (FB) iterations and Peaceman-Rachford (PR) iterations. For each value of α in a range, we depth 
-]
Revised: In this experiment, we study if unrolling the monDEQ with m = 1 , h =up to a ﬁnite depth and constructing an equivalent DNN with this depth leads to a tight estimate of[-the Lipschitz constant of the monDEQ. Concretely, we do this for two operator splitting methods in the monDEQ: Forwardbackward (FB) iterations and Peaceman-Rachford (PR) iterations. For each value of α in a range, we depth 
-]

Original: [Figure]
Revised: [Figure]

Original: bound
Revised: the Lipschitz constant of the monDEQ.

Original: [Figure]
Revised: Concretely, we do this for two operator splitting methods in the monDEQ: Forward-backward (FB) iterations and Peaceman-Rachford (PR) iterations.

Original:  calculate the number of iterations (FB or PR) required for the converge within aspeciﬁed tolerance 1e-3, and construct the equivalent DNN with this depth.
Revised: For each value of α in a range, we calculate the number of iterations (FB or PR) required to converge within a tolerance 1e-3, and construct the equivalent DNN with this depth.

Original: Note that theseequivalent unrolled DNNs compute the same function as the monDEQ (up to tolerance), and therefore, must have the same
Revised: Note that these unrolled DNNs compute the same function as the monDEQ (up to tolerance), and therefore, must have the same

Original: Lipschitz constants of around 10.
Revised: Lipschitz constant (which is around 10).

Original: We compute naive upper bounds on the Lipschitz constants of these DNNs (we cannot use the SDP-based bound of Fazlyab et al.
Revised: We compute naive upper bounds on the Lipschitz constants of these DNNs (we cannot use the SDP-based bound of Fazlyab et al.

Original: (2019) due to technicalities in the construction of the unrolled DNN; refer to Appendix E).
Revised: (2019) due to technicalities in the construction of the unrolled DNN; refer to Appendix D).

Original: We  observe  that the upper bounds corresponding to both PR and FB iterations are in the range 10 5 to 10 13 , suggesting that unrolling the monDEQ and employing standard techniques on the unrolled monDEQs is not a viable way to bound the Lipschitz constant (see Figure 7 in Appendix G).
Revised: We can observe (Figure 7 in Appendix F) that the upper bounds corresponding to both PR and FB iterations are in the range 10 5 to 10 13 , suggesting that unrolling the monDEQ and employing standard techniques on the unrolled monDEQ is not a viable way to bound the Lipschitz constant.

Original: More details about the construction of these equivalent DNNs for both FB and PR iterations are provided in Appendix E.
Revised: More details about the construction of these equivalent DNNs for both FB and PR iterations are provided in Appendix D.

Original: 5. G ENERALIZATION BOUNDS
Revised: 5. G ENERALIZATION BOUNDS

Original: A key advantage of the monDEQ generalization bounds derived in Section 4 is the lack of any depth analog that can cause the bounds to grow exponentially.
Revised: A key advantage of the monDEQ generalization bounds derived in Section 4 is the lack of any depth analog that can cause the bounds to grow exponentially.

Original: To assess this aspect experimentally, we ﬁrst compute the DNN generalization bound following the protocol of Nagarajan & Kolter (2018).
Revised: To assess this aspect experimentally, we ﬁrst compute the DNN generalization bound following the protocol of Nagarajan & Kolter (2018).

Original: We train DNNs (width = 40 ) of varying depth of 3 to 14 layers, and compare to similar monDEQs with various m values.
Revised: We train DNNs (width = 40 ) of varying depth of 3 to 14 layers, and compare to similar monDEQs with various m values.

Original: Each model is trained on a sample of 4096 MNIST examples until the margin error at margin γ = 10 reaches below 10% which serves to standardize the experiments across choice of batch size and learning rate.
Revised: Each model is trained on a sample of 4096 MNIST examples until the margin error at margin γ = 10 reaches below 10% which serves to standardize the experiments across choice of batch size and learning rate.

Original: As widely reported, we see that DNN bounds increase exponentially with depth, ranging numerically from 10 4 for depth 3 networks to 10 8 (see Figure 3a).
Revised: As widely reported, we see that DNN bounds increase exponentially with depth, ranging numerically from 10 4 for depth 3 networks to 10 8 (see Figure 3a).

Original: For monDEQs of width = 40 , the bound decreases monotonically with m , and is conﬁned to the range 10 4 to 10 6 , as seen in Figure 3b (note the difference in scale).
Revised: For monDEQs of width = 40 , the bound decreases monotonically with m , and is conﬁned to the range 10 4 to 10 6 , as seen in Figure 3b (note the difference in scale).

Original: In contrast, the true test error of the DNNs increases only slightly with depth, and that of the monDEQs increases only slightly with m .
Revised: In contrast, the true test error of the DNNs increases only slightly with depth, and that of the monDEQs increases only slightly with m .

Original: Note that the DNNs and monDEQ s have comparable test error (see Figure 8 in Appendix H).
Revised: Note that the DNNs and monDEQ s have comparable test error (see Figure 8 in Appendix G).

Original: Finally, as done for Lipschitz bounds above, we compare our generalization bound to what we obtain by unrolling the monDEQ into a DNN, and then computing the Neyshabur et al.
Revised: Finally, as done for Lipschitz bounds above, we compare our generalization bound to what we obtain by unrolling the monDEQ into a DNN, and then computing the Neyshabur et al.

Original: (2018) bound moreor-less directly (see Appendix F for details).
Revised: (2018) bound moreor-less directly (see Appendix E).

Original: We do this only for forward-backward iterations, as the inverted operators of Peaceman-Rachford iterations complicate the analysis.
Revised: We do this only for FB iterations, as the inverted operators of PR iterations complicate the analysis.

Original: As we see in Figure 8c, the resulting bounds are quite high, though the difference with our bound is not as great as was seen for the unrolled Lipschitz bounds above.
Revised: As seen in Figure 8c, the resulting bounds are quite high, though the difference with our bound is not as great as was seen for the unrolled Lipschitz bounds above.

Original: We attribute this to the fact that our generalization bound technique is a minimal modiﬁcation to that of Neyshabur et al.
Revised: We attribute this to the fact that our generalization bound technique is a minimal modiﬁcation to that of Neyshabur et al.

Original: ; we expect that it can besigniﬁcantly tightened with more reﬁned analysis.
Revised: ; we expect that it can be tightened with more reﬁned analysis.

Original: 5. A DVERSARIAL ROBUSTNESS OF MON DEQ S
Revised: 5. A DVERSARIAL ROBUSTNESS OF MON DEQ S

Original: In this section, we empirically demonstrate an important use case for the tight Lipschitz constants of the monDEQ: robustness to adversarial examples.
Revised: In this section, we empirically demonstrate an important use case for the tight Lipschitz constants of the monDEQ: robustness to adversarial examples.

Original: We experiment with both certiﬁed adversarial (cid:96)robustness as well as empirical robustness to adversarial PGD (cid:96) 2 -bounded attacks.
Revised: We experiment with both certiﬁed adversarial (cid:96)robustness as well as empirical robustness to adversarial PGD (cid:96) 2 -bounded attacks.

Original: Here we describe our results on MNIST, and report similar experiments on CIFAR-10 in Appendix I.
Revised: Here we describe our results on MNIST, and report similar experiments on CIFAR-10 in Appendix H.

Original: [Figure]
Revised: [Figure]

Revised: C

Original:  i 
Revised: [Figure] t i f i [Figure] R

Original: [Figure]
Revised: [Figure]

Original: i
Revised: s

Original: [Figure]
Revised: [Figure]

Revised: T e s t

Original: A cc u r a c y
Revised: A cc u r a c y

Original: [Figure]
Revised: [Figure]

Original: Certiﬁed adversarial robustness Consider any point x (cid:48) within an l 2 ball of radius (cid:15) around x .
Revised: Certiﬁed adversarial robustness Consider any point x (cid:48) within an l 2 ball of radius (cid:15) around x .

Original: Then, we have that
Revised: Then, we have that

Original: [Equation]
Revised: [Equation]

Original: Deﬁne margin ( x ) = f
Revised: Deﬁne margin ( x ) = f

Original: ( x ) y − max i (cid:54) =
Revised: ( x ) y − max i (cid:54) =

Original: y f ( x ) i , where f ( x ) y is the logit corresponding to the label y of the input x .
Revised: y f ( x ) i , where f ( x ) y is the logit corresponding to the label y of the input x .

Original: Then if L(cid:15) ≤ 12 margin ( x ) , we are certiﬁed robust to any perturbed input x (cid:48) within an l 2 -ball of radius (cid:15) around x .
Revised: Then if L(cid:15) ≤ 12 margin ( x ) , we are certiﬁed robust to any perturbed input x (cid:48) within an l 2 -ball of radius (cid:15) around x .

Original: Thus, we can empirically compare DNNs and monDEQs with regards to this certiﬁcate on MNIST.
Revised: Thus, we can empirically compare DNNs and monDEQs with regards to this certiﬁcate on MNIST.

Original: For a range of (cid:15) values, we compute the (certiﬁed) robust test accuracy (fraction of points in test set for which the aforementioned condition holds) for trained DNNs as well as monDEQs.
Revised: For a range of (cid:15) values, we compute the (certiﬁed) robust test accuracy (fraction of points in test set for which the aforementioned condition holds) for trained DNNs as well as monDEQs.

Original: We note here that our choice of (cid:15) values corresponds to inputs normalized in the range [0 , 1] .
Revised: We note here that our choice of (cid:15) values corresponds to inputs normalized in the range [0 , 1] .

Original: For DNNs, just as in Figure 1a, we vary the depth for ﬁxed h = 40 , and use the
Revised: For DNNs, just as in Figure 1a, we vary the depth for ﬁxed h = 40 , and use the

Original: L values computed by using the method in Fazlyab et al.
Revised: L values computed by using the method in Fazlyab et al.

Original: For monDEQs, we set width h = 40 , m = 0 .
Revised: For monDEQs, we set width h = 40 , m = 0 .

Original: 1 , 20 and substitute our upper bound for L .
Revised: 1 , 20 and substitute our upper bound for L .

Original: Note that since the L values for monDEQs observed in Section 5.1 were signiﬁcantly smaller than those for the DNNs, one would expect the condition for the certiﬁcate to hold more easily for monDEQs.
Revised: Note that since the L values for monDEQs observed in Section 5.1 were signiﬁcantly smaller than those for the DNNs, one would expect the condition for the certiﬁcate to hold more easily for monDEQs.

Original: Indeed, we verify this in our experiments.
Revised: Indeed, we verify this in our experiments.

Original: In Figure 4a, we can observe that the robust test accuracy for the monDEQ with m = 20 at (cid:15) = 0 .
Revised: In Figure 4a, we can observe that the robust test accuracy for the monDEQ with m = 20 at (cid:15) = 0 .

Original: 2 is 51% , while that for the best DNN ( d = 3) , is 26% .
Revised: 2 is 51% , while that for the best DNN ( d = 3) , is just 4% .

Original: This illustrates that monDEQs allow for better certiﬁcates to adversarial robustness, owing to their small Lipschitz constants, and the ability to control it by setting m .
Revised: This illustrates that monDEQs allow for better certiﬁcates to adversarial robustness, owing to their small Lipschitz constants, and the ability to control it by setting m .

Original: Empirical robustness We also assess the empirical robustness of monDEQs to l 2 -bounded Projected Gradient Descent attacks (implemented as part of the Foolbox toolbox (Rauber et al., 2017; 2020)) on both, trained monDEQs and DNNs on MNIST, and compute the accuracy on these adversarially perturbed test examples.
Revised: Empirical robustness We also assess the empirical robustness of monDEQs to l 2 -bounded Projected Gradient Descent attacks (implemented as part of the Foolbox toolbox (Rauber et al., 2017; 2020)) on both, trained monDEQs and DNNs on MNIST, and compute the accuracy on these adversarially perturbed test examples.

Original: Figure 4b shows the results: in general, over a range of (cid:15) values, the robust test accuracy of the monDEQ with m = 20 is larger than that of the DNNs.
Revised: Figure 4b shows the results: in general, over a range of (cid:15) values, the robust test accuracy of the monDEQ with m = 20 is larger than that of the DNNs.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we derived Lipschitz bounds for monotone DEQs, a recently proposed class of impicitlayer networks, and showed that they depend in a straighforward manner on the strong monotonicity parameter m of these networks.
Revised: In this paper, we derived Lipschitz bounds for monotone DEQs, a recently proposed class of impicitlayer networks, and showed that they depend in a straighforward manner on the strong monotonicity parameter m of these networks.

Original: Having derived a Lipschitz bound with respect to perturbation in the weights, we were able to derive a PAC-Bayesian generalization bound for the monotone DEQ, which does not depend exponentially on depth.
Revised: Having derived a Lipschitz bound with respect to perturbation in the weights, we were able to derive a PAC-Bayesian generalization bound for the monotone DEQ, which does not depend exponentially on depth.

Original: We showed empirically that our bounds are sensible, can be controlled by choosing m suitably, and do not suffer with increasing depth of the network.
Revised: We showed empirically that our bounds are sensible, can be controlled by choosing m suitably, and do not suffer with increasing depth of the network.

Original: As future work, we aim to analyze the vacuousness of the derived generalization bound.
Revised: As future work, we aim to analyze the vacuousness of the derived generalization bound.

Original: As such, since our bound does not suffer exponentially with depth, we hope to be able to make the analysis tighter and derive a non-vacuous generalization bound.
Revised: As such, since our bound does not suffer exponentially with depth, we hope to be able to make the analysis tighter and derive a non-vacuous generalization bound.

Original: A P ROOF OF P ROPOSITION
Revised: A P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Now, note that by the strong monotonicity of the monDEQ,
Revised: Now, note that by the strong monotonicity of the monDEQ,

Original: [Equation]
Revised: [Equation]

Original: which implies that ( x − y ) T ( I − W )( x − y ) ≥ m (cid:107) x
Revised: which implies that ( x − y ) T ( I − W )( x − y ) ≥ m (cid:107) x

Original: − y (cid:107) 22 .
Revised: − y (cid:107) 22 .

Original: Substituting this bound above
Revised: Substituting this bound above

Original: [Equation]
Revised: [Equation]

Original: we have that
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: Thus, we have that L [ T ] ≤ (cid:112) 1 − 2 αm + α 2 L
Revised: Thus [Equation] we have that L [ T ] ≤ (cid:112) 1 − 2 αm + α 2 L

Original: [ I − W ]
Revised: [ I − W [Equation]

Original: B P ROOF OF P ROPOSITION
Revised: B P ROOF OF T HEOREM

Original: Proof.
Revised: In order to derive the perturbation bound in Theorem

Original: [Equation]
Revised: [Equation]

Original: − 1 ( W, U, b ) + α ( Ux + b ) (cid:107)( ReLU is 1-Lipschitz ) ≤ (cid:107) T (cid:107) 2 (cid:107) f k [-− 1 ( W, U, b ) (cid:107) 
 + α (cid:107) Ux + b (cid:107)= L [ T ] (cid:107) f k − 1 ( W, U, b ) (cid:107) 
-]
Revised: we ﬁrst state the following proposition, which bounds the norm of the output after k [-− 1 ( W, U, b ) (cid:107) 
 + α (cid:107) Ux + b (cid:107)= L [ T ] (cid:107) f k − 1 ( W, U, b ) (cid:107) 
-]forward-backward iterations.

Original: [-+ α (cid:107) Ux + b (cid:107)≤ L [ T ] k (cid:107) f 
 ( W, U, b ) (cid:107) 
 + α (cid:107)-]
Revised: [-+ α (cid:107) Ux + b (cid:107)≤ L [ T ] k (cid:107) f 
 ( W, U, b ) (cid:107) 
 + α (cid:107)-]Proposition 2.

Original: Ux + b (cid:107)k − 1 (cid:88) i = L [ T ] i = α (cid:107) Ux + b (cid:107)k − 1 (cid:88) i = L [ T ] i ( since f 0 ( W, U, b ) = 0) ≤ α (cid:107)
Revised: Let f k ( W, U, b ) denote the k th iterate of the forward-backward iterations of the monDEQ parameterized by W, U, b on a ﬁxed arbitrary input x .

Original: Ux + b (cid:107)∞ (cid:88) i =

Original: L [ T ] i = α (cid:107)

Original: Ux + b (cid:107)1 − L [ T] ( since L [ T ] < 1) C P ROOF OF T HEOREM Proof.
Revised: Further, let T ( W ) = (1 − α )

Revised: I + αW .

Revised: Then, we have that

Revised: [Equation]

Revised: Proof of Proposition 2.

Revised: [Equation]

Revised: We are now ready to prove Theorem 2.

Revised: Proof of Theorem 2.

Original: Denote ¯ f k = f k
Revised: Denote ¯ f k = f k

Original: ( ¯ W, ¯ U, ¯ b ) and f k = f k
Revised: ( ¯ W, ¯ U, ¯ b ) and f k = f k

Original: ( W, U, b ) .
Revised: ( W, U, b ) .

Original: Further, denote ∆ k = (cid:107) ¯ f k − f k (cid:107) 2 .
Revised: Further, denote ∆ k = (cid:107) ¯ f k − f k (cid:107) 2 .

Original: For α ∈ (cid:16) 0 , min (cid:16) 2 m L [ I− W ] 2 , 2 ¯ m L [ I − ¯ W ] 2 (cid:17)(cid:17) , we have from Proposition 1 that both (cid:107) T ( W ) (cid:107) 2 , (cid:107) T ( ¯ W ) (cid:107) 2 <
Revised: For α ∈ (cid:16) 0 , min (cid:16) [Equation] m L [ I

Revised: [Equation]

Revised: W ]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: , we have from Proposition 1 that both

Revised: (cid:107) T ( W ) (cid:107) 2 , (cid:107) T ( ¯ W ) (cid:107)

Revised: 2 < 1 .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Notice here again that the above inequality holds for all k .
Revised: Notice here again that the above inequality holds for all k .

Original: Taking the limit as k → ∞ similar to the step in the proof of Theorem
Revised: Taking the limit as k → ∞ similar to the step in the proof of Theorem

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Finally, taking α →
Revised: Finally [Equation] taking α →

Original: [Equation]
Revised: [Equation]

Original: we have that
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: L’Hopital’s rule )

Original: D P ROOF OF T HEOREM
Revised: C P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: We ﬁrst state Lemma 1 from Neyshabur et al.
Revised: We ﬁrst state Lemma 1 from Neyshabur et al.

Original: Lemma 1 (Lemma 1 from Neyshabur et al.
Revised: Lemma 1 (Lemma 1 from Neyshabur et al.

Original: Let f w be any predictor with parameters w , and let P denote any distribution on the parameters that is independent of the training data.
Revised: Let f w be any predictor with parameters w , and let P denote any distribution on the parameters that is independent of the training data.

Original: Then, for any δ, γ > 0 , with probability ≥ 1 − δ over the training data of size M , for any w , and any random perturbation u such that P [max x (cid:107)
Revised: Then, for any δ, γ > 0 , with probability ≥ 1 − δ over the training data of size M , for any w , and any random perturbation u such that P [max x (cid:107)

Original: f w + u ( x ) − f w ( x ) (cid:107) ∞ <
Revised: f w + u ( x ) − f w ( x ) (cid:107) ∞ <

Original: γ 4 ] ≥ 12 , we have
Revised: γ 4 ] ≥ 12 , we have

Original: [Equation]
Revised: [Equation]

Original: Now, we derive a perturbation bound for the monDEQ when we incorporate the a fully connected layer at the end, as mentioned in Section 4.
Revised: Now, we derive a perturbation bound for the monDEQ when we incorporate the a fully connected layer at the end, as mentioned in Section 4.

Original: That is, we consider f o ( x ) =
Revised: That is, we consider f o ( x ) =

Original: W o f ( x ) + b 0 where f is the output of the ﬁxed-point iterations of the monDEQ.
Revised: W o f ( x ) + b 0 where f is the output of the ﬁxed-point iterations of the monDEQ.

Original: Next, we consider perturbations ∆ A , ∆ B , ∆ U , ∆ b , ∆ W o , ∆ b o for A, B, U, b, W o , b o respectively.
Revised: Next, we consider perturbations ∆ A , ∆ B , ∆ U , ∆ b , ∆ W o , ∆ b o for A, B, U, b, W o , b o respectively.

Original: The entries in the perturbation matrices are each drawn independently from a Gaussian N (0 , σ 2 ) .
Revised: The entries in the perturbation matrices are each drawn independently from a Gaussian N (0 , σ 2 ) .

Original: Let ¯ f o to denote the function at the perturbed values of the weights.
Revised: Let ¯ f o to denote the function at the perturbed values of the weights.

Original: Then, we have that
Revised: Then, we have that

Original: [Equation]
Revised: [Equation]

Original: where ∆ is the bound from Theorem 2.
Revised: where ∆ is the bound from Theorem 2.

Original: Now, let β = max( (cid:107) U (cid:107) 2 , (cid:107) A (cid:107) 2 , (cid:107) W o (cid:107) 2 , (cid:107) b (cid:107) 2 ) .
Revised: Now, let β = max( (cid:107) U (cid:107) 2 , (cid:107) A (cid:107) 2 , (cid:107) W o (cid:107) 2 , (cid:107) b (cid:107) 2 ) .

Original: Just as in (Neyshabur et al., 2018), since we cannot use β in determining the parameters of the prior distribution P in Lemma 1, we will consider predetermined values ˜ β on a grid, and then do a union bound.
Revised: Just as in (Neyshabur et al., 2018), since we cannot use β in determining the parameters of the prior distribution P in Lemma 1, we will consider predetermined values ˜ β on a grid, and then do a union bound.

Original: For now, we ﬁx ˜ β , and consider all the values β such that | β − ˜ β
Revised: For now, we ﬁx ˜ β , and consider all the values β such that | β − ˜ β

Original: | < c 1 β for some constant c 1 < 1 .
Revised: | < c 1 β for some constant c 1 < 1 .

Original: Since the entries in the perturbations are drawn from N (0 , σ 2 ) , we have the following bound on the l 2 norms of these perturbations:
Revised: Since the entries in the perturbations are drawn from N (0 , σ 2 ) , we have the following bound on the l 2 norms of these perturbations:

Original: [Equation]
Revised: [Equation]

Original: where ∆ · is a placeholder for each of the perturbation matrices.
Revised: where ∆ · is a placeholder for each of the perturbation matrices.

Original: Thus, we have that with probability ≥ 1/2, all of the (cid:107) ∆ · (cid:107) 2 are bounded above by σ (cid:112) 2 h ln(24 h ) :
Revised: Thus, we have that with probability ≥ 1/2, all of the (cid:107) ∆ · (cid:107) 2 are bounded above by σ (cid:112) 2 h ln(24 h ) :

Original: = ω .
Revised: = ω .

Original: Now, we bound the perturbation in (cid:107) W (cid:107) 2 when A and B are perturbed.
Revised: Now, we bound the perturbation in (cid:107) W (cid:107) 2 when A and B are perturbed.

Original: We have that
Revised: We have that

Original: [Equation]
Revised: [Equation]

Original: probability
Revised: probability

Original: [Equation]
Revised: [Equation]

Original:  Substituting this above, we have that for all x , with probability at least 1 / 2 ,[Equation]
Revised: [Equation] Substituting this above, we have that for all x , with probability at least 1 / 2 ,

Original: ¯

Original: [Equation]
Revised: [Equation]

Original: Here, let c 2 > 0 be some constant such that ω ≤ c 2 β .
Revised: Here, let c 2 > 0 be some constant such that ω ≤ c 2 β .

Original: Thus, we have that
Revised: Thus, we have that

Original: [Equation]
Revised: [Equation]

Original: ¯

Original: [Equation]

Original: Setting σ =
Revised: Setting σ =

Original: [Equation]
Revised: [Equation]

Original: − c 1 ) 2 + m 2 (1 − c 1 ) 3 ) makes this ≤ γ 4 .
Revised: c 1 ) 2 + m 2 (1 − c 1 ) 3 ) makes this ≤ γ 4 .

Original: Since we need ω ≤ c 2 β we can take the smallest c 2 such that
Revised: Since we need ω ≤ c 2 β we can take the smallest c 2 such that

Original: [Equation]
Revised: [Equation]

Revised: β

Original: Taking c 

Revised: Taking c

Original: = (1+ c 1 ) γ 4˜ β sufﬁces.
Revised: = (1+ c 1 ) γ 4˜ β sufﬁces.

Original: We will later plug in the value
Revised: We will later plug in the value

Original: [Equation]
Revised: [Equation]

Original: to bound c 2 .
Revised: to bound c 2 .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Then, by Lemma 1, we have that with probability 1 − δ , 
Revised: Then, by Lemma 1, we have that with probability 1 − δ , L 0 ( f [Equation] ≤

Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: Now we need to take a union bound over ˜ β
Revised: Now we need to take a union bound over ˜ β

Original: so that the above result holds for all β .
Revised: so that the above result holds for all β .

Original: Observe that we only need to consider β in the range
Revised: Observe that we only need to consider β in the range

Original: [Equation]
Revised: [Equation]

Original: If β ≤ [Equation] then | f ( x ) | ≤ [Equation]
Revised: If β ≤ γm 2( B +1) then | f ( x ) | ≤ β ( B +1) m

Original: m ≤ [Equation]
Revised: ≤ γ 2 so ˆ L γ ( f ) > 1 .

Original: If [Equation] ) then the second term in the ﬁrst term in the numerator is greater than 1 so the theorem holds trivially.
Revised: If β ≥ γm √ M 2( B +1) then the second term in the ﬁrst term in the numerator is greater than 1 so the theorem holds trivially.

Original: So | β − ˜ β | ≤ c 1 γm 2( B +1) guarantees | β − ˜ β | ≤ c 1 β in this range.
Revised: So | β − ˜ β | ≤ c 1 γm 2( B +1) guarantees | β − ˜ β | ≤ c 1 β in this range.

Original: So we can use a cover of size √
Revised: So we can use a cover of size √

Original: M/ 2 c 1 : This amounts to replacing ln 6 M/δ with ln 3 M 3 / 2 /c 1 δ .
Revised: M/ 2 c 1 : This amounts to replacing ln 6 M/δ with ln 3 M 3 / 2 /c 1 δ .

Original: Finally, substituting c 3 = (1+ c 1 ) γ 4(1 − c 1 ) β , we get the theorem statement.
Revised: Finally, substituting c 3 = (1+ c 1 ) γ 4(1 − c 1 ) β , we get the theorem statement.

Original: We can also effectively optimize over c 1 to remove it from the ﬁnal bound.
Revised: We can also effectively optimize over c 1 to remove it from the ﬁnal bound.

Original: E U NROLLING F ORWARD -B ACKWARD AND P EACEMAN -R ACHFORD ITERATIONS
Revised: D U NROLLING F ORWARD -B ACKWARD AND P EACEMAN -R ACHFORD ITERATIONS

Original: In this section, we derive the form of the equivalent feedforward DNN that computes the same quantity as running k iterations of either Forward-Backward or Peaceman-Rachford iterations, which are operator-splitting methods for computing the ﬁxed point of equation 1 (refer Winston & Kolter (2020)).
Revised: In this section, we derive the form of the equivalent feedforward DNN that computes the same quantity as running k iterations of either Forward-Backward or Peaceman-Rachford iterations, which are operator-splitting methods for computing the ﬁxed point of equation 1 (refer Winston & Kolter (2020)).

Original: E. U NROLLING F ORWARD -B ACKWARD I TERATIONS 
Revised: D. U NROLLING F ORWARD -B ACKWARD I TERATIONS One step in the Forward-Backward iterations computes

Original: One step in the Forward-Backward iterations computes

Original: [Equation]
Revised: [Equation]

Original: To simulate k steps of these computations as a depth k -feedforward network, let us construct the following weight matrices:
Revised: To simulate k steps of these computations as a depth k -feedforward network, let us construct the following weight matrices:

Original: [Equation]
Revised: [Equation]

Original: Then, we can observe that
Revised: Then, we can observe that

Original: [Equation]
Revised: [Equation]

Original: Here, σ applies only to the top h coordinates corresponding to z ( i ) .
Revised: Here, σ applies only to the top h coordinates corresponding to z ( i ) .

Original: Finally, we multiply the output of the monDEQ after k iterations with the output weights, to obtain f o ( x ) as
Revised: Finally, we multiply the output of the monDEQ after k iterations with the output weights, to obtain f o ( x ) as

Original: [Equation]
Revised: [Equation]

Original: Thus, the equivalent depth- k DNN which we construct would have weight matrices
Revised: Thus, the equivalent depth- k DNN which we construct would have weight matrices

Original: W 1 , [ W u ] k − 1 i =1 , W f .
Revised: W 1 , [ W u ] k − 1 i =1 , W f .

Original: In this DNN, the ReLUs in the intermediate layers would only apply to the top coordinates corresponding to z (this is the technical reason why we can’t use the SDP bound given by (Fazlyab et al., 2019) for computing the Lipschitz constants of these unrolled networks, since they require the same nonlinearity to apply pointwise at all coordinates).
Revised: In this DNN, the ReLUs in the intermediate layers would only apply to the top coordinates corresponding to z (this is the technical reason why we can’t use the SDP bound given by (Fazlyab et al., 2019) for computing the Lipschitz constants of these unrolled networks, since they require the same nonlinearity to apply pointwise at all coordinates).

Original: E. U NROLLING P EACEMAN -R ACHFORD I TERATIONS
Revised: D. U NROLLING P EACEMAN -R ACHFORD I TERATIONS

Original: Deﬁne V =
Revised: Deﬁne V =

Original: ( I + α ( I − W )) − 1 .
Revised: ( I + α ( I − W )) − 1 .

Original: One step in the Peaceman-Rachford iterations computes
Revised: One step in the Peaceman-Rachford iterations computes

Original: [Equation]
Revised: [Equation]

Original: To simulate k steps of these computations as a depth k -feedforward network, let us construct the following weight matrices:
Revised: To simulate k steps of these computations as a depth k -feedforward network, let us construct the following weight matrices:

Original: [Equation]
Revised: [Equation]

Original: Then, we can observe that
Revised: Then, we can observe that

Original: [Equation]
Revised: [Equation]

Original: Here, σ applies only to the top h coordinates corresponding to z ( i ) .
Revised: Here, σ applies only to the top h coordinates corresponding to z ( i ) .

Original: Finally, we multiply the output of the monDEQ after k iterations with the output weights, to obtain f o ( x ) as
Revised: Finally, we multiply the output of the monDEQ after k iterations with the output weights, to obtain f o ( x ) as

Original: [Equation]
Revised: [Equation]

Original: Thus, the equivalent depth- k DNN which we construct would have weight matrices
Revised: Thus, the equivalent depth- k DNN which we construct would have weight matrices

Original: W 1 , [ W u ] k − 1 i =1 , W f , and the ReLUs in the intermediate layers would only apply to the top coordinates corresponding to z .
Revised: W 1 , [ W u ] k − 1 i =1 , W f , and the ReLUs in the intermediate layers would only apply to the top coordinates corresponding to z .

Original: F G ENERALIZATION BOUND FOR UNROLLED MON DEQ
Revised: E G ENERALIZATION BOUND FOR UNROLLED MON DEQ

Original: In this section, we derive a generalization bound for an unrolled monDEQ after unrolling the forward-backward iterations d times and constructing the equivalent depth- d network.
Revised: In this section, we derive a generalization bound for an unrolled monDEQ after unrolling the forward-backward iterations d times and constructing the equivalent depth- d network.

Original: The analysis follows the derivation of the generalization bound for DNNs by (Neyshabur et al., 2018), but we have to be careful since the weight matrices across the hidden layers are all the same and are of a certain parameterized form.
Revised: The analysis follows the derivation of the generalization bound for DNNs by (Neyshabur et al., 2018), but we have to be careful since the weight matrices across the hidden layers are all the same and are of a certain parameterized form.

Original: Since the analysis in (Neyshabur et al., 2018) does not include biases in the DNNs, here, we consider unrolling monDEQs which do not have the bias term b .
Revised: Since the analysis in (Neyshabur et al., 2018) does not include biases in the DNNs, here, we consider unrolling monDEQs which do not have the bias term b .

Original: Take each layer weights to be
Revised: Take each layer weights to be

Original: [Equation]
Revised: [Equation]

Original: where the network is now a function of
Revised: where the network is now a function of

Original: [Equation]
Revised: [Equation]

Original: and the ReLU applies only to the z ( i ) component.
Revised: and the ReLU applies only to the z ( i ) component.

Original: And
Revised: And

Original: [Equation]
Revised: [Equation]

Original: Our prior distribution will now be over ∆ W o , ∆ A , ∆ B , ∆ U ∼ N (0 , σ 2 I ) .
Revised: Our prior distribution will now be over ∆ W o , ∆ A , ∆ B , ∆ U ∼ N (0 , σ 2 I ) .

Original: Similar to the corresponding step in D above, we have that with probability ≥ 12 , the perturbations are each bounded by ω :
Revised: Similar to the corresponding step in C above, we have that with probability ≥ 12 , the perturbations are each bounded by ω :

Original: = σ (cid:112) 2 h ln(16 h ) .
Revised: = σ (cid:112) 2 h ln(16 h ) .

Original: Also
Revised: Also

Original: [Equation]
Revised: [Equation]

Original: And so
Revised: And so

Original: [Equation]
Revised: [Equation]

Original: We take β = max {(cid:107) A (cid:107) 2 , (cid:107) B (cid:107) 2 , (cid:107) W u (cid:107) 2 , (cid:107) W o (cid:107) 2 } .
Revised: We take β = max {(cid:107) A (cid:107) 2 , (cid:107) B (cid:107) 2 , (cid:107) W u (cid:107) 2 , (cid:107) W o (cid:107) 2 } .

Original: We assume that | ˜ β − β | ≤ 1 d β and d ≥ 2 , so we have β < e ˜ β .
Revised: We assume that | ˜ β − β | ≤ 1 d β and d ≥ 2 , so we have β < e ˜ β .

Original: Then applying Lemma 2 in (Neyshabur et al., 2018), we have
Revised: Then applying Lemma 2 in (Neyshabur et al., 2018), we have

Original: [Equation]
Revised: [Equation]

Original: if we choose 
Revised: if we choose σ = γ

Original: [Equation]
Revised: [Equation]

Original: =
Revised: Let (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Let
Revised: W · (cid:107) 2 F = (cid:107) A [Equation] [Equation]

Revised: +

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original:  W 
Revised: U (cid:107) 2 F + (cid:107) W [Equation] (cid:107)

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Revised: KL

Original: [Equation]
Revised: [Equation]

Original: Now, instantiating Lemma 1 above, we have that with probability at least 1 − δ ,L
Revised: Now, instantiating Lemma 1 above, we have that with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Following Neyshabur et al.
Revised: Following Neyshabur et al.

Original: (2018), we need to take a union bound over ˜ β
Revised: (2018), we need to take a union bound over ˜ β

Original: so that the above result holds for all β .
Revised: so that the above result holds for all β .

Original: Observe that we only need to consider β in the range
Revised: Observe that we only need to consider β in the range

Original: [Equation]
Revised: [Equation]

Original: If β d ≤ γm 2 B then | f
Revised: If β d ≤ γm 2 B then | f

Original: ( x ) | ≤ β d B ≤ γ 

Revised: ( x ) | ≤ β d B ≤ γ

Original: so ˆ L γ ( f ) > 1 .
Revised: so ˆ L γ ( f ) > 1 .

Original: If β d ≥ γ √ M 2 B then the second term in the ﬁrst term in the numerator is greater than 1 so the result holds trivially.
Revised: If β d ≥ γ √ M 2 B then the second term in the ﬁrst term in the numerator is greater than 1 so the result holds trivially.

Original: So | β − ˜ β | ≤ 1 d (cid:0) γ 2 B (cid:1) 1 /d guarantees | β − ˜ β | ≤ 1 d β in this range.
Revised: So | β − ˜ β | ≤ 1 d (cid:0) γ 2 B (cid:1) 1 /d guarantees | β − ˜ β | ≤ 1 d β in this range.

Original: So we can use a cover of size dM 1 / 2 d , just as in Neyshabur et al.
Revised: So we can use a cover of size dM 1 / 2 d , just as in Neyshabur et al.

Original: G A DDITIONAL MON DEQ L IPSCHITZ BOUND RESULTS
Revised: F A DDITIONAL MON DEQ L IPSCHITZ BOUND RESULTS

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: (a) Width 40 monDEQs shown in Figure 1b

Original: [Figure]
Revised: [Figure]

Revised: Convolutional monDEQs shown in Figure 1d

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: MNIST test error
Revised: MNIST test error

Original: In Figure 5 we plot the test error of the width 40 monDEQs for which the
Revised: In Figure 5 we plot the test error of the width 40 monDEQs for which the

Original: Lipschitz bounds are given in Figure 1b.
Revised: Lipschitz bounds are given in Figure 1b.

Original: We see that it increases from 2.4% for m = 0 .
Revised: We see that it increases from 2.4% for m = 0 .

Original: 5 to 4.2% for m = 20 .
Revised: 5 to 4.2% for m = 20 .

Original: For comparison, the DNNs shown in Figure 1a have test error between 2.8% and 3.2%, with no trend w.r.t. depth.
Revised: For comparison, the DNNs shown in Figure 1a have test error between 2.8% and 3.2%, with no trend w.r.t. depth.

Original: MNIST width experiments Figure 6 shows the lower and upper bounds for DNNs and monDEQs of varying widths, as described in Section 5.1.
Revised: MNIST width experiments Figure 6 shows the lower and upper bounds for DNNs and monDEQs of varying widths, as described in Section 5.1.

Original: Figure 6a is transcribed from Fazlyab et al.
Revised: Figure 6a is transcribed from Fazlyab et al.

Original: (2019) ﬁgure 2(a).
Revised: (2019) ﬁgure 2(a).

Original:  The details of the CNN sm, CNN med, and CNN lg models used in
Revised: CIFAR-10 CNN models The details of the CNN sm, CNN med, and CNN lg models used in

Original: Section 5.1 are as follows:
Revised: Section 5.1 are as follows:

Original: CNN sm: Layer #Layer
Revised: CNN sm: Layer #

Original:  channels out stride
Revised: Layer # channels out stride

Original: Conv2D Conv2D
Revised: Conv2D Conv2D

Original: Linear Linear
Revised: Linear Linear

Original: CNN med: Layer #Layer
Revised: CNN med Layer #

Original:  channels out stride
Revised: Layer # channels out stride

Original: Conv2D Conv2D
Revised: Conv2D Conv2D

Original: Conv2D Conv2D
Revised: Conv2D Conv2D

Original: Linear Linear
Revised: Linear Linear

Original: CNN lg: Layer #Layer
Revised: CNN lg Layer #

Original:  # channels out stride
Revised: Layer # channels out stride

Original: Conv2D Conv2D
Revised: Conv2D Conv2D

Original: Conv2D Conv2D
Revised: Conv2D Conv2D

Original: Conv2D Conv2D
Revised: Conv2D Conv2D

Original: Linear Linear
Revised: Linear Linear

Original: All models use ReLU activations and kernels of size 3.
Revised: All models use ReLU activations and kernels of size 3.

Original: Unrolling monDEQs
Revised: Unrolling monDEQs

Original: The approximate upper bounds on monDEQ Lipschitz constants obtained by unrolling the operator splitting methods are shown in Figure 7.
Revised: The approximate upper bounds on monDEQ Lipschitz constants obtained by unrolling the operator splitting methods are shown in Figure 7.

Original: H A DDITIONAL MON DEQ GENERALIZATION BOUND RESULTS
Revised: G A DDITIONAL MON DEQ GENERALIZATION BOUND RESULTS

Original: We validate that both the monDEQ and DNN models considered in Section 5.2 for generalization bound comparisons indeed achieve comparable accuracy as well.
Revised: We validate that both the monDEQ and DNN models considered in Section 5.2 for generalization bound comparisons indeed achieve comparable accuracy as well.

Original: Figures 8a, 8b illustrate this: we can observe that the various models considered all achieve test error in the same range.
Revised: Figures 8a, 8b illustrate this: we can observe that the various models considered all achieve test error in the same range.

Original: Next, we compute generalization bounds for unrolled monDEQs (Section F) for a variety of α values, to get a sense of the quality of our generalization bound for the monDEQ.
Revised: Next, we compute generalization bounds for unrolled monDEQs (Section E) for a variety of α values, to get a sense of the quality of our generalization bound for the monDEQ.

Original: In Figure 8c, we can observe that the generalization bounds for these unrolled networks are larger (on the order of 10 6 ) as compared to the generalization bound for the monDEQ (on the order 10 5 ).
Revised: In Figure 8c, we can observe that the generalization bounds for these unrolled networks are larger (on the order of 10 6 ) as compared to the generalization bound for the monDEQ (on the order 10 5 ).

Original: This shows that our bound is tighter.
Revised: This shows that our bound is tighter.

Original: [Figure]
Revised: [Figure]

Original: I A DDITIONAL ADVERSARIAL ROBUSTNESS RESULTS
Revised: H A DDITIONAL ADVERSARIAL ROBUSTNESS RESULTS

Original: [Figure]
Revised: [Figure]

Original: We run the same experiments that we ran in Section 5.3 on the CIFAR-10 dataset.
Revised: We run the same experiments that we ran in Section 5.3 on the CIFAR-10 dataset.

Original: We train single convolution monDEQs (128 channels) with a range of m values, as well as 3 CNN models: CNN sm, CNN med and CNN lg whose architectures are described in Appendix G above.
Revised: We train single convolution monDEQs (128 channels) with a range of m values, as well as 3 CNN models: CNN sm, CNN med and CNN lg whose architectures are described in Appendix F above.

Original: As in the case of the MNIST experiments, we see that varying m gives a range of trade offs between clean accuracy and adversarial robustness.
Revised: As in the case of the MNIST experiments, we see that varying m gives a range of trade offs between clean accuracy and adversarial robustness.

Original: In terms of certiﬁed robustness, we see in Figure 9a that all of the monDEQ s besides m = 20 have both better clean and better robust accuracy than all the
Revised: In terms of certiﬁed robustness, we see in Figure 9a that all of the monDEQ s besides m = 20 have both better clean and better robust accuracy than all the

Original: CNNs (with the exception of the CNN with d = 2 , which has much lower clean accuracy).
Revised: CNNs (with the exception of the CNN with d = 2 , which has much lower clean accuracy).

Original: In Figure 9b, in terms of empirical robustness, we see that m parameterizes a similar trade-off between robust and clean accuracy.
Revised: In Figure 9b, in terms of empirical robustness, we see that m parameterizes a similar trade-off between robust and clean accuracy.

Original: In fact, it is possible to choose m (e.g. m = 1 here), such that the monDEQ outperforms all CNNs in terms of clean accuracy and at all attack sizes.
Revised: In fact, it is possible to choose m (e.g. m = 1 here), such that the monDEQ outperforms all CNNs in terms of clean accuracy and at all attack sizes.