Original: C ERTIFIED R OBUSTNESS FOR D EEP E QUILIBRIUM M ODELS VIA I NTERVAL B OUND P ROPAGATION
Revised: C ERTIFIED R OBUSTNESS FOR D EEP E QUILIBRIUM M ODELS VIA I NTERVAL B OUND P ROPAGATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep equilibrium layers (DEQs) have demonstrated promising performance and are competitive with standard explicit models on many benchmarks.
Revised: Deep equilibrium layers (DEQs) have demonstrated promising performance and are competitive with standard explicit models on many benchmarks.

Original: However, little is known about certifying robustness for these models.
Revised: However, little is known about certifying robustness for these models.

Original: Inspired by interval bound propagation (IBP), we propose the IBP-MonDEQ layer, a DEQ layer whose robustness can be veriﬁed by computing upper and lower interval bounds on the output.
Revised: Inspired by interval bound propagation (IBP), we propose the IBP-MonDEQ layer, a DEQ layer whose robustness can be veriﬁed by computing upper and lower interval bounds on the output.

Original: Our key insights are that these interval bounds can be obtained as the ﬁxed-point solution to an IBP-inspired equilibrium equation and furthermore, that this solution always exists and is unique when the layer obeys a certain parameterization.
Revised: Our key insights are that these interval bounds can be obtained as the ﬁxed-point solution to an IBP-inspired equilibrium equation and furthermore, that this solution always exists and is unique when the layer obeys a certain parameterization.

Original: This ﬁxed point can be interpreted as the result of applying IBP to an inﬁnitely deep, weight-tied neural network, which may be of independent interest, as IBP bounds are typically unstable for deeper networks.
Revised: This ﬁxed point can be interpreted as the result of applying IBP to an inﬁnitely deep, weight-tied neural network, which may be of independent interest, as IBP bounds are typically unstable for deeper networks.

Original: Our empirical comparison reveals that models with IBP-MonDEQ layers can achieve comparable or better (cid:96) 8 certiﬁed robustness than similarly-sized fully explicit networks.
Revised: Our empirical comparison reveals that models with IBP-MonDEQ layers can achieve comparable or better (cid:96) 8 certiﬁed robustness than similarly-sized fully explicit networks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A recent development in neural network design has been the introduction of implicit layers (Amos &
Revised: A recent development in neural network design has been the introduction of implicit layers (Amos &

Original: Kolter, 2017; Chen et al., 2018; Agrawal et al., 2019; Bai et al., 2019; 2020; El Ghaoui et al., 2021), where the output is deﬁned implicity as the solution to certain sets of conditions, rather than explicitly, via closed-form functions.
Revised: Kolter, 2017; Chen et al., 2018; Agrawal et al., 2019; Bai et al., 2019; 2020; El Ghaoui et al., 2021), where the output is deﬁned implicity as the solution to certain sets of conditions, rather than explicitly, via closed-form functions.

Original: These layers are promising alternatives to standard explicit deep learning layers and have demonstrated improved expressivity and inductive biases in a variety of settings, for example, processing time series (Rubanova et al., 2019), generative modeling (Grathwohl et al., 2018), solving logical reasoning tasks (Wang et al., 2019), solving two player games (Ling et al., 2018), and many others.
Revised: These layers are promising alternatives to standard explicit deep learning layers and have demonstrated improved expressivity and inductive biases in a variety of settings, for example, processing time series (Rubanova et al., 2019), generative modeling (Grathwohl et al., 2018), solving logical reasoning tasks (Wang et al., 2019), solving two player games (Ling et al., 2018), and many others.

Original: One particularly promising class of implicit layers is deep equilibrium layers (DEQs) (Bai et al., 2019), which deﬁne the output as the solution to an input-dependent ﬁxed point equation.
Revised: One particularly promising class of implicit layers is deep equilibrium layers (DEQs) (Bai et al., 2019), which deﬁne the output as the solution to an input-dependent ﬁxed point equation.

Original: DEQ-based models have matched or outperformed traditional explicit models even in commonly benchmarked settings (Bai et al., 2019; 2020).
Revised: DEQ-based models have matched or outperformed traditional explicit models even in commonly benchmarked settings (Bai et al., 2019; 2020).

Original: Though recent empirical successes of DEQs have been promising, their implicit nature and inherent mathematical complexity also give rise to basic concerns.
Revised: Though recent empirical successes of DEQs have been promising, their implicit nature and inherent mathematical complexity also give rise to basic concerns.

Original: In order for DEQs to realize their promise, these concerns should ideally be mitigated or resolved.
Revised: In order for DEQs to realize their promise, these concerns should ideally be mitigated or resolved.

Original: For example, one major issue with DEQs is well-posedness – a solution to the ﬁxed point equation deﬁning the layer might not exist.
Revised: For example, one major issue with DEQs is well-posedness – a solution to the ﬁxed point equation deﬁning the layer might not exist.

Original: On the other hand, explicit layers always have well-deﬁned outputs.
Revised: On the other hand, explicit layers always have well-deﬁned outputs.

Original: A recent line of work has focused on addressing this important concern (Winston & Kolter, 2020; Revay et al., 2020; Xie et al., 2021).
Revised: A recent line of work has focused on addressing this important concern (Winston & Kolter, 2020; Revay et al., 2020; Xie et al., 2021).

Original: This paper tackles a less-studied, but also important, question for DEQs: certiﬁed adversarial robustness.
Revised: This paper tackles a less-studied, but also important, question for DEQs: certiﬁed adversarial robustness.

Original: Because robustness is a basic concern for safe deployment of deep models (Szegedy et al., 2013; Goodfellow et al., 2014), for explicit models there is a large literature dedicated to certifying robustness, or guaranteeing correctness of the predictions even when the input is subject to imperceptible adversarial perturbations (see e.g. (Raghunathan et al., 2018a; Wong & Kolter, 2018; Gowal et al., 2018; Dvijotham et al.
Revised: Because robustness is a basic concern for safe deployment of deep models (Szegedy et al., 2013; Goodfellow et al., 2014), for explicit models there is a large literature dedicated to certifying robustness, or guaranteeing correctness of the predictions even when the input is subject to imperceptible adversarial perturbations (see e.g. (Raghunathan et al., 2018a; Wong & Kolter, 2018; Gowal et al., 2018; Dvijotham et al.

Original: ; Xiao et al., 2018; Cohen et al., 2019)).
Revised: ; Xiao et al., 2018; Cohen et al., 2019)).

Original: Many certiﬁed robustness methods require opening up the black box of the model and therefore only work for explicit models.
Revised: Many certiﬁed robustness methods require opening up the black box of the model and therefore only work for explicit models.

Original: It is unclear how to certify robustness of DEQs, which are only deﬁned implicitly.
Revised: It is unclear how to certify robustness of DEQs, which are only deﬁned implicitly.

Original: The certiﬁed robustness method motivating this work is interval bound propagation (IBP) (Mirman et al., 2018; Gowal et al., 2018), a simple and cheap way to certify robustness to (cid:96) 8 perturbations.
Revised: The certiﬁed robustness method motivating this work is interval bound propagation (IBP) (Mirman et al., 2018; Gowal et al., 2018), a simple and cheap way to certify robustness to (cid:96) 8 perturbations.

Original: IBP computes layerwise upper and lower bounds for each coordinate of the adversarially perturbed hidden layers.
Revised: IBP computes layerwise upper and lower bounds for each coordinate of the adversarially perturbed hidden layers.

Original: These bounds follow basic rules of interval arithmetic and are simple to obtain in closed form as explicit functions of a layer’s weights.
Revised: These bounds follow basic rules of interval arithmetic and are simple to obtain in closed form as explicit functions of a layer’s weights.

Original: However, it is unclear how to apply this idea when layers are only deﬁned implicitly.
Revised: However, it is unclear how to apply this idea when layers are only deﬁned implicitly.

Original: In this paper, we propose the IBP-MonDEQ, a deep equilibrium layer which is certiﬁably robust to (cid:96) 8 perturbations.
Revised: In this paper, we propose the IBP-MonDEQ, a deep equilibrium layer which is certiﬁably robust to (cid:96) 8 perturbations.

Original: Motivated by principles from IBP, we deﬁne the IBP-MonDEQ output as the solution of an augmented ﬁxed-point equation involving 3 quantities: the unperturbed output of the layer and upper and lower interval bounds on this output.
Revised: Motivated by principles from IBP, we deﬁne the IBP-MonDEQ output as the solution of an augmented ﬁxed-point equation involving 3 quantities: the unperturbed output of the layer and upper and lower interval bounds on this output.

Original: As with IBP for explicit models, interval bounds on the IBP-MonDEQ output are computed during the forward pass of the network and can be composed with other layers to certify robustness of the entire model.
Revised: As with IBP for explicit models, interval bounds on the IBP-MonDEQ output are computed during the forward pass of the network and can be composed with other layers to certify robustness of the entire model.

Original: More concretely, we build upon monotone operator deep equilibrium (MonDEQ) layers proposed by Winston & Kolter (2020), which take the preceding layer v p x q as input and outputs the solution z ‹ , which is guaranteed existence and uniqueness, to the following ﬁxed-point equation:
Revised: More concretely, we build upon monotone operator deep equilibrium (MonDEQ) layers proposed by Winston & Kolter (2020), which take the preceding layer v p x q as input and outputs the solution z ‹ , which is guaranteed existence and uniqueness, to the following ﬁxed-point equation:

Original: [Equation]
Revised: [Equation]

Original: We propose an IBP-inspired ﬁxed-point equation „
Revised: We propose an IBP-inspired ﬁxed-point equation „

Original: [Equation]
Revised: [Equation]

Original: which maps upper and lower interval bounds s v, v on v to s z ‹ and z ‹ , which provide upper and lower interval bounds on z ‹
Revised: which maps upper and lower interval bounds s v, v on v to s z ‹ and z ‹ , which provide upper and lower interval bounds on z ‹

Original: Figure 1 depicts this process.
Revised: Figure 1 depicts this process.

Original: The augmented ﬁxed-point equation is derived by unrolling the computation of z ‹ into an inﬁnitely deep, explicit network and applying IBP to the forward pass of this network.
Revised: The augmented ﬁxed-point equation is derived by unrolling the computation of z ‹ into an inﬁnitely deep, explicit network and applying IBP to the forward pass of this network.

Original: One immediate challenge is that it is not clear that a ﬁxed point solution to (1.2) should always exist, especially given its interpretation as the result of applying IBP to an inﬁnitely deep network.
Revised: One immediate challenge is that it is not clear that a ﬁxed point solution to (1.2) should always exist, especially given its interpretation as the result of applying IBP to an inﬁnitely deep network.

Original: Indeed, a major drawback of IBP is that its performance degrades with deeper networks, as observed by Shi et al.
Revised: Indeed, a major drawback of IBP is that its performance degrades with deeper networks, as observed by Shi et al.

Original: and also shown in Figure 2 (left).
Revised: and also shown in Figure 2 (left).

Original: One potential explanation for this failure is that IBP bounds tend to be unstable with depth and can diverge for deeper models (Figure 2, right).
Revised: One potential explanation for this failure is that IBP bounds tend to be unstable with depth and can diverge for deeper models (Figure 2, right).

Original: On the other hand, we show that a unique ﬁxed-point solution to (1.2) is guaranteed to exist when W admits a simple unconstrained parameterization which is easy to enforce throughout training.
Revised: On the other hand, we show that a unique ﬁxed-point solution to (1.2) is guaranteed to exist when W admits a simple unconstrained parameterization which is easy to enforce throughout training.

Original: Thus, our results pinpoint a class of inﬁnitely deep networks for which IBP bounds are provably stable, which may be of independent interest.
Revised: Thus, our results pinpoint a class of inﬁnitely deep networks for which IBP bounds are provably stable, which may be of independent interest.

Original: We experimentally compare the proposed IBP-MonDEQ layer against IBP for standard explicit models.
Revised: We experimentally compare the proposed IBP-MonDEQ layer against IBP for standard explicit models.

Original: We consider common certiﬁed robustness benchmark settings and evaluate architectures of various sizes.
Revised: We consider common certiﬁed robustness benchmark settings and evaluate architectures of various sizes.

Original: Our results show that models with IBP-MonDEQ layers can achieve comparable or better (cid:96) 8 certiﬁed robustness relative to fully explicit models with similar parameter counts.
Revised: Our results show that models with IBP-MonDEQ layers can achieve comparable or better (cid:96) 8 certiﬁed robustness relative to fully explicit models with similar parameter counts.

Original: In summary, our contributions are as follows: 1) We study the certiﬁed robustness of DEQs, proposing the IBP-MonDEQ, a class of DEQs with a guaranteed unique ﬁxed point with provable interval bounds on the ﬁxed point value.
Revised: In summary, our contributions are as follows: 1) We study the certiﬁed robustness of DEQs, proposing the IBP-MonDEQ, a class of DEQs with a guaranteed unique ﬁxed point with provable interval bounds on the ﬁxed point value.

Original: The proposed IBP-MonDEQs form an expressive class of inﬁnitely-deep models for which IBP is provably stable.
Revised: The proposed IBP-MonDEQs form an expressive class of inﬁnitely-deep models for which IBP is provably stable, which may be of independent interest. 3)

Original: Our experiments demonstrate that IBPMonDEQ layers are competitive with standard explicit layers for (cid:96) 8 -certiﬁed robustness.
Revised: Our experiments demonstrate that IBP-MonDEQ layers are competitive with standard explicit layers for (cid:96) 8 -certiﬁed robustness.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: (cid:96)  certiﬁed robustness.
Revised: (cid:96) 8 certiﬁed robustness.

Original: Consider a K -way classiﬁcation task with neural network classiﬁer F : R d Ñ R K .
Revised: Consider a K -way classiﬁcation task with neural network classiﬁer F : R d Ñ R K .

Original: For a given input x with true label y , the classiﬁer F is adversarially robust to (cid:96)perturbations with radius (cid:15) if
Revised: For a given input x with true label y , the classiﬁer F is adversarially robust to (cid:96)perturbations with radius (cid:15) if

Original: [Equation]
Revised: [Equation]

Original: Safety-critical applications require certifying the robustness of F , i.e., verifying whether 2.1 holds.
Revised: Safety-critical applications require certifying the robustness of F , i.e., verifying whether 2.1 holds.

Original: Directly optimizing over δ to verify (2.1) is challenging because the objective is non-convex (Madry et al., 2017).
Revised: Directly optimizing over δ to verify (2.1) is challenging because the objective is non-convex (Madry et al., 2017).

Original: Thus, recent work on certiﬁed robustness has focused on verifying (2.1) via computationally tractable relaxations (Raghunathan et al., 2018a; Wong & Kolter, 2018; Dvijotham et al.;
Revised: Thus, recent work on certiﬁed robustness has focused on verifying (2.1) via computationally tractable relaxations (Raghunathan et al., 2018a; Wong & Kolter, 2018; Dvijotham et al.;

Original: Raghunathan et al., 2018b; Weng et al., 2018; Gowal et al., 2018;
Revised: Raghunathan et al., 2018b; Weng et al., 2018; Gowal et al., 2018;

Original: Salman et al., 2019b).
Revised: Salman et al., 2019b).

Original: Interval bound propagation.
Revised: Interval bound propagation.

Original: IBP is a computationally efﬁcient method for certifying (cid:96) 8 robustness of neural networks (Mirman et al., 2018; Gowal et al., 2018).
Revised: IBP is a computationally efﬁcient method for certifying (cid:96) 8 robustness of neural networks (Mirman et al., 2018; Gowal et al., 2018).

Original: It proposes to verify (2.1) via a (potentially loose) upper bound on F p x ` δ q y ´ F p x ` δ q y 1 which is obtained by computing propagating upper and lower bounds on each layer through the forward pass of the network.
Revised: It proposes to verify (2.1) via a (potentially loose) upper bound on F p x ` δ q y ´ F p x ` δ q y 1 which is obtained by computing propagating upper and lower bounds on each layer through the forward pass of the network.

Original: More precisely, let z p x q compute some hidden layer of the network on input x .
Revised: More precisely, let z p x q compute some hidden layer of the network on input x .

Original: We say that s z p x, (cid:15) q , z p x, (cid:15) q are interval bounds on z at x for perturbation (cid:15) if the following holds for all coordinates i :
Revised: We say that s z p x, (cid:15) q , z p x, (cid:15) q are interval bounds on z at x for perturbation (cid:15) if the following holds for all coordinates i :

Original: [Equation]
Revised: [Equation]

Original: We omit dependencies on x and (cid:15) when clear from context.
Revised: We omit dependencies on x and (cid:15) when clear from context.

Original: Letting k denote the layer index, the bounds s z k and z k are obtained inductively via simple interval arithmetic.
Revised: Letting k denote the layer index, the bounds s z k and z k are obtained inductively via simple interval arithmetic.

Original: For an afﬁne layer z k “
Revised: For an afﬁne layer z k “

Original: Wz k ´ 1 ` b and pre-computed bounds z k ´ 1 , s z k ´ 1 , IBP computes s z k , z k as follows:
Revised: Wz k ´ 1 ` b and pre-computed bounds z k ´ 1 , s z k ´ 1 , IBP computes s z k , z k as follows:

Original: [Equation]
Revised: [Equation]

Original: Here p W q ` ﬁ max t W, 0 u and p W q ´ ﬁ min t W, 0 u denote the matrix W with negative or positive values truncated to 0.
Revised: Here p W q ` ﬁ max t W, 0 u and p W q ´ ﬁ min t W, 0 u denote the matrix W with negative or positive values truncated to 0.

Original: For simplicity we focus on ReLU networks, with σ denoting the ReLU activation.
Revised: For simplicity we focus on ReLU networks, with σ denoting the ReLU activation.

Original: For layers z k “ σ p z k ´ 1 q which apply σ coordinate-wise, IBP computes s z k “ σ p s z k ´ 1 q and z k “ σ p z k ´ 1 q .
Revised: For layers z k “ σ p z k ´ 1 q which apply σ coordinate-wise, IBP computes s z k “ σ p s z k ´ 1 q and z k “ σ p z k ´ 1 q .

Original: Initial bounds are obtained via p s z 0 p x, (cid:15) q , z 0 p x, (cid:15) qq
Revised: Initial bounds are obtained via p s z 0 p x, (cid:15) q , z 0 p x, (cid:15) qq

Original: “ p x ` (cid:15) 1 , x ´ (cid:15) 1 q , wheredenotes the all 1’s vector.
Revised: “ p x ` (cid:15) 1 , x ´ (cid:15) 1 q , wheredenotes the all 1’s vector.

Original: The interval bounds are propagated through all layers of the network by following the simple rules above.
Revised: The interval bounds are propagated through all layers of the network by following the simple rules above.

Original: To certify (2.1) for the whole network, one straightforward method is to conﬁrm that the margins of the interval bounds on the logits are positive: p F p x, (cid:15) qq y ´ p s F p x, (cid:15) qq y 1 ą 0 @ y 1 ‰ y .
Revised: To certify (2.1) for the whole network, one straightforward method is to conﬁrm that the margins of the interval bounds on the logits are positive: p F p x, (cid:15) qq y ´ p s F p x, (cid:15) qq y 1 ą 0 @ y 1 ‰ y .

Original: One important note about IBP is that the bounds should be optimized during training in order for the method to provide nontrivial robustness guarantees.
Revised: One important note about IBP is that the bounds should be optimized during training in order for the method to provide nontrivial robustness guarantees.

Original: Monotone operator equilibrium networks.
Revised: Monotone operator equilibrium networks.

Original: Proposed by Winston & Kolter (2020), MonDEQs are a class of DEQs inspired by monotone operator theory (Ryu & Boyd, 2016) which have guaranteed unique ﬁxed point solutions to the following equilibrium equation:
Revised: Proposed by Winston & Kolter (2020), MonDEQs are a class of DEQs inspired by monotone operator theory (Ryu & Boyd, 2016) which have guaranteed unique ﬁxed point solutions to the following equilibrium equation:

Original: [Equation]
Revised: [Equation]

Original: Let I h
Revised: Let I h

Original: ˆ h denote the identity matrix on h dimensions, with the subscript omitted when clear.
Revised: ˆ h denote the identity matrix on h dimensions, with the subscript omitted when clear.

Original: A unique ﬁxed-point solution is guaranteed for (2.4) for the following class of W :
Revised: A unique ﬁxed-point solution is guaranteed for (2.4) for the following class of W :

Original: Proposition 2.1 ((Winston & Kolter, 2020)) .
Revised: Proposition 2.1 ((Winston & Kolter, 2020)) .

Original: Suppose W P R h
Revised: Suppose W P R h

Original: ˆ h satisﬁes that I ´ W is positive deﬁnite (PD), i.e. I ´ W ą 0 .
Revised: ˆ h satisﬁes that I ´ W is positive deﬁnite (PD), i.e. I ´ W ą 0 .

Original: Then @ v p x q P R h , a solution z ‹ to (2.4) exists and is unique.
Revised: Then @ v p x q P R h , a solution z ‹ to (2.4) exists and is unique.

Original: Here 0 denotes the all 0’s matrix and A ą 0 indicates that u J Au ą 0 for all nonzero u .
Revised: Here 0 denotes the all 0’s matrix and A ą 0 indicates that u J Au ą 0 for all nonzero u (note that A does not need to be symmetric).

Original: Winston & Kolter (2020) guarantee that I ´ W is PD using the following unconstrained parameterization, which is enforced throughout training: W “ p 

Revised: Winston & Kolter (2020) guarantee that I ´ W is PD using the following unconstrained parameterization, which is enforced throughout training: W “ p

Original: ´ m q
Revised: ´ m q

Original: I ´ AA J ` B ´ B J , for positive hyperparameter m .
Revised: I ´ AA J ` B ´ B J , for positive hyperparameter m .

Original: Section 3.1 builds on these results to derive certiﬁed upper and lower bounds on z ‹ .
Revised: Section 3.1 builds on these results to derive certiﬁed upper and lower bounds on z ‹ .

Original:  C ERTIFYING ROBUSTNESS OF M ON DEQ S USING IBP
Revised:  C ERTIFYING ROBUSTNESS OF M ON DEQ S USING IBP

Original: In this section, we describe our core methodology for developing certiﬁably robust MonDEQ layers.
Revised: In this section, we describe our core methodology for developing certiﬁably robust MonDEQ layers.

Original: We will ﬁrst demonstrate how to obtain interval bounds for MonDEQ layers by computing the solution to a certain IBP-inspired ﬁxed point equation (3.3).
Revised: We will ﬁrst demonstrate how to obtain interval bounds for MonDEQ layers by computing the solution to a certain IBP-inspired ﬁxed point equation (3.3).

Original: In Section 3.1, we characterize a new parameterization for W for which a unique ﬁxed point exists.
Revised: In Section 3.1, we characterize a new parameterization for W for which a unique ﬁxed point exists.

Original: In Section 3.3, we provide theoretical justiﬁcation that the resulting IBP-MonDEQ layers remain expressive.
Revised: In Section 3.3, we provide theoretical justiﬁcation that the resulting IBP-MonDEQ layers remain expressive.

Original: Our aim is to derive upper and lower interval bounds s z ‹ and z ‹ for the ﬁxed point solution to (2.4).
Revised: Our aim is to derive upper and lower interval bounds s z ‹ and z ‹ for the ﬁxed point solution to (2.4).

Original: A common interpretation of DEQs is that they compute an inﬁnitely deep, unrolled explicit network (Bai et al., 2019; Winston & Kolter, 2020):
Revised: A common interpretation of DEQs is that they compute an inﬁnitely deep, unrolled explicit network (Bai et al., 2019; Winston & Kolter, 2020):

Original: [Equation]
Revised: [Equation]

Original: with lim k Ñ8 z k “ z ‹ .
Revised: with lim k Ñ8 z k “ z ‹ .

Original: This equivalence is informal and mainly serves to motivate our derivation of the IBP-MonDEQ.
Revised: This equivalence is informal and mainly serves to motivate our derivation of the IBP-MonDEQ.

Original: Given interval bounds s v and v on v satisfying v ď v ď s v , where the inequalities hold elementwise, we can also follow IBP and (2.3) to iteratively obtain interval bounds on z k :
Revised: Given interval bounds s v and v on v satisfying v ď v ď s v , where the inequalities hold elementwise, we can also follow IBP and (2.3) to iteratively obtain interval bounds on z k :

Original: [Equation]
Revised: [Equation]

Original: Just as we took lim k Ñ8 z k , we consider lim k Ñ8 s z k , z k , motivating another ﬁxed-point problem:
Revised: Just as we took lim k Ñ8 z k , we consider lim k Ñ8 s z k , z k , motivating another ﬁxed-point problem:

Original: [Equation]
Revised: [Equation]

Original: As shown in Figure 1, this IBP-inspired ﬁxed point equation essentially maps the region t v 1 : v ď v 1 ď
Revised: As shown in Figure 1, this IBP-inspired ﬁxed point equation essentially maps the region t v 1 : v ď v 1 ď

Original: s v u to the region t z ‹1 : z ‹ ď z ‹1 ď s z ‹ u , where inequalities are coordinate-wise.
Revised: s v u to the region t z ‹1 : z ‹ ď z ‹1 ď s z ‹ u , where inequalities are coordinate-wise.

Original: If they exist, the ﬁxed points s z ‹ and z ‹ will indeed lead to valid interval bounds (as deﬁned in (2.2)) on z ‹ .
Revised: If they exist, the ﬁxed points s z ‹ and z ‹ will indeed lead to valid interval bounds (as deﬁned in (2.2)) on z ‹ .

Original: Proposition 3.2 states this observation formally.
Revised: Proposition 3.2 states this observation formally.

Original: 3. F IXED POINTS OF THE IBP- INSPIRED EQUILIBRIUM EQUATION
Revised: 3. F IXED POINTS OF THE IBP- INSPIRED EQUILIBRIUM EQUATION

Original: Following the derivation above, the IBP ﬁxed point equation (3.3) can be interpreted as the result of applying IBP to an inﬁnitely deep, weight-tied neural network.
Revised: Following the derivation above, the IBP ﬁxed point equation (3.3) can be interpreted as the result of applying IBP to an inﬁnitely deep, weight-tied neural network.

Original: However, IBP is notoriously unstable for deeper networks, so it is unclear whether a ﬁxed point solution to (3.3) always exists.
Revised: However, IBP is notoriously unstable for deeper networks, so it is unclear whether a ﬁxed point solution to (3.3) always exists.

Original: This section characterizes a class of W for which (3.3) does have a ﬁxed point (Theorem 3.1).
Revised: This section characterizes a class of W for which (3.3) does have a ﬁxed point (Theorem 3.1).

Original: Figure 2 (left) demonstrates that the performance of IBP degrades for deep networks.
Revised: Figure 2 (left) demonstrates that the performance of IBP degrades for deep networks.

Original: We add linear layers to the architecture proposed by Gowal et al.
Revised: We add linear layers to the architecture proposed by Gowal et al.

Original: (2018) for IBP, which is a wide but shallow network with 6 linear layers, and ﬁnd that the ﬁnal error certiﬁed by IBP increases with the depth of the model.
Revised: (2018) for IBP, which is a wide but shallow network with 6 linear layers, and ﬁnd that the ﬁnal error certiﬁed by IBP increases with the depth of the model.

Original: This illustrates the challenge of getting IBP to work for deep models.
Revised: This illustrates the challenge of getting IBP to work for deep models.

Original: The challenges associated with depth also apply to the class of MonDEQs proposed by (Winston & Kolter, 2020).
Revised: The challenges associated with depth also apply to the class of MonDEQs proposed by (Winston & Kolter, 2020).

Original: We will deﬁne x W ﬁ „ p W q ` p W q ´ p W q ´ p W q ` 
Revised: We will deﬁne x W ﬁ „ p W q ` p W q ´ p W q ´ p W q ` 

Original: P R 2 h
Revised: P R 2 h

Original: ˆ 2 h to be the linear transformation for IBP.
Revised: ˆ 2 h to be the linear transformation for IBP.

Original: Recall that Proposition 2.1 demonstrates existence and uniqueness of a ﬁxed point solution to (2.4) when I h
Revised: Recall that Proposition 2.1 demonstrates existence and uniqueness of a ﬁxed point solution to (2.4) when I h

Original: ˆ h ´ W is PD, but this does not guarantee the positive-deﬁniteness of I
Revised: ˆ h ´ W is PD, but this does not guarantee the positive-deﬁniteness of I

Original: 2 h ˆ 2 h
Revised: 2 h ˆ 2 h

Original: ´ x W , which would be sufﬁcient for a ﬁxed point solution to (3.3).
Revised: ´ x W , which would be sufﬁcient for a ﬁxed point solution to (3.3).

Original: Indeed, Figure 2 (right) shows that the ﬁxed-point solver fails to solve (3.3) for the standard MonDEQ, as the interval bounds diverge.
Revised: Indeed, Figure 2 (right) shows that the ﬁxed-point solver fails to solve (3.3) for the standard MonDEQ, as the interval bounds diverge.

Original: Intuitively, this failure occurs because the spectral radius of x W is always larger than that of W (see
Revised: Intuitively, this failure occurs because the spectral radius of x W is always larger than that of W (see

Original: Lemma 3.4), so repeated application of (3.2) diverges.
Revised: Lemma 3.4), so repeated application of (3.2) diverges.

Original: Thus, further restrictions on W are required.
Revised: Thus, further restrictions on W are required.

Original: The following theorem exactly characterizes the class of W for which I 2 h ˆ 2 h
Revised: The following theorem exactly characterizes the class of W for which I 2 h ˆ 2 h

Original: ´ x W is PD.
Revised: ´ x W is PD.

Original: For these
Revised: For these

Original: W , existence and uniqueness of ﬁxed points to (2.4) and (3.3) is guaranteed to hold.
Revised: W , existence and uniqueness of ﬁxed points to (2.4) and (3.3) is guaranteed to hold.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: In the setting above, I 2 h ˆ 2 h
Revised: In the setting above, I 2 h ˆ 2 h

Original: ´ x W ą 

Revised: ´ x W ą

Original: if and only if W is parameterized as follows:
Revised: if and only if W is parameterized as follows:

Original: [Equation]
Revised: [Equation]

Original: where D, M P R h
Revised: where D, M P R h

Original: ˆ h and D is a diagonal matrix satisfying D i,i ą p| M | 1 q i `p| M | J 1 q i. As an immediate consequence, for W parameterized by (3.4) , unique solutions z ‹ , s z ‹ , z ‹ to the standard (2.4) and IBP (3.6) equilibrium equations exist for all v, s v, v P R h .
Revised: ˆ h and D is a diagonal matrix satisfying D i,i ą p| M | 1 q i `p| M | J 1 q i, where | M | P R h ˆ h takes the entry-wise absolute value of M . Thus, for W parameterized by (3.4) , unique solutions z ‹ , s z ‹ , z ‹ to the standard (2.4) and IBP (3.6) equilibrium equations exist @ v, s v, v P R h .

Original: Theorem 3.1 is our main theoretical result, and proposes a simple class of W for which the IBP equilibrium equation has a guaranteed unique ﬁxed point.
Revised: Theorem 3.1 is our main theoretical result, and proposes a simple class of W for which the IBP equilibrium equation has a guaranteed unique ﬁxed point.

Original: Due to the simplicity of the characterization,it is easy to enforce (3.4) and directly optimize M during training, as described in Section 4.
Revised: In this characterization, M is an unconstrained matrix (and is also treated as an unconstrained parameter in our implementation, as described in Section 4).

Original: In
Revised: The scaling matrix D normalizes M .

Original:  Section 3.3, we theoretically analyze the expressivity of the induced class of DEQ layers.
Revised: In Section 3.3, we theoretically analyze the expressivity of the induced class of DEQ layers.

Original: The proof of Theorem 3.1 is discussed in Section 3.2.
Revised: The proof of Theorem 3.1 is discussed in Section 3.2.

Original: Next, it remains to check that s z ‹ and z ‹ do provide valid interval bounds on z ‹ , as the derivation in
Revised: Next, it remains to check that s z ‹ and z ‹ do provide valid interval bounds on z ‹ , as the derivation in

Original: Section 3 was only heuristic.
Revised: Section 3 was only heuristic.

Original: The following proposition makes this derivation rigorous.
Revised: The following proposition makes this derivation rigorous.

Original: Proposition 3.2.
Revised: Proposition 3.2.

Original: In the setting above, suppose x W satisﬁes I
Revised: In the setting above, suppose x W satisﬁes I

Original: h
Revised: h

Original: ˆ 2 h
Revised: ˆ 2 h

Original: ´ x W ą 0 , so that unique ﬁxedpoint solutions s z ‹ , z ‹ to (3.3) always exist.
Revised: ´ x W ą 0 , so that unique ﬁxedpoint solutions s z ‹ , z ‹ to (3.3) always exist.

Original: Deﬁne the ﬁxed-point solution z ‹ p v q as a function of v as follows: z ‹
Revised: Deﬁne the ﬁxed-point solution z ‹ p v q as a function of v as follows: z ‹

Original: [Equation]
Revised: [Equation]

Original: p Wz ‹ p v q ` v q .
Revised: q [Equation] v q .

Original: Then for any v ď s v and all i , it must hold that
Revised: Then for any v ď s v and all i , it must hold that

Original: [Equation]
Revised: [Equation]

Original: We present the proof in Section A. Combining Theorem 3.1 and Proposition 3.2, we ﬁnally obtain the IBP-MonDEQ layer, which simultaneously solves two equilbrium equations to output both z ‹ and valid upper and lower interval bounds s z ‹ , z ‹ on z ‹ :
Revised: We present the proof in Section A. Combining Theorem 3.1 and Proposition 3.2, we ﬁnally obtain the IBP-MonDEQ layer, which simultaneously solves two equilbrium equations to output both z ‹ and valid upper and lower interval bounds s z ‹ , z ‹ on z ‹ :

Original: [Equation]
Revised: [Equation]

Original: To include an IBP-MonDEQ layer in a deeper model, we set v p x q to be the output of the preceding layer, and choose s v p x, (cid:15) q , v p x, (cid:15) q to be upper and lower interval bounds on v p x q .
Revised: To include an IBP-MonDEQ layer in a deeper model, we set v p x q to be the output of the preceding layer, and choose s v p x, (cid:15) q , v p x, (cid:15) q to be upper and lower interval bounds on v p x q .

Original: The interval bounds s z ‹ and z ‹ can be propagated through the rest of the network following the standard IBP methods.
Revised: The interval bounds s z ‹ and z ‹ can be propagated through the rest of the network following the standard IBP methods.

Original: LBEN parameterization.
Revised: LBEN parameterization.

Original: We can relax the restrictions imposed by the parameterization (3.4) by leveraging the Lipschitz-bounded equilibrium network (LBEN) parameterization proposed by Revay et al.
Revised: We can relax the restrictions imposed by the parameterization (3.4) by leveraging the Lipschitz-bounded equilibrium network (LBEN) parameterization proposed by Revay et al.

Original: Revay et al.
Revised: Revay et al.

Original: (2020) show that a guaranteed unique ﬁxed point of (2.4) still exists if the
Revised: (2020) show that a guaranteed unique ﬁxed point of (2.4) still exists if the

Original: PD condition on I
Revised: PD condition on I

Original: ´ W is relaxed to require Λ ´ Λ W ą 0 for Λ P D
Revised: ´ W is relaxed to require Λ ´ Λ W ą 0 for Λ P D

Original: h
Revised: h

Original: ˆ h ` , the set of positive diagonal matrices.
Revised: ˆ h ` , the set of positive diagonal matrices.

Original: The following result exactly characterizes W such that D Λ P D
Revised: The following result exactly characterizes W such that D Λ P D

Original: h
Revised: h

Original: ˆ h ` : Λ ´ Λ | W | ą 0 , which is also sufﬁcient for guaranteeing a unique ﬁxed point solution to (3.3).
Revised: ˆ h ` : Λ ´ Λ | W | ą 0 , which is also sufﬁcient for guaranteeing a unique ﬁxed point solution to (3.3).

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: For weight matrix W , D Λ P D
Revised: For weight matrix W , D Λ P D

Original: h
Revised: h

Original: ˆ h ` such that Λ ´ Λ | W | ą 

Revised: ˆ h ` such that Λ ´ Λ | W | ą

Original: if and only if W can be parameterized as follows:
Revised: if and only if W can be parameterized as follows:

Original: [Equation]
Revised: [Equation]

Original: for some Γ , D P D
Revised: for some Γ , D P D

Original: h
Revised: h

Original: ˆ h ` with D satisfying D
Revised: ˆ h ` with D satisfying D

Original: i,i ą p| M | 1 `| M | J 1 q.
Revised: i,i ą p| M | 1 `| M | J 1 q.

Original: For W parameterized by (3.7) , we have the following immediate consequences: 1) unique solutions z ‹ , s z ‹ , z ‹ to the equilbrium equations (3.6) exist for all v, s v, v P R h , and 2) s z ‹ , z ‹ are upper and lower interval bounds on z ‹ in the sense of Proposition 3.2.
Revised: For W parameterized by (3.7) , we have the following immediate consequences: 1) unique solutions z ‹ , s z ‹ , z ‹ to the equilbrium equations (3.6) exist for all v, s v, v P R h , and 2) s z ‹ , z ‹ are upper and lower interval bounds on z ‹ in the sense of Proposition 3.2.

Original: The proof is provided in Section A. Note that Theorem 3.3 improves the expressivity of the IBPMonDEQ layer because the class of W proposed in Theorem 3.3 contains that of Theorem 3.1.
Revised: The proof is provided in Section A. Note that Theorem 3.3 improves the expressivity of the IBPMonDEQ layer because the class of W proposed in Theorem 3.3 contains that of Theorem 3.1.

Original: In Section 3.3, we show that, perhaps surprisingly, this more ﬂexible parameterization allows IBPMonDEQ layers to express all explicit networks.
Revised: In Section 3.3, we show that, perhaps surprisingly, this more ﬂexible parameterization allows IBPMonDEQ layers to express all explicit networks.

Original: Section 4 demonstrates that this parameterization can also lead to empirical improvements.
Revised: Section 4 demonstrates that this parameterization can also lead to empirical improvements.

Original: 3. P ROOF OF T HEOREM 3. 
Revised: 3. P ROOF OF T HEOREM 3.

Original: We sketch the proof of Theorem 3.1.
Revised: We sketch the proof of Theorem 3.1.

Original: To prove I 2 h ˆ 2 h
Revised: To prove I 2 h ˆ 2 h

Original: ´ x W ą 0 , we require a simple way to reason about the spectrum of x W .
Revised: ´ x W ą 0 , we require a simple way to reason about the spectrum of x W .

Original: The following lemma formalizes a connection between eigenvalues of x
Revised: The following lemma formalizes a connection between eigenvalues of x

Original: W ` x W Jand | W |`| W | J, where | W | P R h
Revised: W ` x W Jand | W |`| W | J.

Original: ˆ h takes the entry-wise absolute value of W .

Original: Lemma 3.4.
Revised: Lemma 3.4.

Original: All eigenvalues of x W ` x W Jare either eigenvalues of W ` W Jor | W |`| W | J. As a direct consequence, the following equivalence holds:[Equation]
Revised: All eigenvalues of x W ` x W Jare either eigenvalues of W ` W Jor | W |`| W | J. As a direct consequence, the following equivalence holds:

Original: [Equation]
Revised: [Equation]

Original: Note that the condition that I
Revised: Note that the condition that I

Original: h
Revised: h

Original: ˆ h
Revised: ˆ h

Original: ´ | W | is PD is a stricter requirement than requiring positivedeﬁniteness of I
Revised: ´ | W | is PD is a stricter requirement than requiring positivedeﬁniteness of I

Original: h
Revised: h

Original: ˆ h ´ W , as it always holds that λ max p | W |`| W | J
Revised: ˆ h ´ W , as it always holds that λ max p | W |`| W | J

Original: [Equation]
Revised: [Equation]

Original: max p W ` W Jq , where λ max denotes the largest eigenvalue.
Revised: q , where λ max denotes the largest eigenvalue.

Original: Thus, in order for both (2.4) and (3.3) to have unique ﬁxed points, it sufﬁces to have I
Revised: Thus, in order for both (2.4) and (3.3) to have unique ﬁxed points, it sufﬁces to have I

Original: h
Revised: h

Original: ˆ h ´ | W | ą 0 .
Revised: ˆ h ´ | W | ą 0 .

Original: The proof of Lemma 3.4 is completed in Section A and relies on the following basic fact about structured block matrices.
Revised: The proof of Lemma 3.4 is completed in Section A and relies on the following basic fact about structured block matrices.

Original: Claim 3.5.
Revised: Claim 3.5.

Original: Let M P R 2 h
Revised: Let M P R 2 h

Original: ˆ 2 h be any matrix of the form M “ „
Revised: ˆ 2 h be any matrix of the form M “ „

Original: A
Revised: A

Original: B B
Revised: B B

Original: A  , where A, B P R h ˆ h .
Revised: A  , where A, B P R h ˆ h .

Original: Then all eigenvalues of M are either eigenvalues of A ` B or A ´ B .
Revised: Then all eigenvalues of M are either eigenvalues of A ` B or A ´ B .

Original: Following Lemma 3.4, we now complete the proof of Theorem 3.1 by analyzing I
Revised: Following Lemma 3.4, we now complete the proof of Theorem 3.1 by analyzing I

Original: h
Revised: h

Original: ˆ h ´
Revised: ˆ h ´

Original: | W | .
Revised: | W | .

Original: Proof of Theorem 3.1.
Revised: Proof of Theorem 3.1.

Original: By Lemma 3.4, it is equivalent to characterize when I
Revised: By Lemma 3.4, it is equivalent to characterize when I

Original: h
Revised: h

Original: ˆ h ´ | W | ą 0 .
Revised: ˆ h ´ | W | ą 0 .

Original: We ﬁrst show that (3.4) implies positive-deﬁniteness.
Revised: We ﬁrst show that (3.4) implies positive-deﬁniteness.

Original: We write
Revised: We write

Original: [Equation]
Revised: [Equation]

Original: By the lower bound on entries of D , the matrix D ´ | M |`| M | Jis symmetric and strictly diagonally dominant (SDD), and thus PD.
Revised: By the lower bound on entries of D , the matrix D ´ | M |`| M | Jis symmetric and strictly diagonally dominant (SDD), and thus PD.

Original: Thus, we obtain I ´ | W |`| W | Ją 0 , so I ´ | W | ą 0 .
Revised: Thus, we obtain I ´ | W |`| W | J [Equation]

Original: For the reverse direction, we appeal to the theory of M-matrices (Plemmons, 1977).
Revised: For the reverse direction, we appeal to the theory of M-matrices (Plemmons, 1977).

Original: We note that
Revised: We note that

Original: I ´ | W |`| W | Jis a nonsingular M-matrix, i.e., a matrix with negative off-diagonal entries whose eigenvalues have positive real parts.
Revised: I ´ | W |`| W | Jis a nonsingular M-matrix, i.e., a matrix with negative off-diagonal entries whose eigenvalues have positive real parts.

Original: By an equivalent characterization of M-matrices, there exists a diagonal matrix E P D ` such that E p
Revised: By an equivalent characterization of M-matrices, there exists a diagonal matrix E P D ` such that E p

Original: I ´ | W |`| W | Jq E is SDD.
Revised: I ´ | W |`| W | Jq E is SDD.

Original: Rearranging gives | W | “ I ´
Revised: Rearranging gives | W | “ I ´

Original: E ´ 1 AE ´ 1 ` S , where A is symmetric and SDD, and S is skew-symmetric ( S J “ ´ S ).
Revised: E ´ 1 AE ´ 1 ` S , where A is symmetric and SDD, and S is skew-symmetric ( S J “ ´ S ).

Original: Now we choose M satisfying | M | “ E 2 ´
Revised: Now we choose M satisfying | M | “ E 2 ´

Original: A ` ESE and D “ E 2 , so that W has the parameterization W “ D ´ 1 { 2 MD ´ 1 { 2 .
Revised: A ` ESE and D “ E 2 , so that W has the parameterization W “ D ´ 1 { 2 MD ´ 1 { 2 .

Original: To check that D i,i ą p| M | 1 q i `p| M | J 1 q i, we note that p| M | 1 q i `p| M | J 1 q i“ pp E
Revised: To check that D i,i ą p| M | 1 q i `p| M | J 1 q i, we note that p| M | 1 q i `p| M | J 1 q i“ pp E

Original: 2 ´
Revised: 2 ´

Original: A q 1 q i “ E 2 i,i ´
Revised: A q 1 q i “ E 2 i,i ´

Original: A i,i ´ ř j ‰ i
Revised: A i,i ´ ř j ‰ i

Original: A i,j ă E 2 i,i , where we use the fact that A i,i ą ř j ‰ i | A i,j | because A is SDD.
Revised: A i,j ă E 2 i,i , where we use the fact that A i,i ą ř j ‰ i | A i,j | because A is SDD.

Original: Plugging in D i,i “ E 2 i,i gives the desired inequality.
Revised: Plugging in D i,i “ E 2 i,i gives the desired inequality.

Original: 3. E XPRESSIVITY OF THE IBP-M ON DEQ
Revised: 3. E XPRESSIVITY OF THE IBP-M ON DEQ

Original: Guaranteeing that (3.3) has a ﬁxed point requires placing further restrictions on W on top of those already proposed for MonDEQs (Winston & Kolter, 2020).
Revised: Guaranteeing that (3.3) has a ﬁxed point requires placing further restrictions on W on top of those already proposed for MonDEQs (Winston & Kolter, 2020).

Original: This section examines the expressivity of the resulting IBP-MonDEQ parameterization.
Revised: This section examines the expressivity of the resulting IBP-MonDEQ parameterization.

Original: To sanity check that our parameterization is not too restrictive, we ﬁrst contrast against normalization techniques for controlling Lipschitz constants in neural networks.
Revised: To sanity check that our parameterization is not too restrictive, we ﬁrst contrast against normalization techniques for controlling Lipschitz constants in neural networks.

Original: Next, we demonstrate that with the relaxed parameterization of Theorem 3.3, the IBP-MonDEQ is able to express traditional explicit feedforward networks.
Revised: Next, we demonstrate that with the relaxed parameterization of Theorem 3.3, the IBP-MonDEQ is able to express traditional explicit feedforward networks.

Original: Operator norm of W .
Revised: Operator norm of W .

Original: We ﬁrst investigate the operator norm of W .
Revised: We ﬁrst investigate the operator norm of W .

Original: A number of papers constrain the weight matrices so that their operator norms are bounded by 1, which prevents blowup of the naive upper bound ś i } W i } op on the Lipschitz constant of the whole network (Cisse et al., 2017;
Revised: A number of papers constrain the weight matrices so that their operator norms are bounded by 1, which prevents blowup of the naive upper bound ś i } W i } op on the Lipschitz constant of the whole network (Cisse et al., 2017;

Original: Miyato et al., 2018; Farnia et al., 2018; Trockman & Kolter, 2021).
Revised: Miyato et al., 2018; Farnia et al., 2018; Trockman & Kolter, 2021).

Original: This approach has applications in robustness, but there are also concerns that constraining the operator norms to 1 severely harms expressivity (Huster et al., 2018; Anil et al., 2019).
Revised: This approach has applications in robustness, but there are also concerns that constraining the operator norms to 1 severely harms expressivity (Huster et al., 2018; Anil et al., 2019).

Original: We verify that the parameterization of
Revised: We verify that the parameterization of

Original: Theorem 3.1 does not imply the stringent constraint } W } op ď 1 .
Revised: Theorem 3.1 does not imply the stringent constraint } W } op ď 1 .

Original: Example 3.6.
Revised: Example 3.6.

Original: The following choice of W satisﬁes the parameterization (3.4) with } W } op “ 1 . 9 :
Revised: The following choice of W satisﬁes the parameterization (3.4) with } W } op “ 1 . 9 :

Original: W “
Revised: W “

Original: [Equation]
Revised: [Equation]

Original:  .
Revised:  .

Original: We set M “ W and D “ I 2 ˆ 2 , so W “ D ´ 1 { 2 MD ´ 1 { 2 as in (3.4) .
Revised: We set M “ W and D “ I 2 ˆ 2 , so W “ D ´ 1 { 2 MD ´ 1 { 2 as in (3.4) .

Original: Example 3.6 may be surprising given superﬁcial similarities between (3.4) and the following parameterization, which does imply } W } op ď 1 : W “ D ´ 1 M , where D satisﬁes D i,i ě p| M | 1 q i .
Revised: Example 3.6 may be surprising given superﬁcial similarities between (3.4) and the following parameterization, which does imply } W } op ď 1 : W “ D ´ 1 M , where D satisﬁes D i,i ě p| M | 1 q i .

Original: The parameterization in Theorem 3.1 avoids this constraint because the requirement is instead
Revised: The parameterization in Theorem 3.1 avoids this constraint because the requirement is instead

Original: D i,i ě p| M | 1 `| M | J 1 q i, so mass in M can be asymmetrically distributed between the upper and lower triangular halves, as in Example 3.6.
Revised: D i,i ě p| M | 1 `| M | J 1 q i, so mass in M can be asymmetrically distributed between the upper and lower triangular halves, as in Example 3.6.

Original: Expressing explicit feedforward networks.
Revised: Expressing explicit feedforward networks.

Original: The following result states that using the parameterization in Theorem 3.3, a single IBP-MonDEQ layer can express all explicit feedforward networks.
Revised: The following result states that using the parameterization in Theorem 3.3, a single IBP-MonDEQ layer can express all explicit feedforward networks.

Original: Proposition 3.7.
Revised: Proposition 3.7.

Original: Consider explicit neural net embedding functions of the following form:
Revised: Consider explicit neural net embedding functions of the following form:

Original: [Equation]
Revised: [Equation]

Original: where z L : R d Ñ R h and A i P R h
Revised: where z L : R d Ñ R h and A i P R h

Original: ˆ h @
Revised: ˆ h @

Original: 1 ď i ď L .
Revised: 1 ď i ď L .

Original: There is a DEQ layer z ‹ : R d Ñ R Lh outputting z ‹ p x q “ σ p Wz ‹ p x q ` Bx q , where W, B P R
Revised: There is a DEQ layer z ‹ : R d Ñ R Lh outputting z ‹ p x q “ σ p Wz ‹ p x q ` Bx q , where W, B P R

Original: Lh ˆ Lh , such that W obeys the parameterization of
Revised: Lh ˆ Lh , such that W obeys the parameterization of

Original: Theorem 3.3, and the last h coordinates of z ‹ satisfy
Revised: Theorem 3.3, and the last h coordinates of z ‹ satisfy

Original: [Equation]
Revised: [Equation]

Original: Proposition 3.7 states that the IBP-MonDEQ parameterized in Theorem 3.3 is at least as powerful as standard explicit networks, and is somewhat surprising given that the parameterizations (3.4) and (3.7) may appear restrictive.
Revised: Proposition 3.7 states that the IBP-MonDEQ parameterized in Theorem 3.3 is at least as powerful as standard explicit networks, and is somewhat surprising given that the parameterizations (3.4) and (3.7) may appear restrictive.

Original: The proof, provided in Section A, leverages the fact that Γ provides ﬂexibility in scaling the weights, essentially allowing us to cancel out the normalization matrix D .
Revised: The proof, provided in Section A, leverages the fact that Γ provides ﬂexibility in scaling the weights, essentially allowing us to cancel out the normalization matrix D .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we empirically compare models with IBP-MonDEQ layers against fully explicit models.
Revised: In this section, we empirically compare models with IBP-MonDEQ layers against fully explicit models.

Original: Our results show that IBP-MonDEQ models can achieve comparable or improved performance on common certiﬁed robustness benchmarks compared to IBP with standard explicit models.
Revised: Our results show that IBP-MonDEQ models can achieve comparable or improved performance on common certiﬁed robustness benchmarks compared to IBP with standard explicit models.

Original: Explicit models.
Revised: Explicit models.

Original: Table 1 illustrates the explicit architectures: a smaller 3L AYER model withconvolutional layers and 128 output channels, and a larger 7L AYER model.
Revised: Table 1 illustrates the explicit architectures: a smaller 3L AYER model withconvolutional layers and 128 output channels, and a larger 7L AYER model.

Original: The 7L AYER model is a wide and shallow network with 17 million parameters proposed by Gowal et al.
Revised: The 7L AYER model is a wide and shallow network with 17 million parameters proposed by Gowal et al.

Original: (2018) for IBP.
Revised: (2018) for IBP.

Original: IBP-MonDEQ models.
Revised: IBP-MonDEQ models.

Original: As shown in Table 1, we consider replacing a convolutional layer in a fullyexplicit model with an IBP-MonDEQ layer where W computes an equivalently-sized convolutional transformation.
Revised: As shown in Table 1, we consider replacing a convolutional layer in a fullyexplicit model with an IBP-MonDEQ layer where W computes an equivalently-sized convolutional transformation.

Original: This leads to DEQ-3 and DEQ-7 models with equivalent parameter counts as their fully-explicit counterparts.
Revised: This leads to DEQ-3 and DEQ-7 models with equivalent parameter counts as their fully-explicit counterparts.

Original: We use ` LBEN to indicate that the IBP-MonDEQ layer uses the
Revised: We use ` LBEN to indicate that the IBP-MonDEQ layer uses the

Original: LBEN parameterization in Theorem 3.3.
Revised: LBEN parameterization in Theorem 3.3.

Original: To solve for the ﬁxed points, we use the forward-backward algorithm with Anderson acceleration (Walker & Ni, 2011; imp, 2020).
Revised: To solve for the ﬁxed points, we use the forward-backward algorithm with Anderson acceleration (Walker & Ni, 2011; imp, 2020).

Original: To guarantee that (3.6) has a unique ﬁxed point, we enforce either (3.4) or (3.7) (depending on whether we use the LBEN parameterization) for the IBP-MonDEQ weight W .
Revised: To guarantee that (3.6) has a unique ﬁxed point, we enforce either (3.4) or (3.7) (depending on whether we use the LBEN parameterization) for the IBP-MonDEQ weight W .

Original: To enforce (3.4), we set W “ D ´ 1 { 2 MD ´ 1 { 2 where M is the only trainable parameter and D “ diag ´
Revised: To enforce (3.4), we set W “ D ´ 1 { 2 MD ´ 1 { 2 where M is the only trainable parameter and D “ diag ´

Original: [Equation]
Revised: [Equation]

Original: q ¯ is a ﬁxed function of M , where diag p¨q maps a vector to a diagonal matrix.
Revised: q ¯ is a ﬁxed function of M , where diag p¨q maps a vector to a diagonal matrix.

Original: Here m P p 0 , 1 q is a
Revised: Here m P p 0 , 1 q is a

Original: [Table] [Table]
Revised: [Table] [Table]

Original: DEQ-7 0.79 (0.06) 1.16 (0.03) 2.13 (0.05) 2.13 (0.07) 35.57 (0.28) 56.35 (0.37) (+LBEN) 0.80 (0.04) 1.13 (0.02) 2.04 (0.05) 2.10 (0.07) 36.12 (0.06) 56.58 (0.25) tunable hyperparameter governing a tradeoff between expressivity and convergence rate: a smaller m leads to more expressivity but also slower convergence of the equilbrium solvers.
Revised: DEQ-7 0.79 (0.06) 1.16 (0.03) 2.13 (0.05) 2.13 (0.07) 35.57 (0.28) 56.35 (0.37) (+LBEN) 0.80 (0.04) 1.13 (0.02) 2.04 (0.05) 2.10 (0.07) 36.12 (0.06) 56.58 (0.25) tunable hyperparameter governing a tradeoff between expressivity and convergence rate: a smaller m leads to more expressivity but also slower convergence of the equilbrium solvers.

Original: For the LBEN parameterization (3.7), we parameterize the additional diagonal scaling matrix
Revised: For the LBEN parameterization (3.7), we parameterize the additional diagonal scaling matrix

Original: Γ P D
Revised: Γ P D

Original: h
Revised: h

Original: ˆ h ` by Γ “ diag
Revised: ˆ h ` by Γ “ diag

Original: [Equation]
Revised: [Equation]

Original: ¯ , where γ ą 1 is a scalar hyperparameter, and β P R h is a trainable parameter β P R h .
Revised: ¯ , where γ ą 1 is a scalar hyperparameter, and β P R h is a trainable parameter β P R h .

Original: This parameterization enforces that the condition number of Γ is bounded by γ .
Revised: This parameterization enforces that the condition number of Γ is bounded by γ .

Original: C ERTIFIED ROBUSTNESS RESULTS
Revised: C ERTIFIED ROBUSTNESS RESULTS

Original: Tables 2 shows certiﬁed and standard classiﬁcation errors of 3 and 7 layer models trained with IBP on the MNIST and CIFAR10 datasets for various values of (cid:15) .
Revised: Tables 2 shows certiﬁed and standard classiﬁcation errors of 3 and 7 layer models trained with IBP on the MNIST and CIFAR10 datasets for various values of (cid:15) .

Original: For all settings, IBP-MonDEQ models are able to achieve lower mean certiﬁed errors than their fully explicit counterparts.
Revised: For all settings, IBP-MonDEQ models are able to achieve lower mean certiﬁed errors than their fully explicit counterparts.

Original: The improvement is particularly striking in the case of 3 layer models on MNIST, where IBPMonDEQ models perform much better, e.g. 9.87% better on MNIST for (cid:15) “ 0 . 4 .
Revised: The improvement is particularly striking in the case of 3 layer models on MNIST, where IBPMonDEQ models perform much better, e.g. 9.87% better on MNIST for (cid:15) “ 0 . 4 .

Original: We also ﬁnd that using the more relaxed LBEN parameterization (Theorem 3.3) generally leads to better performance for 3 layer models, but the reverse trend may hold for larger 7 layer models.
Revised: We also ﬁnd that using the more relaxed LBEN parameterization (Theorem 3.3) generally leads to better performance for 3 layer models, but the reverse trend may hold for larger 7 layer models.

Original: We hypothesize that the LBEN parameterization may lead to optimization difﬁculties in the IBP-MonDEQ layer when there are other additional layers in the network.
Revised: We hypothesize that the LBEN parameterization may lead to optimization difﬁculties in the IBP-MonDEQ layer when there are other additional layers in the network.

Original: Investigating and improving the optimization of
Revised: Investigating and improving the optimization of

Original: IBP-MonDEQ layers is an interesting direction for future work.
Revised: IBP-MonDEQ layers is an interesting direction for future work.

Original: Implementation details.
Revised: Implementation details.

Original: We train all models using IBP for 300 epochs using the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 5e-4, and report errors at the 300th epoch of training averaged over 3 runs.
Revised: We train all models using IBP for 300 epochs using the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 5e-4, and report errors at the 300th epoch of training averaged over 3 runs.

Original: We use a smooth warmup schedule for (cid:15)
Revised: We use a smooth warmup schedule for (cid:15)

Original: (Xu et al., 2020; Shi et al., 2021), interpolating between (cid:15) “ 0 and (cid:15) “ (cid:15) train , a target training (cid:15) lower bounded by (cid:15) test .
Revised: (Xu et al., 2020; Shi et al., 2021), interpolating between (cid:15) “ 0 and (cid:15) “ (cid:15) train , a target training (cid:15) lower bounded by (cid:15) test .

Original: Following Shi et al.
Revised: Following Shi et al.

Original: (2021), we use batch normalization (BN) after every linear layer besides the last F C -10 layer.
Revised: (2021), we use batch normalization (BN) after every linear layer besides the last F C -10 layer.

Original: There is no BN within an IBP-MonDEQ layer, but BN is applied to outputs of the IBP-MonDEQ layers.
Revised: There is no BN within an IBP-MonDEQ layer, but BN is applied to outputs of the IBP-MonDEQ layers.

Original: For 3 layer models, we insert a 4 x 4 pooling layer before the F C -10 layer.
Revised: For 3 layer models, we insert a 4 x 4 pooling layer before the F C -10 layer.

Original: We also use the IBP initialization proposed by (Shi et al., 2021) for all explicit layer weights, which prevents IBP bounds from exploding at initialization.
Revised: We also use the IBP initialization proposed by (Shi et al., 2021) for all explicit layer weights, which prevents IBP bounds from exploding at initialization.

Original: Additional details are in Section B.
Revised: Additional details are in Section B.

Original:  A DDITIONAL RELATED WORKS
Revised:  A DDITIONAL RELATED WORKS

Original: Implicit layers.
Revised: Implicit layers.

Original: Various fundamental properties of DEQs have been investigated in the literature, most notably convergence to and existence of a ﬁxed point (Winston & Kolter, 2020; Revay et al., 2020; Bai et al., 2021).
Revised: Various fundamental properties of DEQs have been investigated in the literature, most notably convergence to and existence of a ﬁxed point (Winston & Kolter, 2020; Revay et al., 2020; Bai et al., 2021).

Original: Bai et al.
Revised: Bai et al.

Original: (2021) propose empirical Jacobian regularization techniques for speeding up equilibrium solving in DEQs.
Revised: (2021) propose empirical Jacobian regularization techniques for speeding up equilibrium solving in DEQs.

Original: Another studied property of DEQs is their Lipschitzness: Revay et al.
Revised: Another studied property of DEQs is their Lipschitzness: Revay et al.

Original: (2020) and Pabbaraju et al.
Revised: (2020) and Pabbaraju et al.

Original: (2020) derive bounds on the Lipschitz constants of
Revised: (2020) derive bounds on the Lipschitz constants of

Original: MonDEQs which are applicable in certiﬁed robustness settings.
Revised: MonDEQs which are applicable in certiﬁed robustness settings.

Original: However, because the Lipschitz constants are tailored to the (cid:96) 2 norm, these works only consider (cid:96) 2 bounded adversaries.
Revised: However, because the Lipschitz constants are tailored to the (cid:96) 2 norm, these works only consider (cid:96) 2 bounded adversaries.

Original: Chen et al.
Revised: Chen et al.

Original: (2021) propose polynomial optimization programs for certifying robustness of MonDEQs, but their methods only succeed for (cid:96) 2 perturbations and do not work empirically against (cid:96) 8 adversaries.
Revised: (2021) propose polynomial optimization programs for certifying robustness of MonDEQs, but their methods only succeed for (cid:96) 2 perturbations and do not work empirically against (cid:96) 8 adversaries.

Original: Kawaguchi (2021) theoretically analyze optimization of equilibrium models.
Revised: Kawaguchi (2021) theoretically analyze optimization of equilibrium models.

Original: Other prominent classes of implicit models include neural ODEs (Chen et al., 2018;
Revised: Other prominent classes of implicit models include neural ODEs (Chen et al., 2018;

Original: Dupont et al., 2019; Liu et al., 2019; Finlay et al., 2020), and convex optimization layers (Amos et al., 2017; Amos & Kolter, 2017; Agrawal et al., 2019).
Revised: Dupont et al., 2019; Liu et al., 2019; Finlay et al., 2020), and convex optimization layers (Amos et al., 2017; Amos & Kolter, 2017; Agrawal et al., 2019).

Original: Neural ODE outputs are implicitly deﬁned as the solution to some differential equation.
Revised: Neural ODE outputs are implicitly deﬁned as the solution to some differential equation.

Original: Adversarial robustness is also an interesting topic of study for neural
Revised: Adversarial robustness is also an interesting topic of study for neural

Original: ODEs, which may be more robust than explicit models (Yan et al., 2019), though the cause of this might be gradient obfuscation (Huang et al., 2020).
Revised: ODEs, which may be more robust than explicit models (Yan et al., 2019), though the cause of this might be gradient obfuscation (Huang et al., 2020).

Original: Certiﬁed robustness.
Revised: Certiﬁed robustness.

Original: A common technique in certiﬁed robustness is to express a convex program which lower bounds the worst-case perturbed output of the network, either via semideﬁnite relaxations (Raghunathan et al., 2018a;b), or linear relaxations (Wong & Kolter, 2018;
Revised: A common technique in certiﬁed robustness is to express a convex program which lower bounds the worst-case perturbed output of the network, either via semideﬁnite relaxations (Raghunathan et al., 2018a;b), or linear relaxations (Wong & Kolter, 2018;

Original: Wong et al., 2018; Dvijotham et al.
Revised: Wong et al., 2018; Dvijotham et al.

Original: ; Zhang et al., 2018; Wang et al., 2018).
Revised: ; Zhang et al., 2018; Wang et al., 2018).

Original: One advantage of IBP compared to these approaches is its simplicity and efﬁciency, making it viable for training larger models (Gowal et al., 2018).
Revised: One advantage of IBP compared to these approaches is its simplicity and efﬁciency, making it viable for training larger models (Gowal et al., 2018).

Original: Zhang et al.
Revised: Zhang et al.

Original: (2019) propose CROWN-IBP, which combines IBP with linear relaxation methods during warmup training.
Revised: (2019) propose CROWN-IBP, which combines IBP with linear relaxation methods during warmup training.

Original: Xu et al.
Revised: Xu et al.

Original: (2020) extend this approach to general computation graphs.
Revised: (2020) extend this approach to general computation graphs.

Original: Shi et al.
Revised: Shi et al.

Original: (2021) propose methods for speeding up optimization of IBP-trained networks.
Revised: (2021) propose methods for speeding up optimization of IBP-trained networks.

Original: Another well-studied certiﬁed robustness approach is randomized smoothing (Li et al., 2018;
Revised: Another well-studied certiﬁed robustness approach is randomized smoothing (Li et al., 2018;

Original: Cohen et al., 2019; Lecuyer et al., 2019;
Revised: Cohen et al., 2019; Lecuyer et al., 2019;

Original: Salman et al., 2019a; Yang et al., 2020), which can in theory produce probabilistic certiﬁcates of robustness for generic models (including, in theory, DEQs).
Revised: Salman et al., 2019a; Yang et al., 2020), which can in theory produce probabilistic certiﬁcates of robustness for generic models (including, in theory, DEQs).

Original: However, these certiﬁcates are random, not deterministic, and have some error probability.
Revised: However, these certiﬁcates are random, not deterministic, and have some error probability.

Original: In addition, randomized smoothing has theoretical limitations for certifying (cid:96) 8 robustness for high-dimensional data (Blum et al., 2020; Kumar et al., 2020;
Revised: In addition, randomized smoothing has theoretical limitations for certifying (cid:96) 8 robustness for high-dimensional data (Blum et al., 2020; Kumar et al., 2020;

Original: Yang et al., 2020).
Revised: Yang et al., 2020).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This work aims to certify robustness of DEQ models.
Revised: This work aims to certify robustness of DEQ models.

Original: We propose the IBP-MonDEQ layer, a DEQ layer which solves two equilibrium equations: one for the unperturbed ﬁxed-point output, and one for upper and lower interval bounds on adversarially perturbed values of the ﬁxed point.
Revised: We propose the IBP-MonDEQ layer, a DEQ layer which solves two equilibrium equations: one for the unperturbed ﬁxed-point output, and one for upper and lower interval bounds on adversarially perturbed values of the ﬁxed point.

Original: Our mathematical analysis reveals how to parameterize the IBP-MonDEQ so that existence and uniqueness of a ﬁxed point is guaranteed.
Revised: Our mathematical analysis reveals how to parameterize the IBP-MonDEQ so that existence and uniqueness of a ﬁxed point is guaranteed.

Original: One interesting interpretation of these results is that they characterize a class of inﬁnitely deep, weight-tied neural nets for which IBP is stable.
Revised: One interesting interpretation of these results is that they characterize a class of inﬁnitely deep, weight-tied neural nets for which IBP is stable.

Original: An interesting direction for future work is to explore whether other methods besides IBP for certifying robustness of explicit networks can also be adapted to work for DEQs.
Revised: An interesting direction for future work is to explore whether other methods besides IBP for certifying robustness of explicit networks can also be adapted to work for DEQs.

Original:  E THICS AND REPRODUCIBILITY STATEMENTS
Revised:  E THICS AND REPRODUCIBILITY STATEMENTS

Original: We do not expect ethical concerns to arise from this research because DEQ methods are still exploratory in nature, and so the main impacts of this work are still mainly theoretical and algorithmic.
Revised: We do not expect ethical concerns to arise from this research because DEQ methods are still exploratory in nature, and so the main impacts of this work are still mainly theoretical and algorithmic.

Original: All proofs omitted from the main body of the paper are in Section
Revised: All proofs omitted from the main body of the paper are in Section

Original: A. Additional implementation details are provided in Section B, and code is submitted with the supplementary material.
Revised: A. Additional implementation details are provided in Section B, and code is submitted with the supplementary material.

Original: A M ISSING PROOFS IN S ECTION
Revised: A M ISSING PROOFS IN S ECTION

Original: We ﬁrst ﬁll in the proofs of Claim 3.5 and Lemma 3.4.
Revised: We ﬁrst ﬁll in the proofs of Claim 3.5 and Lemma 3.4.

Original: Proof of Claim 3.5.
Revised: Proof of Claim 3.5.

Original: Let u “ „ st  be any eigenvector of M , with eigenvalue λ .
Revised: Let u “ „ st  be any eigenvector of M , with eigenvalue λ .

Original: There are two cases:
Revised: There are two cases:

Original: Case 1: s ‰ ´ t .
Revised: Case 1: s ‰ ´ t .

Original: As λ „ st  “ Mu “ „
Revised: As λ „ st  “ Mu “ „

Original: As ` Bt
Revised: As ` Bt

Original: Bs ` At  , it follows that λs “ As ` Bt and λt “ Bs ` At .
Revised: Bs ` At  , it follows that λs “ As ` Bt and λt “ Bs ` At .

Original: Thus, it follows that p
Revised: Thus, it follows that p

Original: A ` B qp s ` t q “ As ` Bt ` Bs ` At “ λ p s ` t q .
Revised: A ` B qp s ` t q “ As ` Bt ` Bs ` At “ λ p s ` t q .

Original: As the condition of this case assumed p s ` t q ‰ 0 , λ is an eigenvalue of A ` B .
Revised: As the condition of this case assumed p s ` t q ‰ 0 , λ is an eigenvalue of A ` B .

Original: Case 2: s “ ´ t .
Revised: Case 2: s “ ´ t .

Original: Then λ „ s ´ s  “ Mu “ „ p A ´ B q s ´p
Revised: Then λ „ s ´ s  “ Mu “ „ p A ´ B q s ´p

Original: A ´ B q s  .
Revised: A ´ B q s  .

Original: It follows that λs “ p A ´ B q s , so λ is an eigenvalue of A ´ B .
Revised: It follows that λs “ p A ´ B q s , so λ is an eigenvalue of A ´ B .

Original: Using Claim 3.5, we can complete the proof of Lemma 3.4.
Revised: Using Claim 3.5, we can complete the proof of Lemma 3.4.

Original: Proof of Lemma 3.4.
Revised: Proof of Lemma 3.4.

Original: Deﬁne A ﬁ p W q ` `p W q J`and B ﬁ p W q ´ `p W q J´, so x W “ „
Revised: Deﬁne A ﬁ p W q ` `p W q J`and B ﬁ p W q ´ `p W q J´, so x W “ „

Original: A
Revised: A

Original: B B
Revised: B B

Original: A  .
Revised: A  .

Original: We note that A ` B “ pp W q ` `p W q ´ q`pp W q ` `p W q ´ q J“ W ` W Jand likewise A ´ B “ | W |`| W | J. We now directly apply Claim 3.5 to conclude that eigenvalues of Ă W are either eigenvalues of A ` B or
Revised: We note that A ` B “ pp W q ` `p W q ´ q`pp W q ` `p W q ´ q J“ W ` W Jand likewise A ´ B “ | W |`| W | J. We now directly apply Claim 3.5 to conclude that eigenvalues of x W are either eigenvalues of A ` B or

Original: A ´ B , as desired.
Revised: A ´ B , as desired.

Original: (3.8) follows by observing that λ max  p  | W |`| W | Jq ě λ max p W ` W Jq .
Revised: To see the left-to-right implication of (3.8), we use the fact that the set of eigenvalues of x W ` x W J contains those of | W | ` | W | J as a subset. For the right-to-left implication, we note that I ´ | W | ą 0 implies that λ max pp| W | ` | W | J q{ 2 q ă 1 . Next, we can observe that λ max p| W | ` | W | J q “ λ max p x W ` x W J q because eigenvalues of x W ` x W J are either eigenvalues of W ` W J or eigenvalues of | W |`| W | J . Now we can use the fact that λ max p W ` W J q ď λ max p| W |`| W | J q , which would give us λ max pp x W ` x W J q{ 2 q ă 1 , or I ´ x

Original:  Next, we complete the proof of Proposition 3.2.
Revised: W ą 0 . Finally, to see that λ max p W ` W J q ď λ max p| W | ` | W | J q , let v be the eigenvector corresponding to the maximum eigenvalue of W ` W J . The key observation is that v J p W ` W J q v ď | v | J p| W |` | W | J q| v | because terms in the expansion of the ﬁrst quantity will be replaced by positive versions with the same magnitude in the r.h.s. Next, we complete the proof of Proposition 3.2.

Original: Proof of Proposition 3.2.
Revised: Proof of Proposition 3.2.

Original: By results of Winston & Kolter (2020), there exists 

Revised: By results of Winston & Kolter (2020), there exists

Original: ă α ď 1 such that the damped iteration
Revised: ă α ď 1 such that the damped iteration

Original: [Equation]
Revised: [Equation]

Original: is guaranteed to converge for any initial s z 0 , z 0 : lim k Ñ8 s z k , z k “ s z ‹ , z ‹ .
Revised: is guaranteed to converge for any initial s z 0 , z 0 : lim k Ñ8 s z k , z k “ s z ‹ , z ‹ .

Original: For any v satisfying v ď v ď s v , we consider iterating (A.1) with s z 

Revised: For any v satisfying v ď v ď s v , we consider iterating (A.1) with s z

Original: “ z 0 “ z ‹ p v q .
Revised: “ z 0 “ z ‹ p v q .

Original: We claim that the following invariant holds: s z k ě z ‹ p v q ě z k @ k ě 0 .
Revised: We claim that the following invariant holds: s z k ě z ‹ p v q ě z k @ k ě 0 .

Original: To see this, we also consider the damped iteration for z ‹ p v q , noting that the following equality holds by (Winston & Kolter, 2020):
Revised: To see this, we also consider the damped iteration for z ‹ p v q , noting that the following equality holds by (Winston & Kolter, 2020):

Original: [Equation]
Revised: [Equation]

Original: Now we apply induction on k .
Revised: Now we apply induction on k .

Original: The base case k “ 0 given by our choice of s z 0 , z 0 .
Revised: The base case k “ 0 given by our choice of s z 0 , z 0 .

Original: We compare corresponding terms in (A.1) and (A.2) for computing s u, u , and u ‹ .
Revised: We compare corresponding terms in (A.1) and (A.2) for computing s u, u , and u ‹ .

Original: We note that s z k ´ 1 ě z ‹ p v q by the inductive hypothesis,
Revised: We note that s z k ´ 1 ě z ‹ p v q by the inductive hypothesis,

Original: [Equation]
Revised: [Equation]

Original: ě z ‹ p v q by the inductive hypothesis and properties of the IBP iteration, and s v ě v by assumption.
Revised: by the inductive hypothesis and properties of the IBP iteration, and s v ě v by assumption.

Original: Thus, it follows that s u k ě u ‹ , and likewise, u k ď u ‹ .
Revised: Thus, it follows that s u k ě u ‹ , and likewise, u k ď u ‹ .

Original: Since σ is nondecreasing, we obtain s z k ě z ‹ p v q ě z k , completing the induction.
Revised: Since σ is nondecreasing, we obtain s z k ě z ‹ p v q ě z k , completing the induction.

Original: Taking limits as k Ñ 8 , we obtain s z ‹ ě z ‹ p v q ě z ‹ , as desired.
Revised: Taking limits as k Ñ 8 , we obtain s z ‹ ě z ‹ p v q ě z ‹ , as desired.

Original: Next, we complete the proof of Theorem 3.3.
Revised: Next, we complete the proof of Theorem 3.3.

Original: The following lemma will be useful.
Revised: The following lemma will be useful.

Original: Lemma A.1.
Revised: Lemma A.1.

Original: In the setting of Theorem 3.3, suppose that W satisﬁes Λ ´ Λ W ą 0 for some Λ P D
Revised: In the setting of Theorem 3.3, suppose that W satisﬁes Λ ´ Λ W ą 0 for some Λ P D

Original: h
Revised: h

Original: ˆ h ` .
Revised: ˆ h ` .

Original: Then there exists 

Revised: Then there exists

Original: ă α ď 1 such that the following iteration converges:
Revised: ă α ď 1 such that the following iteration converges:

Original: [Equation]
Revised: [Equation]

Original: with lim k Ñ8 z k “ z ‹ , where z ‹ is the solution to the ﬁxed point equation z ‹ “ σ p Wz ‹ ` v q .
Revised: with lim k Ñ8 z k “ z ‹ , where z ‹ is the solution to the ﬁxed point equation z ‹ “ σ p Wz ‹ ` v q .

Original: Proof of Lemma A.1.
Revised: Proof of Lemma A.1.

Original: We ﬁrst observe that the linear operator I ´
Revised: We ﬁrst observe that the linear operator I ´

Original: α p Λ ´ Λ W q is contractive for sufﬁciently small α , as Λ ´ Λ W ą 0 .
Revised: α p Λ ´ Λ W q is contractive for sufﬁciently small α , as Λ ´ Λ W ą 0 .

Original: Thus, since ReLU is 1-Lipschitz, the iteration in (A.3) contracts and therefore converges to a ﬁxed point.
Revised: Thus, since ReLU is 1-Lipschitz, the iteration in (A.3) contracts and therefore converges to a ﬁxed point.

Original: To check that this gives the same ﬁxed point as (3.3), we will check that the following holds:
Revised: To check that this gives the same ﬁxed point as (3.3), we will check that the following holds:

Original: [Equation]
Revised: [Equation]

Original: Note that for coordinates i where p Wz ‹ ` v q i ą 0 , the deﬁnition of z ‹ implies z ‹ i “ p Wz ‹ ` v q i , so (A.4) is veriﬁed for such coordinates.
Revised: Note that for coordinates i where p Wz ‹ ` v q i ą 0 , the deﬁnition of z ‹ implies z ‹ i “ p Wz ‹ ` v q i , so (A.4) is veriﬁed for such coordinates.

Original: Otherwise, for coordinates i where p Wz ‹ ` v q i ď 0 , since σ is the ReLU activation, z ‹ i “ 0 , so we can again verify (A.4) for these coordinates.
Revised: Otherwise, for coordinates i where p Wz ‹ ` v q i ď 0 , since σ is the ReLU activation, z ‹ i “ 0 , so we can again verify (A.4) for these coordinates.

Original: We now complete the proof of Theorem 3.3.
Revised: We now complete the proof of Theorem 3.3.

Original: Proof of Theorem 3.3.
Revised: Proof of Theorem 3.3.

Original: First, we consider any W of the form (3.7).
Revised: First, we consider any W of the form (3.7).

Original: Taking Λ “ Γ ´ 2 , we have Λ ´ Λ W “ Γ ´ 1 p
Revised: Taking Λ “ Γ ´ 2 , we have Λ ´ Λ W “ Γ ´ 1 p

Original: I ´ D ´ 1 { 2 MD ´ 1 { 2 q Γ ´ 1 .
Revised: I ´ D ´ 1 { 2 MD ´ 1 { 2 q Γ ´ 1 .

Original: Now by invoking Theorem 3.1, we conclude that the middle term in this product is PD, so Λ ´ Λ W ą 0 .
Revised: Now by invoking Theorem 3.1, we conclude that the middle term in this product is PD, so Λ ´ Λ W ą 0 .

Original: To prove the reverse implication, we note that Λ ´ Λ | W | “ Λ 1 { 2 p I ´ Λ 1 { 2 | W | Λ ´ 1 { 2 q Λ 1 { 2 , so Λ ´ Λ | W | ą 

Revised: To prove the reverse implication, we note that Λ ´ Λ | W | “ Λ 1 { 2 p I ´ Λ 1 { 2 | W | Λ ´ 1 { 2 q Λ 1 { 2 , so Λ ´ Λ | W | ą

Original: ðñ I ´ Λ 1 { 2 | W | Λ ´ 1 { 2 ą 0 .
Revised: ðñ I ´ Λ 1 { 2 | W | Λ ´ 1 { 2 ą 0 .

Original: By invoking Theorem 3.1, it follows that we have the parameterization Λ 1 { 2 W Λ ´ 1 { 2 “ D 1 { 2 MD 1 { 2 , so setting Γ “ Λ ´ 1 { 2 and rearranging gives the desired result.
Revised: By invoking Theorem 3.1, it follows that we have the parameterization Λ 1 { 2 W Λ ´ 1 { 2 “ D 1 { 2 MD 1 { 2 , so setting Γ “ Λ ´ 1 { 2 and rearranging gives the desired result.

Original: Next, we check that (2.4) and (3.3) have guaranteed unique ﬁxed points whenever W is parameterized as in (3.7).
Revised: Next, we check that (2.4) and (3.3) have guaranteed unique ﬁxed points whenever W is parameterized as in (3.7).

Original: Using the observation derived above, we can check that
Revised: Using the observation derived above, we can check that

Original: [Equation]
Revised: [Equation]

Original: Thus, we can apply Theorem 1 of (Revay et al., 2020) to conclude the existence of a unique ﬁxed point for (2.4).
Revised: Thus, we can apply Theorem 1 of (Revay et al., 2020) to conclude the existence of a unique ﬁxed point for (2.4).

Original: For (3.3), we apply Lemma 3.4 to conclude that p Λ ´ p Λ x W ą 0 , where p Λ ﬁ „ Λ  Λ 
Revised: For (3.3), we apply Lemma 3.4 to conclude that p Λ ´ p Λ x W ą 0 , where p Λ ﬁ „ Λ  Λ 

Original: P
Revised: P

Original: R 2 h ˆ 2 h .
Revised: R 2 h ˆ 2 h .

Original: Thus, Theorem 1 of (Revay et al., 2020) can also be used to conclude that (3.3) has a unique ﬁxed point.
Revised: Thus, Theorem 1 of (Revay et al., 2020) can also be used to conclude that (3.3) has a unique ﬁxed point.

Original: Finally, it remains to check that s z ‹ and z ‹ are indeed valid interval bounds on z ‹ .
Revised: Finally, it remains to check that s z ‹ and z ‹ are indeed valid interval bounds on z ‹ .

Original: The proof of
Revised: The proof of

Original: Proposition 3.2 no longer directly applies because the damped iteration (A.1) might not converge, as we no longer have the guarantee I
Revised: Proposition 3.2 no longer directly applies because the damped iteration (A.1) might not converge, as we no longer have the guarantee I

Original: 2 h ˆ 2 h
Revised: 2 h ˆ 2 h

Original: ´ x W ą 0 .
Revised: ´ x W ą 0 .

Original: Instead, we invoke Lemma A.1, which states the existence of 

Revised: Instead, we invoke Lemma A.1, which states the existence of

Original: ă α ď 1 such that the following iteration converges:
Revised: ă α ď 1 such that the following iteration converges:

Original: [Equation]
Revised: [Equation]

Original: with lim k Ñ8 s z k “ s z ‹ , lim k Ñ8 z k “ z ‹ .
Revised: with lim k Ñ8 s z k “ s z ‹ , lim k Ñ8 z k “ z ‹ .

Original: Using this modiﬁed iteration but keeping other steps the same, we can follow Proposition 3.2 to complete the proof.
Revised: Using this modiﬁed iteration but keeping other steps the same, we can follow Proposition 3.2 to complete the proof.

Original: Next, we provide the proof of Proposition 3.7.
Revised: Next, we provide the proof of Proposition 3.7.

Original: Proof of Proposition 3.7.
Revised: Proof of Proposition 3.7.

Original: We set B “ 
Revised: We set B “ » ——–

Revised: A0 ¨ ¨

Original: [Equation]
Revised: [Equation]

Revised: 0 ¨ ¨ ¨...

Original:  ¨ ¨ﬁ ﬃﬃﬂ .
Revised: 0 ¨ ¨ ¨ﬁ ﬃﬃﬂ .

Original: According to (3.7), the weight matrix
Revised: According to (3.7), the weight matrix

Original: W should take the following form: W “ Γ D ´ 1 { 2 MD ´ 1 { 2 Γ ´ 1 .
Revised: W should take the following form: W “ Γ D ´ 1 { 2 MD ´ 1 { 2 Γ ´ 1 .

Original: We choose Γ to take the following form:
Revised: We choose Γ to take the following form:

Original: [Equation]
Revised: [Equation]

Revised: ﬃﬃﬂ

Original: [Table]
Revised: [Equation]

Revised: A.10)

Original: M will take the following lower triangular block form:
Revised: M will take the following lower triangular block form:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Revised: A

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Revised: ¨

Original: [Equation] 
Revised: [Equation] ¨

Original: [Equation]
Revised: A

Original: [Equation]
Revised: [Equation]

Revised: ¨

Original: [Equation]
Revised: [Equation]

Revised: ¨ ¨ ¨

Original: A L ﬁ ﬃﬃﬃﬃﬂ (A.11) where if we split M into a grid of h ˆ h blocks, the block in the i -th row
Revised: A L ﬁ ﬃﬃﬃﬃ ﬂ (A.11) where if we split M into a grid of h ˆ h blocks, the block in the i -th row

Original: and i ´ 1 -th column has value
Revised: and i ´ 1 -th column has value

Original: A i , and all other blocks are 0 .
Revised: A i , and all other blocks are 0 .

Original: Finally, D will take the form:
Revised: Finally, D will take the form:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: ﬁ ﬃﬃﬂ (A.12)

Original: In order to satisfy the constraints of (3.7), we require β 1 ą max j Pr h s p| A | J 2 1 q  j 
Revised: In order to satisfy the constraints of (3.7), we require β 1 ą max j Pr h s p| A | J 2 1 q j, β L ą max j Pr h s p| A | L 1 q j, and β i ą max j Pr h s pp| A | i `| A | J i ` 1 q 1 q

Original: [Equation]
Revised: [Equation]

Original: β L ą max j Pr h s p| A | L 1 q j, and β i ą max j Pr h s pp| A | i `| A | J i ` 1 q 1 q jfor 2 ď i ď L ´ 1 .
Revised: for 2 ď i ď L ´ 1 .

Original: We now observe that W is of the form
Revised: We now observe that W is of the form

Original: [Equation]
Revised: [Equation]

Original: γ ´ 1 ¨ ¨ ¨ ...

Original: ¨ ¨ ¨ γ L β ´ 1 { 2 L

Original: A L β ´ 1 { 2 L ´ 1 γ ´ 1 L ´ ﬁ ﬃﬃﬃﬃﬃﬃﬂ (A.13)

Original: We observe now that we have L degrees of freedom in choosing γ , so we can choose γ such that γ
Revised: We observe now that we have L degrees of freedom in choosing γ , so we can choose γ such that γ

Original: i β ´ 1 { 2 i β ´ 1 { 2 i ´ 1 γ
Revised: i β ´ 1 { 2 i β ´ 1 { 2 i ´ 1 γ

Original: ´
Revised: ´

Original: i ´ 1 “ 1 @ 2 ď i ď L , as this only imposes L ´ 1 constraints.
Revised: i ´ 1 “ [Equation] @ 2 ď i ď L , as this only imposes L ´ 1 constraints.

Original: For example, we can set γ 1 “ 1 , and set γ 2 , . . .
Revised: For example, we can set γ 1 “ 1 , and set γ 2 , . . .

Original: , γ L in sequence so that the desired equalities hold.
Revised: , γ L in sequence so that the desired equalities hold.

Original: This gives W “ M .
Revised: This gives W “ M .

Original: Finally, it is straightforward to see that the equilibrium for this constructed W and B concatenates all of the hidden layers of the network, so the desired equivalence between the DEQ and explicit network holds.
Revised: Finally, it is straightforward to see that the equilibrium for this constructed W and B concatenates all of the hidden layers of the network, so the desired equivalence between the DEQ and explicit network holds.

Original: B A DDITIONAL IMPLEMENTATION DETAILS
Revised: B A DDITIONAL IMPLEMENTATION DETAILS

Original: Implementation details for Section 4.
Revised: Implementation details for Section 4.

Original: For all models, we use a batch size of 128 and anneal the learning rate, which is initially 5e-4, by a factor of 0.2 at the 120th and 280th epochs.
Revised: For all models, we use a batch size of 128 and anneal the learning rate, which is initially 5e-4, by a factor of 0.2 at the 120th and 280th epochs.

Original: We use gradient clipping with a max (cid:96) 2 norm of 10.
Revised: We use gradient clipping with a max (cid:96) 2 norm of 10.

Original: To preprocess the data, all inputs are normalized to the range r 0 , 1 s before we subtract the mean and divide by the standard deviation of the training images across each input channel.
Revised: To preprocess the data, all inputs are normalized to the range r 0 , 1 s before we subtract the mean and divide by the standard deviation of the training images across each input channel.

Original: Note that the (cid:15) adversarial perturbations are applied to the r 0 , 1 snormalized image before subtracting the mean and dividing by the standard deviation.
Revised: Note that the (cid:15) adversarial perturbations are applied to the r 0 , 1 snormalized image before subtracting the mean and dividing by the standard deviation.

Original: For MNIST, we do not use data augmentation, and for CIFAR10 we use random crops and horizontal ﬂips.
Revised: For MNIST, we do not use data augmentation, and for CIFAR10 we use random crops and horizontal ﬂips.

Original: We note that during both training and testing, when computing IBP bounds we elide the last F Ctransformation with the output margins of the model (which can be expressed as a linear function of the logits), as Gowal et al.
Revised: We note that during both training and testing, when computing IBP bounds we elide the last F C - [Table] transformation with the output margins of the model (which can be expressed as a linear function of the logits), as Gowal et al.

Original: (2018) show that this leads to performance beneﬁts.
Revised: (2018) show that this leads to performance beneﬁts.

Original: For CIFAR10, we choose (cid:15) train “ (cid:15) test , whereas for MNIST we use a larger value of (cid:15) train , following Gowal et al.
Revised: For CIFAR10, we choose (cid:15) train “ (cid:15) test , whereas for MNIST we use a larger value of (cid:15) train , following Gowal et al.

Original: (2018) and Shi et al.
Revised: (2018) and Shi et al.

Original: The values are displayed in Table 3.
Revised: The values are displayed in Table 3.

Original: To schedule (cid:15) , we set (cid:15) “ 0 for the ﬁrst epoch of training, and afterwards we smoothly increase (cid:15) following Xu et al.
Revised: To schedule (cid:15) , we set (cid:15) “ 0 for the ﬁrst epoch of training, and afterwards we smoothly increase (cid:15) following Xu et al.

Original: The (cid:15) schedule interpolates between (cid:15) “
Revised: The (cid:15) schedule interpolates between (cid:15) “

Original: 0 and (cid:15) “ (cid:15) train in a smooth manner: for the ﬁrst4 of the ramp-up iterations, (cid:15) increases according to a degree 4 monomial.
Revised: 0 and (cid:15) “ (cid:15) train in a smooth manner: for the ﬁrst 14 of the ramp-up iterations, (cid:15) increases according to a degree 4 monomial.

Original: After this, (cid:15) increases linearly until it reaches a ﬁnal value of (cid:15) train .
Revised: After this, (cid:15) increases linearly until it reaches a ﬁnal value of (cid:15) train .

Original: The linear slope and coefﬁcient of the monomial are chosen so that the derivative of (cid:15) is continuous.
Revised: The linear slope and coefﬁcient of the monomial are chosen so that the derivative of (cid:15) is continuous.

Original: For all experiments, we use 79 ramp-up epochs.
Revised: For all experiments, we use 79 ramp-up epochs.

Original: All models besides the DEQ-3 can be trained within a day on a single NVIDIA TitanXp GPU.
Revised: All models besides the DEQ-3 can be trained within a day on a single NVIDIA TitanXp GPU.

Original: The
Revised: The

Original: DEQ-3 can take up to two days because the hidden layer size is larger (128 output channels), so each iteration of the equilibrium solver is more costly.
Revised: DEQ-3 can take up to two days because the hidden layer size is larger (128 output channels), so each iteration of the equilibrium solver is more costly.

Original: In all, the training times of the IBP-MonDEQ models vary based on the current model weights but generally are 3-10 times longer than the fullyexplicit models.
Revised: In all, the training times of the IBP-MonDEQ models vary based on the current model weights but generally are 3-10 times longer than the fullyexplicit models.

Original: As it is already known that DEQ models tend to compare unfavorably to fully explicit models in terms of runtime (Bai et al., 2019; 2021), improving the runtime discrepancy is deferred to future work.
Revised: As it is already known that DEQ models tend to compare unfavorably to fully explicit models in terms of runtime (Bai et al., 2019; 2021), improving the runtime discrepancy is deferred to future work.

Original: Finally, we provide additional IBP-MonDEQ-speciﬁc implementation details.
Revised: Finally, we provide additional IBP-MonDEQ-speciﬁc implementation details.

Original: For IBP-MonDEQ layers, we set m “ 0 . 1 in DEQ-3 models and m “ 0 . 5 in DEQ-7 models.
Revised: For IBP-MonDEQ layers, we set m “ 0 . 1 in DEQ-3 models and m “ 0 . 5 in DEQ-7 models.

Original: We ﬁnd that m “ 0 .outperforms m “ 0 . 1 for DEQ-7 models despite the reduced expressivity, likely due to optimization difﬁculties.
Revised: We ﬁnd that m “ 0 .outperforms m “ 0 . 1 for DEQ-7 models despite the reduced expressivity, likely due to optimization difﬁculties.

Original: We set γ “ 3 when training models with the more relaxed LBEN parameterization.
Revised: We set γ “ 3 when training models with the more relaxed LBEN parameterization.

Original: For
Revised: For

Original: DEQ-7 models, we also ﬁnd it helpful for optimization to use a warmup schedule on the value of m matching the schedule on (cid:15) , and set
Revised: DEQ-7 models, we also ﬁnd it helpful for optimization to use a warmup schedule on the value of m matching the schedule on (cid:15) , and set

Original: m current “ m ` p 0 .
Revised: m current “ m ` p 0 .

Original: 99 ´ m qp 1 ´ (cid:15) current (cid:15) train q
Revised: 99 ´ m qp 1 ´ (cid:15) current (cid:15) train q

Original: so that the value of m current interpolations between 0.99 and m ă 0 . 99 throughout training.
Revised: so that the value of m current interpolations between 0.99 and m ă 0 . 99 throughout training.

Original: The stopping criterion for the ﬁxed-point solver is } z k ` 1 ´ z k } 2 } z k ` 1 }ď t , where t “ 1e-5 is the tolerance threshold, and z k denotes the k -th iterate of the ﬁxed-point solving method.
Revised: The stopping criterion for the ﬁxed-point solver is } z k ` 1 ´ z k } 2 } z k ` 1 }ď t , where t “ 1e-5 is the tolerance threshold, and z k denotes the k -th iterate of the ﬁxed-point solving method.

Original: For models with the LBEN parameterization, we tie the trainable scaling parameter β across channels, so that β has the same shape as the convolutional bias parameter.
Revised: For models with the LBEN parameterization, we tie the trainable scaling parameter β across channels, so that β has the same shape as the convolutional bias parameter.

Original: Details for Figure 2.
Revised: Details for Figure 2.

Original: The models trained in Figure 2 (left) are obtained by taking the 7L AYER model and adding additional C ONV -128-3 x 3-1 layers to reach the desired depth.
Revised: The models trained in Figure 2 (left) are obtained by taking the 7L AYER model and adding additional C ONV -128-3 x 3-1 layers to reach the desired depth.

Original: All other training details for IBP models are the same as above.
Revised: All other training details for IBP models are the same as above.

Original: For the the red curve (training on the standard clean loss), we also use the same implementation details except (cid:15) “ 0 throughout training and testing.
Revised: For the the red curve (training on the standard clean loss), we also use the same implementation details except (cid:15) “ 0 throughout training and testing.

Original: For Figure 2 (right), the bounds are computed for randomly initialized models.
Revised: For Figure 2 (right), the bounds are computed for randomly initialized models.

Original: We obtain the bounds by attempting to solve (3.3) for DEQ models where W corresponds to a fully-connected layer with output dimension 128.
Revised: We obtain the bounds by attempting to solve (3.3) for DEQ models where W corresponds to a fully-connected layer with output dimension 128.

Original: We obtain these bounds on MNIST with (cid:15) “ 0 . 1 and average the mean of s z k ´ z k for each k over the entire MNIST test set.
Revised: We obtain these bounds on MNIST with (cid:15) “ 0 . 1 and average the mean of s z k ´ z k for each k over the entire MNIST test set.

Original: We use the damped forward-backward algorithm as the equilibrium solver with damping factor α “ 0 . 

Revised: We use the damped forward-backward algorithm as the equilibrium solver with damping factor α “ 0 .

Original: (see (A.1)).
Revised: (see (A.1)).

Original: We note that the standard equilibrium iteration (2.4) converges for both MonDEQ and IBP-MonDEQ models, whereas (3.3) fails to converge for MonDEQ models only.
Revised: We note that the standard equilibrium iteration (2.4) converges for both MonDEQ and IBP-MonDEQ models, whereas (3.3) fails to converge for MonDEQ models only.