Original: Optimization Inspired Multi-Branch Equilibrium Models
Revised: Optimization Inspired Multi-Branch Equilibrium Models

Original: Anonymous authors Paper under double-blind review

Original: Abstract
Revised: Abstract

Original: Works have shown the strong connections between some implicit models and optimization problems.
Revised: Works have shown the strong connections between some implicit models and optimization problems.

Original: However, explorations on such relationships are limited.
Revised: However, explorations on such relationships are limited.

Original: Most works pay attention to some common mathematical properties, such as sparsity.
Revised: Most works pay attention to some common mathematical properties, such as sparsity.

Original: In this work, we propose a new type of implicit model inspired by the designing of the systems’ hidden objective functions, called the Multi-branch Optimization induced Equilibrium networks (MOptEqs).
Revised: In this work, we propose a new type of implicit model inspired by the designing of the systems’ hidden objective functions, called the Multi-branch Optimization induced Equilibrium networks (MOptEqs).

Original: The model architecture is designed based on modelling the hidden objective function for the multi-resolution recognition task.
Revised: The model architecture is designed based on modelling the hidden objective function for the multi-resolution recognition task.

Original: Furthermore, we also propose a new training strategy inspired by our understandings of the hidden objective function.
Revised: Furthermore, we also propose a new training strategy inspired by our understandings of the hidden objective function.

Original: In this manner, the proposed model can better utilize the hierarchical patterns for recognition tasks and retain the abilities for interpreting the whole structure as trying to obtain the minima of the problem’s goal.
Revised: In this manner, the proposed model can better utilize the hierarchical patterns for recognition tasks and retain the abilities for interpreting the whole structure as trying to obtain the minima of the problem’s goal.

Original: Comparing with the state-of-the-art models, our MOptEqs not only enjoys better explainability but are also superior to MDEQ with less parameter consumption and better performance on practical tasks.
Revised: Comparing with the state-of-the-art models, our MOptEqs not only enjoys better explainability but are also superior to MDEQ with less parameter consumption and better performance on practical tasks.

Original: Furthermore, we also implement various experiments to demonstrate the eﬀectiveness of our new methods and explore the applicability of the model’s hidden objective function.
Revised: Furthermore, we also implement various experiments to demonstrate the eﬀectiveness of our new methods and explore the applicability of the model’s hidden objective function.

Original:  Introduction
Revised:  Introduction

Original: Whereas Deep Neural Networks (DNNs) have achieved great success in many real-world tasks such as computer vision and neural language process, the limited interpretability of
Revised: Whereas Deep Neural Networks (DNNs) have achieved great success in many real-world tasks such as computer vision and neural language process, the limited interpretability of

Original: DNNs greatly hinders their further development.
Revised: DNNs greatly hinders their further development.

Original: However, many traditional machine learning methods, e.g.,matrix recovery (Zhang et al., 2018b; 2015; Liu & Li, 2016), subspace clustering (You et al., 2016), image deblurring (Liu et al., 2014) and so on, can be interpreted into minimizing the following objective functions:
Revised: However, many traditional machine learning methods, e g.,matrix recovery (Zhang et al., 2018b; 2015; Liu & Li, 2016), subspace clustering (You et al., 2016) image deblurring (Liu et al., 2014) and so on, can be interpreted into minimizing the following objective functions:

Original: [Equation]
Revised: [Equation]

Original:  where A ∈ R m × d 1 , B ∈ R m × d 2 , X ∈ R m × n , and f ( · ) and g ( · ) are convex functions designed by the modelling of their properties.
Revised: AZ + BE . where A ∈ R m × d 1 , B ∈ R m × d 2 , X ∈ R m × n , and f ( · ) and g ( · ) are convex functions designed by the modelling of their properties.

Original: For this count, the interpretability of such methods is much better than DNNs.
Revised: For this count, the interpretability of such methods is much better than DNNs.

Original: Furthermore, these methods can also enjoy state-of-the-art and robust performance on these tasks.
Revised: Furthermore, these methods can also enjoy state-of-the-art and robust performance on these tasks.

Original: We call such interpretability these methods enjoy as "mathematical interpretability", i.e., whether the whole network structure can be summarized as a compact mathematical model that can be mathematically analyzed.
Revised: We call such interpretability these methods enjoy as "mathematical interpretability", i.e., whether the whole network structure can be summarized as a compact mathematical model that can be mathematically analyzed.

Original: Except for these methods, the Optimization Induced Equilibrium Networks (OptEqs) pro- posed by Xie et al. (2021) recover its whole systemfor recognition to an optimization problem.
Revised: Except for these methods, the Optimization Induced Equilibrium Networks (OptEqs) proposed by Xie et al. (2021) recover its whole system to an optimization problem.

Original: The forward propagation for OptEqs tries to solve Eqn.1 to get output y , z ∗ ∈ R d for input x ∈ R d in .
Revised: The forward propagation for OptEqs tries to solve Eqn (1) to get output y , z ∗ ∈ R d for input x ∈ R d in .

Original: We call the front part of Eqn.1 as OptEqs’ equilibrium equation, which is the central part for equilibrium models.
Revised: We call the front part of Eqn (1) as OptEqs’ equilibrium equation, which is the central part for equilibrium models.

Original: And its forward procedure can be regarded as solving the hidden objective functions shown as Eqn.2, [Equation]
Revised: And its forward procedure can be regarded as solving the hidden objective functions shown as Eqn(2),

Original: [Equation]
Revised: [Equation]

Original: W , U , W c are learnable weights, g is the convolution layers projecting input x from real-world domain to feature space like other neural architectures, and f is the penalty for the outputs.
Revised: W , U , W c are learnable weights, g is the convolution layers projecting input x from realworld domain to feature space like other neural architectures, and f is the outputs’ penalty.

Original: Although the interpretabilityfor OptEqs is good, it still remains some weaknesses worth exploring.
Revised: Although OptEqs’ interpretability is satisfying, it still remains some weaknesses worth exploring.

Original: First, although OptEqs performs better than some implicit models, these models only consider a single view (or resolution) of the input in their implicit parts.
Revised: First, although OptEqs performs better than some implicit models, these models only consider a single view (or resolution) of the input in their implicit parts.

Original: We call these models single-view implicit models in the following.
Revised: We call these models single-view implicit models in the following.

Original: However, nearly all the state-of-the-art pattern recognition systems (Lee et al., 2009b; Wang et al., 2019a; Huang et al., 2017; He et al., 2016; Burt & Adelson, 1987) beneﬁt from the multi-layer or multi-resolution feature extractors in domains like computer vision and audio processing.
Revised: However, nearly all the state-of-the-art pattern recognition systems (Lee et al., 2009b; Wang et al., 2019a; Huang et al., 2017; He et al., 2016; Burt & Adelson, 1987) beneﬁt from the multi-layer or multi-resolution feature extractors in domains like computer vision and audio processing.

Original: Secondly,  new modulesinduced by the objective function they proposed are limited, which mainly consider the math properties ofthe output features like sparsity, but whether these properties can beneﬁt image recognition tasks is unexplored.
Revised: Secondly, limited new modules are proposed in their work, which mainly consider the math properties of features like sparsity, but whether these properties can beneﬁt image recognition tasks is unexplored.

Original: Besides OptEqs, MDEQ (Bai et al., 2020) constructs its equilibrium equation z ∗ =
Revised: Besides OptEqs, MDEQ (Bai et al., 2020) constructs its equilibrium equation z ∗ =

Original: F ( z ∗ ; x ) and block F with the inspiration of explicit models especially HRNet Wang et al. (2019a).
Revised: F ( z ∗ ; x ) and block F with the inspiration of explicit models especially HRNet Wang et al. (2019a).

Original: Then they solve the equilibrium equation by the accelerated algorithm for output z ∗ .
Revised: Then they solve the equilibrium equation by the accelerated algorithm for output z ∗ .

Original: Although it shows the state-of-the-art performance with eﬃcient memory cost as other implicit models, such strategies will make the model lose "interpretability" since the MDEQ is a black box because its output is solved in an implicit way (root-ﬁnding method), which makes analysis on its features of intermediate layers almost impossible.
Revised: Although it shows the state-of-the-art performance with eﬃcient memory cost as other implicit models, such strategies will make the model lose "interpretability" since the MDEQ is a black box because its output is solved in an implicit way (root-ﬁnding method), which makes analysis on its features of intermediate layers almost impossible.

Original: Furthermore, its complicated structure also hinders its analysis from mathematical systems.
Revised: Furthermore, its complicated structure also hinders its analysis from mathematical systems.

Original: More eﬃcient blocks with good interpretability still needs exploring.
Revised: More eﬃcient blocks with good interpretability still needs exploring.

Original: Motivated by the limitations of the above models, we would like to design a multi-scale DEQ structure with state-of-the-art performance and preserve mathematical interpretability from the inspiration of OptEqs and multi-scale models.
Revised: Motivated by the limitations of the above models, we would like to design a multi-scale DEQ structure with state-of-the-art performance and preserve mathematical interpretability from the inspiration of OptEqs and multi-scale models.

Original: The contributionsfor our work are listed below:
Revised: Our contributions are listed below:

Original: • We propose a multi-branch implicit model, called Multi-branch OptEqs (MOptEqs), which eﬃciently utilizes diﬀerent scales inputs with better performance and smaller model size.
Revised: • We propose a multi-branch implicit model, called Multi-branch OptEqs (MOptEqs), which eﬃciently utilizes diﬀerent scales inputs with better performance and smaller model size.

Original: Furthermore, it still retains its connection to an optimization problem.
Revised: Furthermore, it still retains its connection to an optimization problem.

Original: • We propose some properties modelled as new terms for the model’s hidden objective function from our analysis on the relationships between branches and their hierarchical dependencies.
Revised: • We propose some properties modelled as new terms for the model’s hidden objective function from our analysis on the relationships between branches and their hierarchical dependencies.

Original: With new problem’s formulation, we obtain the fusion module in our MOptEqs, called Hierarchical Heritage and Diversity Module (HH & D Module).
Revised: With new problem’s formulation, we obtain the fusion module in our MOptEqs, called Hierarchical Heritage and Diversity Module (HH & D Module).

Original: • Apart from the new module design, we also propose the Perturbation Enhanced training strategy (PE) for our MOptEqs from our analysis on the model’s hidden objective function.
Revised: • Apart from the new module design, we also propose the Perturbation Enhanced training strategy (PE) for our MOptEqs from our analysis on the model’s hidden objective function.

Original: The new strategy cannot only enhance the performance of our MOptEqs, but also improve the robustness of our models.
Revised: The new strategy cannot only enhance the performance of our MOptEqs, but also improve the robustness of our models.

Original: 1. Related Works
Revised: 1. Related Works

Original: Implicit Models.
Revised: Implicit Models.

Original: Nearly all modern deep learning approaches use explicit models, which provide an explicit computation graph for the forward propagation.
Revised: Nearly all modern deep learning approaches use explicit models, which provide an explicit computation graph for the forward propagation.

Original: In contrast, the computation graphs for implicit models are "ﬂexible" or can be assumed as having "inﬁnite" depth.
Revised: In contrast, the computation graphs for implicit models are "ﬂexible" or can be assumed as having "inﬁnite" depth.

Original: For example, Neural ODEs (Chen et al., 2018; Massaroli et al., 2020) encode their neural architectures by a diﬀerential system with learnable parameters.
Revised: For example, Neural ODEs (Chen et al., 2018; Massaroli et al., 2020) encode their neural architectures by a diﬀerential system with learnable parameters.

Original: Then the implicit ODE solvers they used is equivalent to a continuous ResNet taking inﬁnitesimal steps.
Revised: Then the implicit ODE solvers they used is equivalent to a continuous ResNet taking inﬁnitesimal steps.

Original: Furthermore, the training process of Neural ODEs can be depicted as ﬁnding a diﬀerential system of a certain type (like heat equations) by updating its learnable parameters which demonstrate that Neural ODE also enjoy interpretability to a certain extent.
Revised: Furthermore, the training process of Neural ODEs can be depicted as ﬁnding a diﬀerential system of a certain type (like heat equations) by updating its learnable parameters which demonstrate that Neural ODE also enjoy interpretability to a certain extent.

Original: Moreover, DEQ (Bai et al., 2019; Winston & Kolter, 2020) is another class of implicit models.
Revised: Moreover, DEQ (Bai et al., 2019; Winston & Kolter, 2020) is another class of implicit models.

Original: The central part of DEQ is the designation of the equilibrium equations z ∗ =
Revised: The central part of DEQ is the designation of the equilibrium equations z ∗ =

Original: F ( z ∗ ; x ) and block F .
Revised: F ( z ∗ ; x ) and block F .

Original: Its forward procedure is trying to solve the equilibrium equations with input x to get the equilibrium state z ∗ as output by accelerated algorithms.
Revised: Its forward procedure is trying to solve the equilibrium equations with input x to get the equilibrium state z ∗ as output by accelerated algorithms.

Original: Since the z ∗ is ﬁxed given F and x , its inference can be regarded as forwarding an explicit network stacked by the same block F for inﬁnite times.
Revised: Since the z ∗ is ﬁxed given F and x , its inference can be regarded as forwarding an explicit network stacked by the same block F for inﬁnite times.

Original: For example, DEQ chooses their F by one set of Conv+ReLU while MDEQ constructs an HRNet-like block as theirs.
Revised: For example, DEQ chooses their F by one set of Conv+ReLU while MDEQ constructs an HRNet-like block as theirs.

Original: However, no evidence shows that the best block constructions in explicit models can perform the best in the DEQ scheme.
Revised: However, no evidence shows that the best block constructions in explicit models can perform the best in the DEQ scheme.

Original: The construction of DEQ blocks or equilibrium equations is an open question worth exploring.
Revised: The construction of DEQ blocks or equilibrium equations is an open question worth exploring.

Original: Furthermore, most DEQ blocks do not have any mathematical insights (like the diﬀusion process in Neural ODE) and perform totally a black box with limited interpretability.
Revised: Furthermore, most DEQ blocks do not have any mathematical insights (like the diﬀusion process in Neural ODE) and perform totally a black box with limited interpretability.

Original: Since implicit models usually adopt accelerated root-ﬁnding algorithms for their forward outputs and backward gradients, they enjoy the advantages of constant and more eﬃcient memory costs compared with DNNs.
Revised: Since implicit models usually adopt accelerated root-ﬁnding algorithms for their forward outputs and backward gradients, they enjoy the advantages of constant and more eﬃcient memory costs compared with DNNs.

Original: Due to the above advantages, the design of implicit models draws much attention these days (Ghaoui et al., 2019; Gould et al., 2019).
Revised: Due to the above advantages, the design of implicit models draws much attention these days (Ghaoui et al., 2019; Gould et al., 2019).

Original: Apart from the models illustrated above, many kinds of other implicit models have been proposed, including diﬀerentiable physics engines (Qiao et al., 2020; de Avila Belbute-Peres et al., 2018), logical structure learning (Wang et al., 2019b) and implicit neural blocks (Li et al., 2020).
Revised: Apart from the models illustrated above, many kinds of other implicit models are proposed, including diﬀerentiable physics engines (Qiao et al., 2020; de Avila Belbute-Peres et al., 2018), logical structure learning (Wang et al., 2019b) and implicit neural blocks (Li et al., 2020).

Original: Model Interpretaiblity.
Revised: Model Interpretaiblity.

Original: Many researchers nowadays are trying to make their proposed method more interpretable.
Revised: Many researchers nowadays are trying to make their proposed method more interpretable.

Original: Although there are many kinds of approaches to achieve this goal, we marginally divide them into two parts: "Empirical" and "Mathematical" interpretability.
Revised: Although there are many kinds of approaches to achieve this goal, we marginally divide them into two parts: "Empirical" and "Mathematical" interpretability.

Original: Many works, such as (Zhang et al., 2018a; Zhang & Zhu, 2018;
Revised: Many works, such as (Zhang et al., 2018a; Zhang & Zhu, 2018;

Original: Bau et al., 2018), attempt to empirically disassemble the black box by characterizing some statistical or structural information like the outputs and gradients of neural networks’ middle layers.
Revised: Bau et al., 2018), attempt to empirically disassemble the black box by characterizing some statistical or structural information like the outputs and gradients of neural networks’ middle layers.

Original: However, these works cannot be directly implemented on DEQ models since implicit ones do not have explicit depths like DNNs, which makes analyzing such models by dissecting each layer’s behavior almost impossible.
Revised: However, these works cannot be directly implemented on DEQ models since implicit ones do not have explicit depths like DNNs, which makes analyzing such models by dissecting each layer’s behavior almost impossible.

Original: Apart from that, researchers (Djolonga & Krause, 2017; Amos & Kolter, 2017; Xie et al., 2019; Chan et al., 2020) also managed to understand the neural architecture by linking it to a mathematical problem.
Revised: Apart from that, researchers (Djolonga & Krause, 2017; Amos & Kolter, 2017; Xie et al., 2019; Chan et al., 2020) also managed to understand the neural architecture by linking it to a mathematical problem.

Original: In this way, researchers can analyze the black box by its corresponding mathematical problem.
Revised: In this way, researchers can analyze the black box by its corresponding mathematical problem.

Original: Furthermore, new components can be proposed due to the analysis of these problems.
Revised: Furthermore, new components can be proposed due to the analysis of these problems.

Original: Our model aims to design a new DEQ architecture with proper mathematical interpretability.
Revised: Our model aims to design a new DEQ architecture with proper mathematical interpretability.

Original: Compared withthe former works, our model can deal with the classiﬁcation tasks withbetter performances with novel components.
Revised: Compared with former works, our model can perform better on the classiﬁcation tasks with novel components.

Original:  Multi-Branch Optimized Induced Equilirium Models 2.The proposed architecture for the Multi-Branch OptEqs 
Revised:  Multi-Branch Optimized Induced Equilirium Models 2.

Revised: The proposed architecture for the Multi-Branch OptEqs

Original: Inspired by OptEqs, we ﬁrst design an objective function for our tasks to solve and then use its ﬁrst-order stationary conditions as MOptEqs’ equilibrium equation.
Revised: Inspired by OptEqs, we ﬁrst design an objective function for our tasks to solve and then use its ﬁrst-order stationary conditions as MOptEqs’ equilibrium equation.

Original: The base of our function is formed by summarizing several objective functions of diﬀerent scales deﬁned for OptEqs (Details are stated in Appendix.A.1).
Revised: The base of our function is formed by summarizing several objective functions of diﬀerent scales deﬁned for OptEqs (Details are stated in Appendix A.1).

Original: However, such a model can not obtain satisfying results since these branches are independent.
Revised: However, such a model can not obtain satisfying results since these branches are independent.

Original: For this count, we need to design some terms describing the dependencies of each branch in the hidden objective function, as shown in the following paragraphs.
Revised: For this count, we need to design some terms describing the dependencies of each branch in the hidden objective function, as shown in the following paragraphs.

Original: Then we can obtain the equilibrium equations for our
Revised: Then we can obtain the equilibrium equations for our

Original: MOptEqs and our MOptEqs’ block F by analyzing the ﬁrst-order stationary condition of our designed optimization problem.
Revised: MOptEqs and our MOptEqs’ block F by analyzing the ﬁrst-order stationary condition of our designed optimization problem.

Original: Hierarchical Heritage Modeling.
Revised: Hierarchical Heritage Modeling.

Original: State-of-the-art explicit models for image tasks are explicitly structured into sequential stages and process diﬀerent resolutions hierarchically (He et al., 2016;
Revised: State-of-the-art explicit models for image tasks are explicitly structured into sequential stages and process diﬀerent resolutions hierarchically (He et al., 2016;

Original: Shelhamer et al., 2017; Lee et al., 2009a), which implies that features should be extracted hierarchically.
Revised: Shelhamer et al., 2017; Lee et al., 2009a), which implies that features should be extracted hierarchically.

Original: In other words, the posterior branches should inherit from their prior ones, which we call such property hierarchical heritage.
Revised: In other words, the posterior branches should inherit from their prior ones, which we call such property hierarchical heritage.

Original: In our work, we are going to formulate such correlation of neighbouring branches in the hidden objective function by adding the following inner-product term into its origin:
Revised: In our work, we are going to formulate such correlation of neighbouring branches in the hidden objective function by adding the following inner-product term into its origin:

Original: [Equation]
Revised: [Equation]

Original: where z i ∈ R
Revised: where z i ∈ R

Original: d
Revised: d

Original: i , z j ∈ R d j , P i : j ∈ R s i × s j denotes the Average Downsample (its transpose can be regarded as a weighted nearest upsample) or Identity matrix suited to the shape of z i and z j
Revised: i , z j ∈ R d j , P i : j ∈ R s i × s j denotes the Average Downsample (its transpose can be regarded as a weighted nearest upsample) or Identity matrix suited to the shape of z i and z j

Original: ( i < j or j = 1 , i = L ).
Revised: ( i < j or j = 1 , i = L ).

Original: This term only estimate the summation forthe similarity of i -th and j -th branch’s channels with the same channel index, which we call them corresponding channels , as shown in the above equation.
Revised: This term estimates the summation for similarity of i -th and j -th branchs channels with the same channel index ( corresponding channels ).

Original: When z i and z j are similar, the inner product will be large.
Revised: When z i and z j are similar, the inner product will be large.

Original: Otherwise, the result will become small, which implies that the relationship of corresponding channels is weak.
Revised: Otherwise, the result will become small, which implies that the relationship of corresponding channels is weak.

Original: In this way, we can ensure the similarity of corresponding channels of the near branches.
Revised: In this way, we can ensure the similarity of corresponding channels of the near branches.

Original: Except for the relations between diﬀerent branches’ corresponding channels, the relationships between dis-corresponding channels are explored in the following section.
Revised: Except for the relations between diﬀerent branches’ corresponding channels, the relationships between dis-corresponding channels are explored in the following section.

Original: Diversity Modeling.
Revised: Diversity Modeling.

Original: In addition to the hierarchical relations for corresponding channels, works (Pang et al., 2019; Amada et al., 2021) have shown enhancing the diversities between branches can also improve the model’s improvements.
Revised: In addition to the hierarchical relations for corresponding channels, works (Pang et al., 2019; Amada et al., 2021) have shown enhancing the diversities between branches can also improve the model’s improvements.

Original: For this account, we also consider making diﬀerent branches extract various features to improve the representation abilities of diﬀerent branches.
Revised: Therefore, we also consider making diﬀerent branches extract various features to improve the representation abilities.

Original: To achieve such a goal, we add a diversity term in the objective function Eqn.(4) if we assume that i < j ,
Revised: To achieve such a goal, we add a diversity term in the objective function Eqn (4) for i < j ,

Original: [Equation]
Revised: [Equation]

Original: where vec − 1 is the inverse vectorization operator convert the vectorized feature z i ∈R
Revised: where vec − 1 is the inverse vectorization operator converting the vectorized feature z i ∈

Original:  d i ×to a matrix in R diC × C with C channels and vec − 1 ( z i ) ( k ) ∈ R diC × 1 denotes the k -th channel of i -th branch in practice.
Revised: R d i × 1 to a matrix in R diC × C with C channels and vec − 1 ( z i ) ( k ) ∈ R diC × 1 denotes the kth channel of i -th branch in practice.

Original: Since we have already considered the relationships between the corresponding channels in the hierarchical term, we only tries to estimate the diversities of dis-corresponding channels of the i -th and j -th branches in this term.
Revised: Since we have already considered the relationships between the corresponding channels in the hierarchical term, we only tries to estimate the diversities of dis-corresponding channels of the i -th and j -th branches in this term.

Original: As the diversity term goes small, the diversities between the branches become stronger.
Revised: As the diversity term goes small, the diversities between the branches become stronger.

Original: The architecture of MOptEqs and its hidden problem.
Revised: The architecture of MOptEqs and its hidden problem.

Original: With the two terms we proposed, we can reformulate the hidden objective for our problemby replacing the L 2 norm with our new modelling as follows:
Revised: With the two terms we proposed, we can reformulate the hidden objective for our problem as follows:

Original: [Equation]
Revised: [Equation]

Original: g ( x ) is the input feature for the raw input x , λ > 0 is a hyperparameter and z i ∈ R d i × 1 are the ﬁnal outputs of MOptEqs.
Revised: g ( x ) is the input feature for the raw input x , λ > 0 is a hyperparameter and z i ∈ R d i × 1 are the ﬁnal outputs of MOptEqs.

Original: We can choose f to constrain z i on our demand and will inﬂuence model’s activation function.
Revised: We can choose f to constrain z i on our demand and will inﬂuence model’s activation function.

Original: If we choose f ( x )
Revised: If we choose f ( x )

Original: =
Revised: =

Original: I { x ≥ 0 } to ensure the outputs to be positive, then the activation function is ReLU.
Revised: I { x ≥ 0 } to ensure the outputs to be positive, then the activation function is ReLU.

Original: We set D ( z L , z L +1 )
Revised: We set D ( z L , z L +1 )

Original: =
Revised: =

Original: D ( z L , z 1 ) and H ( z L , z L +1 ) =
Revised: D ( z L , z 1 ) and H ( z L , z L +1 ) =

Original: H ( z L , z 1 ) when i
Revised: H ( z L , z 1 ) when i

Original: =
Revised: =

Original: L to complete the loop.
Revised: L to complete the loop.

Original: The two terms we added tries to make the corresponding channels of diﬀerent branches correlated by maximizing the H term and enhancing the diversities of diﬀerent branches by minimizing D term.
Revised: The two terms we added tries to make the corresponding channels of diﬀerent branches correlated by maximizing the H term and enhancing the diversities of diﬀerent branches by minimizing D term.

Original: We note that H and D are not conﬂict since they are handling diﬀerent pairs of channels for branches.
Revised: We note that H and D are not conﬂict since they are handling diﬀerent pairs of channels for branches.

Original: As the following proposition shows, we ﬁnally get the equilibrium equations (Eqn.(6)) for our MOptEqs by calculating the ﬁrst-order stationary conditions ∇ z i G = 0 for problem G .
Revised: As the following proposition shows, we ﬁnally get the equilibrium equations (Eqn (6)) for our MOptEqs by calculating the ﬁrst-order stationary conditions ∇ z i G = 0 for problem G .

Original: Proposition 

Revised: Proposition

Original: The proposed multi-branch structure induced by Eqn(5) can be depicted as solving the equilibrium points (cid:101) z ∗ :=
Revised: The proposed multi-branch structure induced by Eqn (5) can be depicted as solving the equilibrium points e z ∗ :=

Original: [ z (cid:62)∗ 1 , ..., z (cid:62)∗ L ] (cid:62) ∈ R(cid:80)
Revised: [ z ⊤∗ 1 , ..., z ⊤∗ L ] ⊤ ∈ R

Original:  Li =1 d i for the following equations:
Revised: ∑ Li =1 d i for the following equations:

Original: [Equation]
Revised: [Equation]

Original: where (cid:102) W =  W.
Revised: where f W =

Original: [Equation]
Revised: [Equation]

Original: W

Original: [Equation]

Original: [Equation]

Original: And h (˜ z ) =
Revised: And h (˜ z ) =

Original: [ h 1 ( z 1 , z L , z 2 ) 
Revised: [ h 1 ( z 1 , z L , z 2 ) ⊤

Original: [Equation]
Revised: [Equation]

Original: , .., h i ( z i , z

Original: i
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 1 , z i +1 ) (cid:62) , .., h L ( z L ,

Original:  zL
Revised: , z

Original: [Equation]
Revised: [Equation]

Original:  1 , z 1 ) (cid:62) ] (cid:62) and each h i is deﬁned as the mapping from R d i × R d
Revised: ⊤ , .., h L ( z L , z L − 1 , z 1 ) ⊤ ] ⊤ and each h i is deﬁned as the mapping from R d i × R d

Original: i +1 ×
Revised: i +1 ×

Original: R d i − 1 to R d i , [Equation]
Revised: R d i − 1 to R d i , h ( z i , z

Original: [Table]
Revised: i − 1 , z i +1 ) =

Original: ∈ ℝ (cid:3031) (cid:3284) 𝑧 (cid:2919)(cid:2878)(cid:2869) ′ 𝑧 (cid:2919) ′𝑧 (cid:2919)(cid:2879)(cid:2869) ′ (cid:2883) SIGN 𝑧 (cid:2919) ′𝑧 (cid:2919)(cid:2878)(cid:2869) ′ (cid:2883){+12( P i : i +
 z i +
+}
Revised: {+12( P i : i +
 z i +
+}

Original: SIGN RD RD 𝑧 (cid:2919) ′ 𝑧 (cid:3043)(cid:3036) ∈ ℝ (cid:3031) (cid:3284) (
Revised: + P ⊤ i

Original: [Figure]
Revised: [Equation]

Original: (cid:3004)×(cid:3009) (cid:3284)(cid:3126)(cid:3117) (cid:3024) (cid:3284)(cid:3126)(cid:3117) ) (ℝ (cid:3004)×(cid:3009) (cid:3284)(cid:3127)(cid:3117) (cid:3024) (cid:3284)(cid:3127)(cid:3117) )
Revised: 1: i z i − 1 )

Original: [Figure]
Revised: [Equation]

Original: −λ −λ
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: The equilibrium points for the above formula are also the ﬁrst-order stationary points for the optimization problem Eqn(5).
Revised: The equilibrium points for the above formula are also the ﬁrst-order stationary points for the optimization problem Eqn(5).

Original: The proof of the proposition is listed in Appendix. A.3.
Revised: The proof of the proposition is listed in Appendix A.3.

Original: In this manner, we obtain the proposed architecture MOptEqs, its forward propagation is equivalent to ﬁnd the equilibrium
Revised: In this manner, we obtain the proposed architecture MOptEqs, its forward propagation is equivalent to ﬁnd the equilibrium

Original: points of Eqn(6), and such process can also be regarded as solving the stationary points of the optimization problem (5).
Revised: points of Eqn(6), and such process can also be regarded as solving the stationary points of the optimization problem (5).

Original: The whole structure is also shown in Figure.1 (a) and we also draw ﬁgures to illustrate the practical process of h i (HH & D) on its right (Figure.1 (b)).
Revised: The whole structure is also shown in Figure 1 (a) and we also draw ﬁgures to illustrate the practical process of h i (HH & D) on its right (Figure 1 (b)).

Original: We also evaluate the eﬀectiveness of such modelling in Section.
Revised: We also evaluate the eﬀectiveness of such modelling in Section.

Original: 3.2’s experiments.
Revised: 3.2’s experiments.

Original: 2. The proposed Training Strategy for MOptEqs
Revised: 2. The proposed Training Strategy for MOptEqs

Original: Researches (Kong et al., 2020; Xie et al., 2020; Tang et al., 2021) have discovered that small perturbations can enhance the generalization abilities and varieties of branches for neural architectures.
Revised: Researches (Kong et al., 2020; Xie et al., 2020; Tang et al., 2021) have discovered that small perturbations can enhance the generalization abilities and varieties of branches for neural architectures.

Original: Most eﬃcient works use adversarial perturbations based on gradients (Xie & Yuille, 2020) for such a trick.
Revised: Most eﬃcient works use adversarial perturbations based on gradients (Xie & Yuille, 2020) for such a trick.

Original: However, such a strategy will slow the training process since they need at least one more back-propagation in each training step for perturbations, which is time-consuming.
Revised: However, such a strategy will slow the training process since they need at least one more back-propagation in each training step for perturbations, which is time-consuming.

Original: Unlike these methods, we can acquire more computation eﬃcient perturbations because our models can be formulated as minimizing an objective function we designed.
Revised: Unlike these methods, we can acquire more computation eﬃcient perturbations because our models can be formulated as minimizing an objective function we designed.

Original: Then we can replace the maximum problem (details in Appendix. A.2) for adversarial perturbation with maximizing our architecture’s hidden objective function G , since we assume MOptEq’s performance will go worse if G ( z ; g ( x )) changes a lot at g ( x ) ’s neighborhoods since the forward propagation of MOptEqs is minimization G ( z ; g ( x )) .
Revised: Then we can replace the maximum problem (details in Appendix A.2) for adversarial perturbation with maximizing our architecture’s hidden objective function G , since we assume MOptEq’s performance will go worse if G ( z ; g ( x )) changes a lot at g ( x ) ’s neighborhoods since the forward propagation of MOptEqs is minimization G ( z ; g ( x )) .

Original: Assumption 1 For a well trained OptEqs with its objective function denoted as G ( z ; g ( x )) and a natural sample x which can be correctly classiﬁed.
Revised: Assumption 1 For a well trained OptEqs with its objective function denoted as G ( z ; g ( x )) and a natural sample x which can be correctly classiﬁed.

Original: If an input perturbation (cid:107) δ (cid:107) ∞ ≤ (cid:15) can cause one of the following changes:
Revised: If an input perturbation ∥ δ ∥ ∞ ≤ ϵ can cause one of the following changes:

Original: [Equation]
Revised: [Equation]

Original: with L 1 , L 2 > 1 , it may lead the OptEqs to the wrong results with high probability.
Revised: with L 1 , L 2 > 1 , it may lead the OptEqs to the wrong results with high probability.

Original: For the second part of the assumption, it’s common sense and widely used in the analysis of adversarial robustness (Zhang et al., 2021; Li et al., 2020).
Revised: For the second part of the assumption, it’s common sense and widely used in the analysis of adversarial robustness (Zhang et al., 2021; Li et al., 2020).

Original: With the above assumption, we can generate the perturbations by maximizing the objective function G ( z ; g ( x )) for our
Revised: With the above assumption, we can generate the perturbations by maximizing the objective function G ( z ; g ( x )) for our

Original: MOptEqs.
Revised: MOptEqs.

Original: Since our MOptEqs can be regarded as a kind of "implicit" ensembling that parallels several equilibrium models, we decide to inject the perturbations acquired for the prior branch to the posterior one, like boosting to enhance the performance.
Revised: Since our MOptEqs can be regarded as a kind of "implicit" ensembling that parallels several equilibrium models, we decide to inject the perturbations acquired for the prior branch to the posterior one, like boosting to enhance the performance.

Original: Then the hidden objective function of MOptEqs with G ( z ; g ( x )) is shown as follows, 
Revised: Then the hidden objective function of MOptEqs with G ( z ; g ( x )) is shown as follows, [Equation] X

Original: [Equation]
Revised: [Equation]

Revised: i

Original: [Table]
Revised: [Equation] D [Equation] i z

Revised: E

Original: [Equation]
Revised: [Equation]

Revised: f

Original: [Table]
Revised: [Equation] [Equation] i z i ) − D

Revised: U i ( g

Original: [Equation]
Revised: [Equation]

Revised: − δ i )

Revised: + b i , W − 1 ⊤ i

Revised: z

Revised: i

Revised: E for i ∈ [1 , L] .

Original: where δ 0 = [-
-]
Revised: where δ 0 = [-
-]0 .

Original: And the problem becomes a bilevel optimization problem.
Revised: And the problem becomes a bilevel optimization problem.

Original: The solution to the lower-level problem is δ i
Revised: The solution to the lower-level problem is δ i

Original: = (cid:15) sign ( U (cid:62) i W − 1 (cid:62) i z i ) .
Revised: = ϵ sign ( U ⊤ i W − 1 ⊤ i z i ) .

Original: Since the perturbation can be obtained by feeding the output of the activation layer σ ( Wz + U g
Revised: Since the perturbation can be obtained by feeding the output of the activation layer σ ( Wz + U g

Original: ( x ) + b ) to the transposed convolution layer with weight
Revised: ( x )+ b ) to the transposed convolution layer with weight

Original: U i , we call it reconstructed perturbation.
Revised: U i , we call it reconstructed perturbation.

Original: Compared with the adversarial perturbations based on gradients, our perturbations can be acquired directly by matrix-vector multiplication instead of iteratively built by gradients.
Revised: Compared with the adversarial perturbations based on gradients, our perturbations can be acquired directly by matrixvector multiplication instead of iteratively built by gradients.

Original: For this account, such a process does not require much computation cost.
Revised: Thus, such a process does not require much computation cost.

Original: Following the above steps, we obtain a harmful perturbed direction for the prior branches and then added them to their posterior branches.
Revised: Following the above steps, we obtain a harmful perturbed direction for the prior branches and then added them to their posterior branches.

Original: We note that our perturbation is added to the input feature g ( x ) instead of the raw input.
Revised: We note that our perturbation is added to the input feature g ( x ) instead of the raw input.

Original: Proposition 2 For a natural sample x and a well trained OptEqs
Revised: Proposition 2 For a natural sample x and a well trained OptEqs

Original: [Equation]
Revised: [Equation]

Original: with its hidden objective function denoted as G ( z ; g ( x )) , for g (cid:48) ( x ) = g ( x ) − δ and δ =(cid:15) sign ( U (cid:62)
Revised: with its hidden objective function denoted as G ( z ; g ( x )) , for g ′ ( x ) = g ( x ) − δ and δ =

Original:  i W − 1 (cid:62) i z i ) , which obeys the constraint (cid:107) δ (cid:107) ∞ ≤ (cid:15) .
Revised: ϵ sign ( U ⊤ i W − 1 ⊤ i z i ) , which obeys the constraint ∥ δ ∥ ∞ ≤ ϵ .

Original: z (cid:48)∗ and z ∗ are deﬁned as follows:
Revised: z ′∗ and z ∗ are deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: If the (cid:107) W (cid:107) 2 , (cid:107) U (cid:107) 2 ≤ 1 and 1 N (cid:107) U (cid:62) W − 1 (cid:62) z ∗ (cid:107) 1 (cid:29) (cid:15) , where N is the element number of g ( x ) , then 
Revised: If the ∥ W ∥ 2 , ∥ U ∥ 2 ≤ 1 and 1 N ∥ U ⊤ W − 1 ⊤ z ∗ ∥ 1 ≫ ϵ , where N is the element number of g ( x ) , then max

Original: [Equation]
Revised: [Equation]

Original: The proposition demonstrates if we choose the perturbed direction to be (cid:15) sign ( U (cid:62) W − 1 (cid:62) z ) , at least one of the changes for the G or z ∗ will be around (cid:15) (cid:107) U (cid:62) W − 1 (cid:62) z ∗ (cid:107) 1 and implies each branch may perform bad on the perturbed data according to our assumptions.
Revised: The proposition demonstrates if we choose the perturbed direction to be ϵ sign ( U ⊤ W − 1 ⊤ z ) , at least one of the changes for the G or z ∗ will be around ϵ ∥ U ⊤ W − 1 ⊤ z ∗ ∥ 1 and implies each branch may perform bad on the perturbed data according to our assumptions.

Original: With the experiments in Section.3.2, we can conclude that our reconstructed perturbation is useful and our assumption is reasonable.
Revised: With the experiments in Section.3.2, we can conclude that our reconstructed perturbation is useful and our assumption is reasonable.

Original: Like boosting strategy, we can feed the perturbed data for the prior branch to its posterior branch to enhance the performance; we call such method as the Perturbation Enhanced training strategy (PE).
Revised: Like boosting strategy, we can feed the perturbed data for the prior branch to its posterior branch to enhance the performance; we call such method as the Perturbation Enhanced training strategy (PE).

Original: Following experiments also show that PE can indeed enhance the performance of our MOptEqs.
Revised: Following experiments also show that PE can indeed enhance the performance of our MOptEqs.

Original: 2. Model Optimization and Forward Convergence
Revised: 2. Model Optimization and Forward Convergence

Original: Forward Propagation and Implicit Diﬀerentiation.
Revised: Forward Propagation and Implicit Diﬀerentiation.

Original: Like other equilibrium models, the forward propagation procedure is solving the given equilibrium functions Eqn. 7.
Revised: Like other equilibrium models, the forward propagation procedure is solving the given equilibrium functions Eqn 7.

Original: For this count, our model also enjoys the constant memory cost advantages as other DEQs.
Revised: For this count, our model also enjoys the constant memory cost advantages as other DEQs.

Original: In our work, run root-ﬁnding algorithms to solve the roots ˜
Revised: In our work, run root-ﬁnding algorithms to solve the roots ˜

Original: z ∗ =
Revised: z ∗ =

Original: [ z ∗ 1 , ...
Revised: [ z ∗ 1 , ...

Original: z ∗ L ] of the following problem to reach the equilibrium states, which is the same as MDEQ (shown in Appendix. A.5):
Revised: z ∗ L ] of the following problem to reach the equilibrium states, which is the same as MDEQ (shown in Appendix A.

Original: [Equation]
Revised: [Equation]

Original: As for the backward propagation, we also adopt the implicit diﬀerentiation method widely used in (Bai et al., 2020; 2019; Chen et al., 2018).
Revised: As for the backward propagation, we also adopt the implicit diﬀerentiation method widely used in (Bai et al., 2020; 2019; Chen et al., 2018).

Original: Instead of tracing the gradients during the forward propagation, the implicit diﬀerentiation method directly backpropagates through the equilibrium state using the Jacobian of T θ = ∇ ˜
Revised: Instead of tracing the gradients during the forward propagation, the implicit diﬀerentiation method directly backpropagates through the equilibrium state using the Jacobian of T θ = ∇ ˜

Original: z G + ˜
Revised: z G +˜

Original: z
Revised: z

Original: at ˜ z ∗ .
Revised: at ˜ z ∗ .

Original: For a given loss l = L (˜ z ∗ , y ) (where y is the target) and the gradients can be written as
Revised: For a given loss l = L (˜ z ∗ , y ) (where y is the target) and the gradients can be written as

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: And the calculation of ∂l∂ ˜ z ∗ ( I − J T θ | ˜
Revised: And the calculation of ∂l∂ ˜ z ∗ ( I − J T θ | ˜

Original: z ∗ ) − 1 is equivalent to solve root m for the following equation:
Revised: z ∗ ) − 1 is equivalent to solve root m for the following equation:

Original: [Equation]
Revised: [Equation]

Original: As for the root-ﬁnding algorithm [Table] we can use Broyden method (Broyden, 1965), Anderson
Revised: As for the root-ﬁnding algorithm, we can use Broyden method (Broyden, 1965), Anderson

Original: Method (Anderson, 1965; Bai et al., 2021) or other root-ﬁnding methods to solve Eqn(7) for the equilibrium state and Eqn(9) for the backward gradient.
Revised: Method (Anderson, 1965; Bai et al., 2021) or other root -ﬁnding methods to solve Eqn(7) for the equilibrium state and Eqn(9) for the backward gradient.

Original: Forward Convergence.
Revised: Forward Convergence.

Original: Like other implicit models, weneed to make some constraints on parameters in order to make the whole MOptEqs T Θ (˜ z ; x ) ( ˜ z here is { z i } Li =1 ) be a contractive mapping.
Revised: Like other implicit models, we make some constraints on parameters in order to make the whole MOptEqs T Θ (˜ z ; x ) ( ˜ z here is { z i } Li =1 ) be a contractive mapping.

Original: For the MOptEqs without considering HH & D (Eqn.10), the models can easily converge with (cid:107) W i (cid:107) 2 ≤ ζ < 1 .
Revised: For the MOptEqs without considering HH & D (Eqn 10), the models can easily converge with ∥ W i ∥ 2 ≤ ζ < 1 .

Original: But since we use the HH & D module, we need to choose proper λ to ensure the convergence, around or less than 1 C will be appropriate, which does not need ζ to be too small.
Revised: But since we use the HH & D module, we need to choose proper λ to ensure the convergence, around or less than 1 C will be appropriate.

Original: We also conduct experiments to verify the convergence in Fig. in Appendix. A.7.
Revised: We also conduct experiments to verify the convergence in Appendix A.7.

Original:  Experimental Results
Revised:  Experimental Results

Original: In this section, we conduct experiments for the image classiﬁcation tasks on CIFAR- 10 ,CIFAR100 Krizhevsky et al.
Revised: In this section, we conduct experiments for the image classiﬁcation tasks on CIFAR10 ,CIFAR- 100 Krizhevsky et al.

Original: and ImageNette implemented on the PyTorch (Paszke et al., 2017) platform to demonstrate the eﬀectiveness of our model.
Revised: and ImageNette implemented on the PyTorch (Paszke et al., 2017) platform to demonstrate the eﬀectiveness of our model.

Original: Details are listed in Appendix..A.6.
Revised: Details are listed in Appendix A.6.

Original: 3. Comparison of Prior Implicit Models
Revised: 3. Comparison of Prior Implicit Models

Original: In this part, we decide to verify the superiority of MOptEqs in two aspects.
Revised: In this part, we decide to verify the superiority of MOptEqs in two aspects.

Original: First, we build the single-scale MOptEqs and compare the empirical results with other single-scale implicit models.
Revised: First, we build the single-scale MOptEqs and compare the empirical results with other single-scale implicit models.

Original: And then we compare the experimental results with prior implicit models that utilize the multi-scale inputs like MDEQs, the state-of-the-art implicit model with multi-resolution.
Revised: And then we compare the experimental results with prior implicit models that utilize the multi-scale inputs like MDEQs, the state-of-the-art implicit model with multi-resolution.

Original: Implicit Models with Single View.
Revised: Implicit Models with Single View.

Original: In this part, we compare our MOptEqs with other single-view implicit models with comparable model sizes.
Revised: In this part, we compare our MOptEqs with other single-view implicit models with comparable model sizes.

Original: Like OptEqs who uses three blocks for their experiment, we ﬁrst construct our small MOptEqs with three branches whose outputs share the same size and the channel number C = 32 .
Revised: Like OptEqs who uses three blocks for their experiment, we ﬁrst construct our small MOptEqs with three branches whose outputs share the same size and the channel number C = 32 .

Original: Results in Table.1 (a) demonstrates that our MOptEqs structure can even enhance the performance for recognition tasks under the single-view cases.
Revised: Results in Table. (a) demonstrates that our MOptEqs structure can even enhance the performance for recognition tasks under the single-view cases.

Original: Compared to the multi-branch model without fusion (POptEqs), our MOptEqs’ HH & D can eﬃciently utilize the relationships between diﬀerent branches and lead the model to better performance.
Revised: Compared to the multi-branch model without fusion (POptEqs), our MOptEqs’ HH & D can eﬃciently utilize the relationships between diﬀerent branches and lead the model to better performance.

Original: Our superiority also holds compared with former multi branches fusion methods: "sum" (non-parametric module) and "conv" (parametric module).
Revised: Our superiority also holds compared with former multi branches fusion methods: "sum" (non-parametric module) and "conv" (parametric module).

Original: Meanwhile, we can also conclude the eﬀectiveness of our perturbation enhanced training strategy from the table since its performance is the best.
Revised: Meanwhile, we can also conclude the eﬀectiveness of our perturbation enhanced training strategy from the table since its performance is the best.

Original: [Table]
Revised: [Table]

Original: Implicit Models with Multiple Scales.
Revised: Implicit Models with Multiple Scales.

Original: Moreover, we conduct experiments compared with other models which handling multi-scale inputs.
Revised: Moreover, we conduct experiments compared with other models which handling multi-scale inputs.

Original: Like MDEQ, we construct our MOptEqs with four branches with resolution size equals to 32 , 16 , 8 , 4 .
Revised: Like MDEQ, we construct our MOptEqs with four branches with resolution size equals to 32 , 16 , 8 , 4 .

Original: Other details can be found in Appendix. A.6.
Revised: Other details can be found in Appendix A.6.

Original: From Table.1 (b) and as shown in Table.2 (a), one can see that our MOptEqs not only outperforms the widely used explicit model
Revised: From Table.1 (b) and as shown in Table.2 (a), one can see that our MOptEqs not only outperforms the widely used explicit model

Original: ResNet- 

Revised: ResNet-

Original: (He et al., 2016), but also shows better performance compared with MDEQ with fewer parameters on CIFAR, which is one of the best models.
Revised: (He et al., 2016), but also shows better performance compared with MDEQ with fewer parameters on CIFAR, which is one of the best models.

Original: The empirical results for the multi-scale models further verify the superiority of MOptEqs and its training strategy.
Revised: The empirical results for the multi-scale models further verify the superiority of MOptEqs and its training strategy.

Original: Apart from experiments on small images, we also conduct experiments on ImageNette, which is a subset of 10 classes from ImageNet.
Revised: Apart from experiments on small images, we also conduct experiments on ImageNette, which is a subset of 10 classes from ImageNet.

Original: Compared with MDEQ, our MOptEqs consistently perform better shown in Table.2 (b).
Revised: Compared with MDEQ, our MOptEqs consistently perform better shown in Table.2 (b).

Original: Nevertheless, as one can see from the results, the diﬀerence between MDEQ and our model becomes much bigger in the CIFAR- 100 and
Revised: Nevertheless, as one can see from the results, the diﬀerence between MDEQ and our model becomes much bigger in the CIFAR- 100 and

Original: Imagenette with the same training hyper-parameters.
Revised: Imagenette with the same training hyper-parameters.

Original: Such a phenomenon demonstrates that the performance of our model is much more stable than MDEQ.
Revised: Such a phenomenon demonstrates that the performance of our model is much more stable than MDEQ.

Original: Apart from these experiments, we also conducted ablation studies for our models in the following section.
Revised: Apart from these experiments, we also conducted ablation studies for our models in the following section.

Original:  The comprehensive understanding of MOptEqs
Revised: 3. The comprehensive understanding of MOptEqs

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: Visualization of the impact for the HH & D module.
Revised: Visualization of the impact for the HH & D module.

Original: In this section, we try to estimate the eﬀect of our HH & D module.
Revised: In this section, we try to estimate the eﬀect of our HH & D module.

Original: We ﬁnish experiments in this part using three-branch
Revised: We ﬁnish experiments in this part using three-branch

Original: MOptEqs with 16 channels for each branch to make the visualization more clear.
Revised: MOptEqs with 16 channels for each branch to make the visualization more clear.

Original: We tries to verify our HH & D module’s eﬀect in two aspects.
Revised: We tries to verify our HH & D module’s eﬀect in two aspects.

Original: First, we plot scatter ﬁgures for the values of(cid:80) Li
Revised: First, we plot scatter ﬁgures for the values of

Original: [Equation]
Revised: [Equation]

Original: i
Revised: (denoted as Average D ) and

Original: [Equation] [Equation]
Revised: [Equation] [Figure]

Original: j

Original: [Equation]
Revised: [Equation]

Original: ( C 2 − C ) (denoted as Average D ) and

Original: (cid:80) Li =1 (cid:80) j ∈N ( i ) H

Original: ( z i , z j ) LC (denoted as Average H ) for 100 randomly chosen samples ( N ( i ) denotes neighboring branch of the i -th branch).
Revised: (denoted as Average H ) for 100 randomly chosen samples ( N ( i ) denotes neighboring branch of the i -th branch).

Original: Since the hidden optimization problem is to maximizing H while minimizing D , we ﬁnd that our MOptEqs (Figure.2(a)) can induce the output features to reach such goal compared with POptEqs (MOptEqs without HH & D) (Figure.2(b)).
Revised: Since the hidden optimization problem is to maximizing H while minimizing D , we ﬁnd that our MOptEqs (Figure 2(a)) can induce the output features to reach such goal compared with POptEqs (MOptEqs without HH & D) (Figure 2(b)).

Original: Furthermore, we also plot heatmaps of the ﬁrst two branches’ correlation score ( | vec − 1 ( z 2 ) vec − 1 ( z 1 ) (cid:62) | ∈ R 16 × 16 ) for a randomly chosen sample shown in Figure.2(c)(d).
Revised: Furthermore, we also plot heatmaps of the ﬁrst two branches’ correlation score ( | vec − 1 ( z 2 ) vec − 1 ( z 1 ) ⊤ | ∈ R 16 × 16 ) for a randomly chosen sample shown in Figure 2(c)(d).

Original: One can see that MOptEqs’ heatmap (Fig.2(c)) is more likely to be a diagonal matrix while heatmap Fig.2(d) for POptEqs looks random.
Revised: One can see that MOptEqs’ heatmap (Figure 2(c)) is more likely to be a diagonal matrix while heatmap Figure 2(d) for POptEqs looks random.

Original: The heatmap shows that our HH & D can induce the model to perform as our demand, which means corresponding channels for adjacent branches can be more correlated, while the dis-corresponding channels are unrelated due to our architecture design.
Revised: The heatmap shows that our HH & D can induce the model to perform as our demand, which means corresponding channels for adjacent branches can be more correlated, while the dis-corresponding channels are unrelated due to our architecture design.

Original: The visualizations and former results on diﬀerent datasets verify the eﬀectiveness of our modelling and our HH & D modules’ design.
Revised: The visualizations and former results on diﬀerent datasets verify the eﬀectiveness of our modelling and our HH & D modules’ design.

Original: [Figure]
Revised: [Figure]

Original: Evaluation of the reconstructed perturbation.
Revised: Evaluation of the reconstructed perturbation.

Original: We compare our perturbation with randomly generated ones in a one-branch well-trained MOptEqs to validate its eﬀectiveness.
Revised: We compare our perturbation with randomly generated ones in a one-branch well-trained MOptEqs to validate its eﬀectiveness.

Original: We ﬁrst feed a natural sample x to the model and acquire output z for x and generate our reconstructed perturbation using such output and then feed it to the model.
Revised: We ﬁrst feed a natural sample x to the model and acquire output z for x and generate our reconstructed perturbation using such output and then feed it to the model.

Original: Furthermore, we add the perturbations generated by Binomial distribution ( P ( δ i = (cid:15) ) =
Revised: Furthermore, we add the perturbations generated by Binomial distribution ( P ( δ i = ϵ ) =

Original: P ( δ i = − (cid:15) )
Revised: P ( δ i = − ϵ )

Original: = 0 . 5 ) and Uniform distribution U [ − (cid:15), (cid:15) ] to input features g ( x ) for comparison.
Revised: = 0 . 5 ) and Uniform distribution U [ − ϵ, ϵ ] to input features g ( x ) for comparison.

Original: The result of each distribution is averaged for ﬁve trials.
Revised: The result of each distribution is averaged for ﬁve trials.

Original: Figure.3(a) shows our perturbation generated by maximizing G is much more eﬀective, which validates the eﬀectiveness of our reconstructed perturbations and the rationality of our assumptions and analysis in Section.2.2.
Revised: Figure 3(a) shows our perturbation generated by maximizing G is much more eﬀective, which validates the eﬀectiveness of our reconstructed perturbations and the rationality of our assumptions and analysis in Section.2.2.

Original: Robustness of our MOptEqs trained by PE strategy.
Revised: Robustness of our MOptEqs trained by PE strategy.

Original: In addition to improving the generalization abilities for models as shown in Sec.3.1, perturbations with proper size may also enhance the robustness of our MOptEqs, shown in Figure.3(b).
Revised: In addition to improving the generalization abilities for models as shown in Sec.3.1, perturbations with proper size may also enhance the robustness of our MOptEqs, shown in Figure 3(b).

Original: We conduct experiments on small MOptEqs (the same setting as Sec.3.1’s single-view model).
Revised: We conduct experiments on small MOptEqs (the same setting as Sec.3.1’s single-view model).

Original: Our perturbation enhanced training strategy goes stronger as (cid:15) increases.
Revised: Our perturbation enhanced training strategy goes stronger as ϵ increases.

Original: The ﬁgure shows that accuracy for the model trained naturally drops quicker than trained by PE strategy.
Revised: The ﬁgure shows that accuracy for the model trained naturally drops quicker than trained by PE strategy.

Original: Overall, trained with appropriate reconstructed perturbation can partly improve the robustness of our MOptEqs.
Revised: Overall, trained with appropriate reconstructed perturbation can partly improve the robustness of our MOptEqs.

Original: MOptEqs vs. adding Regularizers in training loss.
Revised: MOptEqs vs. adding Regularizers in training loss.

Original: Apart from our HH & D modules, adding D and H to the training loss is one of the most popular methods for obtaining outputs with certain properties.
Revised: Apart from our HH & D modules, adding D and H to the training loss is one of the most popular methods for obtaining outputs with certain properties.

Original: In order to demonstrate the superiority of our module empirically, we conduct experiments for our MOptEqs trained by cross-entropy and POptEqs (MOptEqs (w/o.
Revised: In order to demonstrate the superiority of our module empirically, we conduct experiments for our MOptEqs trained by cross-entropy and POptEqs (MOptEqs (w/o.

Original: HH & D)) trained by the cross-entropy loss adding γ (cid:80) Li =

Revised: HH & D)) trained by the cross-entropy loss adding γ P Li =

Original: ( H ( z i , z i +1 )
Revised: ( H ( z i , z i +1 )

Original: − λ D ( z i , z i +1 ))
Revised: − λ D ( z i , z i +1 ))

Original: as regularizers (we call it augmented loss).
Revised: as regularizers (we call it augmented loss).

Original: Then we drew its test accuracy for diﬀerent γ trained by the augmented loss ( λ is the same as ours) shown in Figure.3(c).
Revised: Then we drew its test accuracy for diﬀerent γ trained by the augmented loss ( λ is the same as ours) shown in Figure 3(c).

Original: With proper γ , POptEqs trained by additional loss can perform better than γ = 0 .
Revised: With proper γ , POptEqs trained by additional loss can perform better than γ = 0 .

Original: Such phenomenon demonstrates the eﬀectiveness of our HH & D Modelling.
Revised: Such phenomenon demonstrates the eﬀectiveness of our HH & D Modelling.

Original: Furthermore, the ﬁgure demonstrates the superiority of our model since the traditional method is consistently worse than ours.
Revised: Furthermore, the ﬁgure demonstrates the superiority of our model since the traditional method is consistently worse than ours.

Original: We left some other explorations for our model in the Appendix due to the space limit.
Revised: We left some other explorations for our model in the Appendix.

Original:  Conclusion
Revised:  Conclusion

Original: We introduce the multi-branch optimization induced equilibrium models (MOptEqs), a new extension of OptEqs that can utilize multiscale information for recognition tasks and retain its ability to recover to an optimization problem whose solution is equivalent to the equilibrium states of our model.
Revised: We introduce the multi-branch optimization induced equilibrium models (MOptEqs), a new extension of OptEqs that can utilize multiscale information for recognition tasks and retain its ability to recover to an optimization problem whose solution is equivalent to the equilibrium states of our model.

Original: The model architecture is designed based on modelling the hidden objective function for the multi- resolution recognition task.
Revised: The model architecture is designed based on modelling the hidden objective function for the multi-resolution recognition task.

Original: Furthermore, we also propose a new training strategy inspired by our understandings of the hidden optimization problem.
Revised: Furthermore, we also propose a new training strategy inspired by our understandings of the hidden optimization problem.

Original: The empirical results show the advantages of our proposed methods
Revised: The empirical results show the advantages of our proposed methods.

Original: The success of our HH & D module and PE training strategy demonstrates the deep link between the optimization problem and neural architecture and may motivate further explorations.
Revised: The success of our HH & D module and PE training strategy demonstrates the deep link between the optimization problem and neural architecture and may motivate further explorations.

Revised: Acknowledgments

Revised: Yisen Wang is partially supported by the National Natural Science Foundation of China under Grant 62006153, Project 2020BD006 supported by PKU-Baidu Fund, and Open Research Projects of Zhejiang Lab (No. 2022RC0AB05).

Revised: Zhouchen Lin is supported by the NSF China (No. 61731018), NSFC Tianyuan Fund for

Revised: Mathematics (No. 12026606), Project 2020BD006 supported by PKU-Baidu Fund and

Revised: Qualcomm.

Original: A Appendix A. Paralleling Multi-Branch OptEqs
Revised: A Appendix A. Paralleling Multi-Branch OptEqs

Original: Experiments show that even a simple one-layer OptEqs module enjoys impressive generalization abilities.
Revised: Experiments show that even a simple one-layer OptEqs module enjoys impressive generalization abilities.

Original: If we view a simple one-layer OptEqs module as a powerful feature generator, we can use an "ensemble" scheme on the DEQs by paralleling several DEQ modules.
Revised: If we view a simple one-layer OptEqs module as a powerful feature generator, we can use an "ensemble" scheme on the DEQs by paralleling several DEQ modules.

Original: Then we get the multi-branch implicit structure (formulated as Eqn.10), which is capable of utilizing samples of multi-resolution but also retains the capability of recovering to an optimization problem.
Revised: Then we get the multi-branch implicit structure (formulated as Eqn 10), which is capable of utilizing samples of multi-resolution but also retains the capability of recovering to an optimization problem.

Original: [Equation]
Revised: [Equation]

Original: where z ∗ i denotes the equilibrium outputs of i -th branch, we call this structure ParallelOptEqs (POptEqs).
Revised: where z ∗ i denotes the equilibrium outputs of i -th branch, we call this structure ParallelOptEqs (POptEqs).

Original: Like (Xie et al., 2021) stated for OptEqs, the POptEqs can also be depicted as solving the following problem (Eqn.11) if W i is invertible and σ ( · ) is activation function,[Equation]
Revised: Like (Xie et al., 2021) stated for OptEqs, the POptEqs can also be depicted as solving the following problem (Eqn 11) if W i is invertible and σ ( · ) is activation function,

Original: [Equation]
Revised: [Equation]

Original: where g ( x ) is the input feature with raw input x , z i ∈
Revised: where g ( x ) is the input feature with raw input x , z i ∈

Original: R d i × 1 and f added for the activation function.
Revised: R d i × 1 and f added for the activation function.

Original: For example f is the positive orthant f ( x )
Revised: For example f is the positive orthant f ( x )

Original: =
Revised: =

Original: I { x ≥ 0 } if we use ReLU activation, since ( I + ∂f ) − 1 is the ReLU activation function, other activation functions can be found in Li et al.
Revised: I { x ≥ 0 } if we use ReLU activation, since ( I + ∂f ) − 1 is the ReLU activation function, other activation functions can be found in Li et al.

Original: (2019).The calculation of the above structure (Eqn.10) is equivalent to solving the stationary points ∇ z i G = 0 .
Revised: (2019).The calculation of the above structure (Eqn 10) is equivalent to solving the stationary points ∇ z i G = 0 .

Original: A. Adversarial Perturbations.
Revised: A. Adversarial Perturbations.

Original: Adversarial perturbations aim to cheat well-trained neural networks with small or unnoticeable changes on natural images.
Revised: Adversarial perturbations aim to cheat well-trained neural networks with small or unnoticeable changes on natural images.

Original: The adversarial perturbations can be obtained by solving the following maximum problem for input x nat and label y :
Revised: The adversarial perturbations can be obtained by solving the following maximum problem for input x nat and label y :

Original: [Equation]
Revised: [Equation]

Original: where C is the feasible set of δ in case the change is too large and usually chosen to be an l-inﬁnite ball.
Revised: where C is the feasible set of δ in case the change is too large and usually chosen to be an l-inﬁnite ball.

Original: Due to the neural network is a non-convex function for δ , obtaining the perturbation is not easy.
Revised: Due to the neural network is a non-convex function for δ , obtaining the perturbation is not easy.

Original: The most popular method is the PGD- k method which solves the above problem by implementing the projected gradient method by k times.
Revised: The most popular method is the PGD- k method which solves the above problem by implementing the projected gradient method by k times.

Original: Although the performance of such a method is satisfactory in most cases, it makes the whole training procedure k + 1 times slower than the origin.
Revised: Although the performance of such a method is satisfactory in most cases, it makes the whole training procedure k + 1 times slower than the origin.

Original: In our work, we replace the above objective function with our model’s hidden objective function.
Revised: In our work, we replace the above objective function with our model’s hidden objective function.

Original: Since the hidden optimization problem is convex with respect to the perturbation, we can get the perturbation much easier.
Revised: Since the hidden optimization problem is convex with respect to the perturbation, we can get the perturbation much easier.

Original: Moreover, such perturbation can also enhance the generalization abilities, as shown in the experiments.
Revised: Moreover, such perturbation can also enhance the generalization abilities, as shown in the experiments.

Original: A. Proofs for Proposition. 
Revised: A. Proofs for Proposition.

Original: In this section, we are going to prove that the equilibrium equation Eqn.1 is also the ﬁrst order stationary points for the optimization problem 5.
Revised: In this section, we are going to prove that the equilibrium equation Eqn 1 is also the ﬁrst order stationary points for the optimization problem 5.

Original: Proof 1 The equilibrium equations Eqn.1 in the proposition can be separated into diﬀerent branches as follows:
Revised: Proof 1 The equilibrium equations Eqn 1 in the proposition can be separated into diﬀerent branches as follows:

Original: [Equation]
Revised: [Equation]

Original: where h i : R d i × R d i +1 ×
Revised: where h i : R d i × R d i +1 ×

Original: R d i − 1 → R d i can be deﬁned as:
Revised: R d i − 1 → R d i can be deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: First
Revised: First

Original: [Equation]
Revised: [Equation]

Original: we need to prove that
Revised: we need to prove that

Original: [Equation]
Revised: [Equation]

Original: The derivatives for the right term can be divided as the derivatives for H and D :
Revised: The derivatives for the right term can be divided as the derivatives for H and D :

Original: [Equation]
Revised: [Equation]

Original: which is the ﬁrst term of h i .
Revised: which is the ﬁrst term of h i .

Original: Then we are going to calculate the derivatives for D : ∇ z i 
Revised: Then we are going to calculate the derivatives for D : ∇ z i L X

Original: [Equation]
Revised: [Equation]

Original: with the following equation holds for the sampling matrix P i − 1: i and sampling is done for each channel independently:
Revised: with the following equation holds for the sampling matrix P i − 1: i and sampling is done for each channel independently:

Original: [Equation]
Revised: [Equation]

Original: The equation can be formulated as:
Revised: The equation can be formulated as:

Original: [Equation]
Revised: [Equation]

Revised: A

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: In the same manner
Revised: In the same manner

Original: [Equation]
Revised: [Equation]

Original: we can get,
Revised: we can get,

Original: [Equation]
Revised: [Equation]

Original: Taking all the terms together
Revised: Taking all the terms together

Original: [Equation]
Revised: [Equation]

Original: we’ve proved the correctness of Eqn.14.
Revised: we’ve proved the correctness of Eqn 14.

Original: Then we can proved the relationships between Eqn.12 and the ﬁrst-order stationary points for Eqn.5.
Revised: Then we can proved the relationships between Eqn 12 and the ﬁrst-order stationary points for Eqn 5.

Original: Take z i for an example:
Revised: Take z i for an example:

Original: [Equation]
Revised: [Equation]

Original: Then we can get the equilibrium equation:
Revised: Then we can get the equilibrium equation:

Original: [Equation]
Revised: [Equation]

Original: the proof is the same for all i .
Revised: the proof is the same for all i .

Original: In this manner, we proved the relations between the structure and the optimization problems.
Revised: In this manner, we proved the relations between the structure and the optimization problems.

Original: The proof for the proposition is complete
Revised: The proof for the proposition is complete

Original: [Equation]
Revised: [Equation]

Original: A. Proofs for Proposition. 
Revised: A. Proofs for Proposition.

Original: Proof 2 From the preliminaries above, the diﬀerence between G ( z (cid:48)∗ ; g ( x ) − δ ) and G ( z ∗ ; g ( x )) with δ = (cid:15)SIGN ( U (cid:62) W − 1 (cid:62) z ∗ ) is as follows:
Revised: Proof 2 From the preliminaries above, the diﬀerence between G ( z ′∗ ; g ( x ) − δ ) and G ( z ∗ ; g ( x )) with δ = ϵSIGN ( U ⊤ W − 1 ⊤ z ∗ ) is as follows:

Original: [Equation]
Revised: [Equation]

Original: Since z ∗ is the minimizer of G at g ( x )
Revised: Since z ∗ is the minimizer of G at g ( x )

Original: [Equation]
Revised: [Equation]

Original: the above function can be converted as follows:
Revised: the above function can be converted as follows:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: From the structure of OptEqs
Revised: From the structure of OptEqs

Original: [Equation]
Revised: [Equation]

Original: we can get:
Revised: we can get:

Original: [Equation]
Revised: [Equation]

Original: The inequality is acquired because of the Lipshitzness of common activation function (ReLU and Leacky ReLU) and the bounds on weight matrices.
Revised: The inequality is acquired because of the Lipshitzness of common activation function (ReLU and Leacky ReLU) and the bounds on weight matrices.

Original: With the above results, we can get:
Revised: With the above results [Equation] we can get:

Original: [Equation]
Revised: [Equation]

Original: Then we can obtain the proposition:
Revised: Then we can obtain the proposition:

Original: [Equation]
Revised: [Equation]

Original: The proofs are similar when considering the HH & D modules.
Revised: The proofs are similar when considering the HH & D modules.

Original: A. Forward Process of our MOptEqs’ implicit part.
Revised: A. Forward Process of our MOptEqs’ implicit part.

Original: The forward propagation is illustrated as follows:
Revised: The forward propagation is illustrated as follows:

Original: Algorithm 1: Forward Process of our MOptEqs’ implicit part.
Revised: Algorithm 1: Forward Process of our MOptEqs’ implicit part.

Original: Require: Input x , initial points { z i } Li =1 , perturbation size (cid:15) , δ 0{+
+} = 0 , maximum perturbation size (cid:15) .
Revised: Require: Input x , initial points { z i } Li =1 , perturbation size ϵ , δ {+
+} = 0 , maximum perturbation size ϵ .

Original: Ensure: Equilibrium points { z i } Li [-=
-]
Revised: Ensure: Equilibrium points { z i } Li [-=
-]=

Original: Calculate the input feature g ( x ) .
Revised: Calculate the input feature g ( x ) .

Original: The calculation of ∇ ˜
Revised: The calculation of ∇ ˜

Original: z G (˜ z ; g ( x ))
Revised: z G (˜ z ; g ( x ))

Original: = 

Revised: =

Original: is the same as ﬁnding the equilibrium points
Revised: is the same as ﬁnding the equilibrium points

Original: T θ = ∇ ˜ z G + ˜ z =
Revised: [Equation]

Original: ˜ z . Use Broyden (Broyden, 1965) or Anderson Method (Anderson, 1965) or others to ﬁnd roots of the following equation like MDEQ:
Revised: Use Broyden (Broyden, 1965) or Anderson Method (Anderson, 1965) or others to ﬁnd roots of the following equation like MDEQ:

Original: [Equation]
Revised: [Equation]

Original: z return ˜ 
Revised: return ˜ z

Original: z = { z ∗ i } Li =
Revised: = { z ∗ i } Li =

Original: And the calculation of T θ is shown as follows:
Revised: And the calculation of T θ is shown as follows:

Revised: Algorithm

Original: Require:
Revised: Require:

Original: Input feature g ( x ) , initial points { z i } Li =1 , perturbation size (cid:15) , δ 0 = 0 , maximum perturbation size (cid:15) .
Revised: Input feature g ( x ) , initial points { z i } Li =1 , perturbation size ϵ , δ 0 = 0 , maximum perturbation size ϵ .

Original: Ensure: Output T θ = ∇ ˜
Revised: Ensure: Output T θ = ∇ ˜

Original: z G (˜ z ; g ( x ))
Revised: z G (˜ z ; g ( x ))

Original: + ˜ z for i ∈ RANGE (1 , L ) do k 1 , k 2 ← neighbor of i in L -loopCalculate z pi by h
Revised: + ˜ z for i ∈ RANGE (1 , L ) do k 1 , k 2 ← neighbor of i in L -loop

Original:  i (Shown in Fig.1 (b)): z pi ← h
Revised: Calculate z pi by h i (Shown in Figure 1 (b)): z pi ← h

Original: i ( z i , z i − 1 , z i +1 ) if Use PE Method then y i ← σ ( W i z
Revised: i ( z i , z i − 1 , z i +1 ) if Use PE Method then y i ← σ ( W i z

Original: pi
Revised: pi

Original: + U
Revised: + U

Original: i ( g ( x ) − δ
Revised: i ( g ( x ) − δ

Original: i − 1 )
Revised: i − 1 )

Original: + b i ) δ
Revised: + b i ) δ

Original: i ← (cid:15) SIGN ( U (cid:62) i y i ) else
Revised: i ← ϵ SIGN ( U ⊤ i y i ) else

Original: y i ← σ ( W i z pi
Revised: y i ← σ ( W i z pi

Original: + U
Revised: + U

Original: i g ( x ) + b i ) end if z i ← W (cid:62) i
Revised: i g ( x ) + b i ) end if z i ← W ⊤ i

Original: y i end for return { z ∗ i } Li =
Revised: y i end for return { z ∗ i } Li =

Original: A. Experiments details of MOptEqs for the experiments Classification. A.6. CIFAR-
Revised: A. Experiments details of MOptEqs for the experiments Classification. A.6. CIFAR-

Original: For classiﬁcation tasks of single-view models, we set the channel number of each branch to 32 , λ = 0 .
Revised: For classiﬁcation tasks of single-view models, we set the channel number of each branch to 32 , λ = 0 .

Original: 01 and perturbation size (cid:15) = 0 .
Revised: 01 and perturbation size ϵ = 0 .

Original: 1 for the small MOptEqs with three branches for the single-view comparison as OptEqs.
Revised: 1 for the small MOptEqs with three branches for the single-view comparison as OptEqs.

Original: The batch size is set to be 128 for all the experiments.
Revised: The batch size is set to be 128 for all the experiments.

Original: As for the multi-scale models’ comparison, we use four branches with each branch take inputs with resolutions equals 32 , 16 , 8 , 4 like MDEQ, the output channel number for each branch is 256 for MOptEqs with 1 . 9 M learnable parameters.
Revised: As for the multi-scale models’ comparison, we use four branches with each branch take inputs with resolutions equals 32 , 16 , 8 , 4 like MDEQ, the output channel number for each branch is 256 for MOptEqs with 1 . 9 M learnable parameters.

Original: At the same time, we set the channel number to be 128 for MOptEqs with 0 .
Revised: At the same time, we set the channel number to be 128 for MOptEqs with 0 .

Original: 48 M learnable parameters.
Revised: 48 M learnable parameters.

Original: And we set λ = 0 .
Revised: And we set λ = 0 .

Original: 001 and perturbation size (cid:15) = 0 .
Revised: 001 and perturbation size ϵ = 0 .

Original: 1 for the experiments.
Revised: 1 for the experiments.

Original: We use the widely used SGD algorithm for training procedure.
Revised: We use the widely used SGD algorithm for training procedure.

Original: We set weight decay to be 0 .
Revised: We set weight decay to be 0 .

Original: 001 and the initial learning rate start from 0 .
Revised: 001 and the initial learning rate start from 0 .

Original: 1 and decay by 0 .
Revised: 1 and decay by 0 .

Original: 1 at 100 , 150 , 175 -th epoch with 200 epochs in total like others.
Revised: 1 at 100 , 150 , 175 -th epoch with 200 epochs in total like others.

Original: We use the standard data augmentation for all the experiments.
Revised: We use the standard data augmentation for all the experiments.

Original: Our implementation for small MDEQ only changes the channel number of MDEQ’s 10 M model to obtain the comparable size with our model, without changing the training settings and its hyper-parameters.
Revised: Our implementation for small MDEQ only changes the channel number of MDEQ’s 10 M model to obtain the comparable size with our model, without changing the training settings and its hyper-parameters.

Original: A.6. CIFAR-
Revised: A.6. CIFAR-

Original: Except for CIFAR- 10 , we also ﬁnish the experiments on CIFAR- 100 classiﬁcation 2 to further verify our model’s eﬀectiveness.
Revised: Except for CIFAR- 10 , we also ﬁnish the experiments on CIFAR- 100 classiﬁcation 2 to further verify our model’s eﬀectiveness.

Original: The model we used is the same as the models used in the multi-scale comparison, only changing the output from 10 to 100 for classiﬁcation.
Revised: The model we used is the same as the models used in the multi-scale comparison, only changing the output from 10 to 100 for classiﬁcation.

Original: MDEQ with 11 M parameter is the same model as (Bai et al., 2020) proposed for CIFARexperiment with only changing the output from 10 to 100 for CIFAR- 100 .
Revised: MDEQ with 11 M parameter is the same model as (Bai et al., 2020) proposed for CIFARexperiment with only changing the output from 10 to 100 for CIFAR- 100 .

Original: Moreover, the hyper-parameter setting is also the same as (Bai et al., 2020) proposed in the paper.
Revised: Moreover, the hyper-parameter setting is also the same as (Bai et al., 2020) proposed in the paper.

Original: We use the widely used SGD algorithm for training procedure.
Revised: We use the widely used SGD algorithm for training procedure.

Original: We set weight decay to be 0 .
Revised: We set weight decay to be 0 .

Original: 001 and the initial learning rate start from 0 .
Revised: 001 and the initial learning rate start from 0 .

Original: 1 and decay by 0 .
Revised: 1 and decay by 0 .

Original: 1 at 100 , 150 , 175 -th epoch with 200 epochs in total like others.
Revised: 1 at 100 , 150 , 175 -th epoch with 200 epochs in total like others.

Original: We use the standard data augmentation for all the experiments.
Revised: We use the standard data augmentation for all the experiments.

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: A.6. ImageNette
Revised: A.6. ImageNette

Original: Besides the classiﬁcation for images with small size, we also conduct the experiments on the
Revised: Besides the classiﬁcation for images with small size, we also conduct the experiments on the

Original: Imagenette 3 , which is a subset of 10 classes from ImageNet 4 with 9469 training photos and its test set consists of 3925 images.
Revised: Imagenette 3 , which is a subset of 10 classes from ImageNet 4 with 9469 training photos and its test set consists of 3925 images.

Original: Furthermore, we use the full-size version of Imagenettefurther to verify the superiority of our models on large-scale images.
Revised: Furthermore, we use the full-size version of Imagenettefurther to verify the superiority of our models on large-scale images.

Original: The model architecture for our MOptEqs and MDEQ (for comparison) is the same as the ones used in the multi-scale models’ comparison for CIFAR- 10 except adding two downsampling layers in the head of models to downsample the input size from 256 to 64 .
Revised: The model architecture for our MOptEqs and MDEQ (for comparison) is the same as the ones used in the multiscale models’ comparison for CIFAR- 10 except adding two downsampling layers in the head of models to downsample the input size from 256 to 64 .

Original: As for the hyper-parameters, we only change the weight decay to be 5 e − 5 , batch size to be 32 and extend the whole training epochs to 200 with 100 -th, 150 -th, 175 -th for the learning rate decay for all the models.
Revised: As for the hyper-parameters, we only change the weight decay to be 5 e − 5 , batch size to be 32 and extend the whole training epochs to 200 with 100 -th, 150 -th, 175 -th for the learning rate decay for all the models.

Original: All the hyper-parameters for both MOptEqs and MDEQs are hardly changed during all the datasets.
Revised: All the hyper-parameters for both MOptEqs and MDEQs are hardly changed during all the datasets.

Original: A. Convergence validation for our MOptEqs.
Revised: A. Convergence validation for our MOptEqs.

Original: As shown in Figure.4(a), the Anderson method can quickly ﬁnd the equilibrium points for our small MOptEq used for single-view comparison.
Revised: As shown in Figure 4(a), the Anderson method can quickly ﬁnd the equilibrium points for our small MOptEq used for single-view comparison.

Original: From the ﬁgure, one can see that MOptEqs with proper λ (around or smaller than 1 C , around 0 .
Revised: From the ﬁgure, one can see that MOptEqs with proper λ (around or smaller than 1 C , around 0 .

Original: 03 in this circumstance) can ensure the whole mappings be contractive.
Revised: 03 in this circumstance) can ensure the whole mappings be contractive.

Original: But too large λ will make the model hard to converge or even fail.
Revised: But too large λ will make the model hard to converge or even fail.

Original: Apart from plotting the convergent iterations’ of our forward root-ﬁnding procedure for the small MOptEqs (shown in Figure.4 (a)), we also draw ﬁgures to show the convergence of the forward propagation for our multi-scale MOptEqs ( λ = 0 .
Revised: Apart from plotting the convergent iterations’ of our forward root-ﬁnding procedure for the small MOptEqs (shown in Figure 4 (a)), we also draw ﬁgures to show the convergence of the forward propagation for our multi-scale MOptEqs ( λ = 0 .

Original: 001 ) used in the imagenette and
Revised: 001 ) used in the imagenette and

Original: CIFAR experiments as Figure.4 (b) shown.
Revised: CIFAR experiments as Figure 4 (b) shown.

Original: We use the pre-trained model on Imagenette whose prediction accuracy is reported in the above sections.
Revised: We use the pre-trained model on Imagenette whose prediction accuracy is reported in the above sections.

Original: One can see that our multi-scale
Revised: One can see that our multi-scale

Original: MOptEqs can also quickly converge to the equilibrium points as we expected.
Revised: MOptEqs can also quickly converge to the equilibrium points as we expected.

Original: A. Ablation Studies on the impact of D and H part
Revised: A. Ablation Studies on the impact of D and H part

Original: In this section, we analyzed the inﬂuence of our D and H term for a single-branch MOptEq model trained without PE method shown in Table.
Revised: In this section, we analyzed the inﬂuence of our D and H term for a single-branch MOptEq model trained without PE method shown in Table.

Original: We set λ = 0 .
Revised: We set λ = 0 .

Original: 02 for the experiment.
Revised: 02 for the experiment.

Original: From the table, one can see that the contribution of the H term is slightly higher than the diversity term.
Revised: From the table, one can see that the contribution of the H term is slightly higher than the diversity term.

Original: But considering both two terms can signiﬁcantly improve the results comparing with MOptEqs without considering these two terms.
Revised: But considering both two terms can signiﬁcantly improve the results comparing with MOptEqs without considering these two terms.

Original: [Table]
Revised: [Table]

Original: A. Experiments on the ImageNet
Revised: A. Experiments on the ImageNet

Original: Except CIFAR and Imagenette, we also conducted experiments on ImageNet with 13 M parameters shown as below:
Revised: Except CIFAR and Imagenette, we also conducted experiments on ImageNet with 13 M parameters shown as below:

Original: [Table]
Revised: [Table]

Original: From the results, one can see that our model can achieve the satisfying performance on
Revised: From the results, one can see that our model can achieve the satisfying performance on

Original: ImageNet compared with the state-of-the-art models.
Revised: ImageNet compared with the state-of-the-art models.

Original: A. PE method and robustness
Revised: A. PE method and robustness

Original: From the above experiments, one can see that the PE method may also improve the robustness of our MOptEqs.
Revised: From the above experiments, one can see that the PE method may also improve the robustness of our MOptEqs.

Original: In this section, we conduct experiments to see whether the PE methods can further enhance the robustness of MOptEqs when we use adversarial training methods on these models.
Revised: In this section, we conduct experiments to see whether the PE methods can further enhance the robustness of MOptEqs when we use adversarial training methods on these models.

Original: We ﬁnish comparisons of PGD- 3 training for the single-view MOptEq with or without PE, and the result is shown as follows:
Revised: We ﬁnish comparisons of PGD- 3 training for the single-view MOptEq with or without PE, and the result is shown as follows:

Original: From the table, we canconclude
Revised: From the table, we can

Original: [Table]
Revised: [Table]

Original:  that our PE method can improve the robustness of our MOptEqs model trained by the adversarial training method.
Revised: conclude that our PE method can improve the robustness of our MOptEqs model trained by the adversarial training method.

Original: Such advantages also demonstrate the superiority of our models’ mathematical interpretability since the PE method is acquired based on the model’s hidden optimization problem.
Revised: Such advantages also demonstrate the superiority of our models’ mathematical interpretability since the PE method is acquired based on the model’s hidden optimization problem.

Original: A. Other Training methods: Fixed Point method and One-step gradients A.11. Methods
Revised: A. Other Training methods: Fixed Point method and One-step gradients A.11. Methods

Original: Since our model is not so complicated, we can use the ﬁxed point method for L iteration for the forward propagation.
Revised: Since our model is not so complicated, we can use the ﬁxed point method for L iteration for the forward propagation.

Original: While for the back propagation, we can use BPTT (Mozer, 1989) and (Geng et al., 2021) proposed method, which only backward the ﬁnal forward iterations using the chain rule:
Revised: While for the back propagation, we can use BPTT (Mozer, 1989) and (Geng et al., 2021) proposed method, which only backward the ﬁnal forward iterations using the chain rule:

Original: [Equation]
Revised: [Equation]

Original: where θ ∈ Θ are the learnable parameters for the implicit models, T L Θ denotes the output z for the L -th ﬁxed-point iteration.
Revised: where θ ∈ Θ are the learnable parameters for the implicit models, T L Θ denotes the output z for the L -th ﬁxed-point iteration.

Original: Since the back-propagation of such a strategy is similar to backwards only a single layer of DNN module without the calculation of root-ﬁnding algorithm for the Jacobian in the implicit diﬀerentiation, the computational cost is much less than the implicit method.
Revised: Since the back-propagation of such a strategy is similar to backwards only a single layer of DNN module without the calculation of root-ﬁnding algorithm for the Jacobian in the implicit diﬀerentiation, the computational cost is much less than the implicit method.

Original: However, the gradient approximated by such a method is not accurate and may cause the performance drop.
Revised: However, the gradient approximated by such a method is not accurate and may cause the performance drop.

Original: However, as we illustrated below, such a method can be regarded as a trade-oﬀ if the computational resources of training are not enough.
Revised: However, as we illustrated below, such a method can be regarded as a trade-oﬀ if the computational resources of training are not enough.

Original: A.11. Empirical Results and Computational Efficiency of One-stpe Gradient
Revised: A.11. Empirical Results and Computational Efficiency of One-stpe Gradient

Original: We take the ImageNette dataset as an example, the training memory cost, forward time, and test accuracy are shown as follows (each batch contains 32 images and is ﬁnished on 2 x GTX 1080Ti for 200 epochs):
Revised: We take the ImageNette dataset as an example, the training memory cost, forward time, and test accuracy are shown as follows (each batch contains 32 images and is ﬁnished onx GTX 1080Ti for 200 epochs):

Original: Model
Revised: Model

Original: Memory/Batch
Revised: Memory/Batch

Original: Infer
Revised: Infer

Original: [Table]
Revised: [Table]

Original: /Batch
Revised: Batch

Original: [Table]
Revised: [Table]

Original: MOptEq
Revised: MDEQ

Original: [Table]
Revised: [Table]

Revised: MOptEq(one step)

Original: [Table]
Revised: [Table]

Original: The results shows that the one-step gradient methods can save a lot computation resources while with slightly drop on the performance.
Revised: The results shows that the one-step gradient methods can save a lot computation resources while with slightly drop on the performance.