Original: Optimization Inspired Multi-Branch Equilibrium Models
Revised: Optimization Inspired Multi-Branch Equilibrium Models

Original: Anonymous authors Paper under double-blind review
Revised: Anonymous authors Paper under double-blind review

Original: Abstract
Revised: Abstract

Original: Works have shown the strong connections between some implicit models and optimization problems.
Revised: Works have shown the strong connections between some implicit models and optimization problems.

Original: However, explorations on such relationships are limited.
Revised: However, explorations on such relationships are limited.

Original: Most works pay attention to some common mathematical properties, such as sparsity.
Revised: Most works pay attention to some common mathematical properties, such as sparsity.

Original: In this work, we propose a new type of implicit model inspired by the designing of the systems’ hidden objective functions, called the Multi-branch Optimization induced Equilibrium networks (MOptEqs).
Revised: In this work, we propose a new type of implicit model inspired by the designing of the systems’ hidden objective functions, called the Multi-branch Optimization induced Equilibrium networks (MOptEqs).

Original: The model architecture is designed based on modelling the hidden objective function for the multi-resolution recognition task.
Revised: The model architecture is designed based on modelling the hidden objective function for the multi-resolution recognition task.

Original: Furthermore, we also propose a new training strategy inspired by our understandings of the hidden objective function.
Revised: Furthermore, we also propose a new training strategy inspired by our understandings of the hidden objective function.

Original: In this manner, the proposed model can better utilize the hierarchical patterns for recognition tasks and retain the abilities for interpreting the whole structure as trying to obtain the minima of the problem’s goal.
Revised: In this manner, the proposed model can better utilize the hierarchical patterns for recognition tasks and retain the abilities for interpreting the whole structure as trying to obtain the minima of the problem’s goal.

Original: Comparing with the state-of-the-art models, our MOptEqs not only enjoys better explainability but are also superior to MDEQ with less parameter consumption and better performance on practical tasks.
Revised: Comparing with the state-of-the-art models, our MOptEqs not only enjoys better explainability but are also superior to MDEQ with less parameter consumption and better performance on practical tasks.

Original: Furthermore, we also implement various experiments to demonstrate the eﬀectiveness of our new methods and explore the applicability of the model’s hidden objective function.
Revised: Furthermore, we also implement various experiments to demonstrate the eﬀectiveness of our new methods and explore the applicability of the model’s hidden objective function.

Original:  Introduction
Revised:  Introduction

Original: Whereas Deep Neural Networks (DNNs) have achieved great success in many real-world tasks such as computer vision and neural language process, the limited interpretability of
Revised: Whereas Deep Neural Networks (DNNs) have achieved great success in many real-world tasks such as computer vision and neural language process, the limited interpretability of

Original: DNNs greatly hinders their further development.
Revised: DNNs greatly hinders their further development.

Original: However, many traditional machine learning methods, e.g.,matrix recovery (Zhang et al., 2018b; 2015; Liu & Li, 2016), subspace clustering (You et al., 2016), image deblurring (Liu et al., 2014) and so on, can be interpreted into minimizing the following objective functions:
Revised: However, many traditional machine learning methods, e.g.,matrix recovery (Zhang et al., 2018b; 2015; Liu & Li, 2016), subspace clustering (You et al., 2016), image deblurring (Liu et al., 2014) and so on, can be interpreted into minimizing the following objective functions:

Original: [Equation]
Revised: [Equation]

Original: where A ∈ R m × d 1 , B ∈ R m × d 2 , X ∈ R m × n , and f ( · ) and g ( · ) are convex functions designed by the modelling of their properties.
Revised: where A ∈ R m × d 1 , B ∈ R m × d 2 , X ∈ R m × n , and f ( · ) and g ( · ) are convex functions designed by the modelling of their properties.

Original: For this count, the interpretability of such methods is much better than DNNs.
Revised: For this count, the interpretability of such methods is much better than DNNs.

Original: Furthermore, these methods can also enjoy state-of-the-art and robust performance on these tasks.
Revised: Furthermore, these methods can also enjoy state-of-the-art and robust performance on these tasks.

Original: We call such interpretability these methods enjoy as "mathematical interpretability", i.e., whether the whole network structure can be summarized as a compact mathematical model that can be mathematically analyzed.
Revised: We call such interpretability these methods enjoy as "mathematical interpretability", i.e., whether the whole network structure can be summarized as a compact mathematical model that can be mathematically analyzed.

Original: Except for these methods, the Optimization Induced Equilibrium Networks (OptEqs) proposed by Xie et al. (2021) recover the wholeOptEqs system for recognition to an optimization problem. 
Revised: Except for these methods, the Optimization Induced Equilibrium Networks (OptEqs) pro- posed by Xie et al. (2021) recover its whole system for recognition to an optimization problem. The forward propagation for OptEqs tries to solve Eqn.1 to get output y , z ∗ ∈ R d for input x ∈ R d in .

Original: The forward propagation for OptEqs and its hidden objectives can be depicted as follows, enjoying good interpretability as these traditional works.
Revised: We call the front part of Eqn.1 as OptEqs’ equilibrium equation, which is the central part for equilibrium models. And its forward procedure can be regarded as solving the hidden objective functions shown as Eqn.2,

Original: [Equation] 
Revised: [Equation] [Equation]

Original:  Although  OptEqs proves that its forward propagation can be regarded as minimizing certain objective function and shows its strong interpretability, some weaknessesfor OptEqs are worth exploring.
Revised: W , U , W c are learnable weights, g is the convolution layers projecting input x from real-world domain to feature space like other neural architectures, and f is the penalty for the outputs. Although the interpretability for OptEqs is good, it still remains some weaknesses worth exploring.

Original: First, although OptEqs performs better than some implicit models, these models only consider a single view (or resolution) of the input in their implicit parts.
Revised: First, although OptEqs performs better than some implicit models, these models only consider a single view (or resolution) of the input in their implicit parts.

Original: We call these models single-view implicit models in the following.
Revised: We call these models single-view implicit models in the following.

Original: However, nearly all the stateof-the-art pattern recognition systems (Lee et al., 2009b; Wang et al., 2019a; Huang et al., 2017; He et al., 2016; Burt & Adelson, 1987) beneﬁt from the multi-layer or multi-resolution feature extractors in domains like computer vision and audio processing.
Revised: However, nearly all the state-of-the-art pattern recognition systems (Lee et al., 2009b; Wang et al., 2019a; Huang et al., 2017; He et al., 2016; Burt & Adelson, 1987) beneﬁt from the multi-layer or multi-resolution feature extractors in domains like computer vision and audio processing.

Original: Secondly, new modules induced by the objective function they proposed are limited, which mainly consider the math properties of the output features like sparsity, but whether these properties can beneﬁt image recognition tasks is unexplored.
Revised: Secondly, new modules induced by the objective function they proposed are limited, which mainly consider the math properties of the output features like sparsity, but whether these properties can beneﬁt image recognition tasks is unexplored.

Original:  MDEQ (Bai et al., 2020) is the state-of-the-art implicit model so far, which utilizes the multiscale information from samples by multi-branch structure and branches fusion.
Revised: Besides OptEqs, MDEQ (Bai et al., 2020) constructs its equilibrium equation z ∗ = F ( z ∗ ; x ) and block F with the inspiration of explicit models especially HRNet Wang et al. (2019a).

Original: Nevertheless,
Revised: Then they solve the equilibrium equation by the accelerated algorithm for output z ∗ .

Original:  MDEQ ismore likely to be a traditional neural network than OptEqs and MonDEQ (Winston & Kolter, 2020), since its whole structure is designed like other explicit DNNs (like stacking residual blocks and its traditional fusion method). Such architecture prevents it from retaining good interpretability as we desired.
Revised: Although it shows the state-of-the-art performance with eﬃcient memory cost as other implicit models, such strategies will make the model lose "interpretability" since the MDEQ is a black box because its output is solved in an implicit way (root-ﬁnding method), which makes analysis on its features of intermediate layers almost impossible. Furthermore, its complicated structure also hinders its analysis from mathematical systems.

Original: Therefore, the main challenge for us is to design a class of implicit models that can perform better than MDEQ and retain the ability for recovering the whole structure as an optimization problem.
Revised: More eﬃcient blocks with good interpretability still needs exploring. Motivated by the limitations of the above models, we would like to design a multi-scale DEQ structure with state-of-the-art performance and preserve mathematical interpretability from the inspiration of OptEqs and multi-scale models.

Original: The contributions for our work are listed below:
Revised: The contributions for our work are listed below:

Original: • We propose a multi-branch implicit model, called Multi-branch OptEqs (MOptEqs), which eﬃciently utilizes diﬀerent scales inputs with better performance and smaller model size.
Revised: • We propose a multi-branch implicit model, called Multi-branch OptEqs (MOptEqs), which eﬃciently utilizes diﬀerent scales inputs with better performance and smaller model size.

Original: Furthermore, our model still retains its connection to an optimization problem.
Revised: Furthermore, it still retains its connection to an optimization problem.

Original: • We propose some properties modelled as new terms for the model’s hidden objective function from our analysis on the relationships between branches and their hierarchical dependencies.
Revised: • We propose some properties modelled as new terms for the model’s hidden objective function from our analysis on the relationships between branches and their hierarchical dependencies.

Original: With such a problem’s formulation, we obtain the fusion module in our MOptEqs, called Hierarchical Heritage and Diversity Module (HH & D Module).
Revised: With new problem’s formulation, we obtain the fusion module in our MOptEqs, called Hierarchical Heritage and Diversity Module (HH & D Module).

Original: • Apart from the new module design, we also propose the Perturbation Enhanced training strategy (PE) for our MOptEqs from our analysis on the model’s hidden objective function.
Revised: • Apart from the new module design, we also propose the Perturbation Enhanced training strategy (PE) for our MOptEqs from our analysis on the model’s hidden objective function.

Original: The new strategy cannot only enhance the performance of our MOptEqs, but also improve the robustness of our models.
Revised: The new strategy cannot only enhance the performance of our MOptEqs, but also improve the robustness of our models.

Original: 1. Related Works
Revised: 1. Related Works

Original: Implicit Models.
Revised: Implicit Models.

Original: Nearly all modern deep learning approaches use explicit models, which provide an explicit computation graph for the forward propagation.
Revised: Nearly all modern deep learning approaches use explicit models, which provide an explicit computation graph for the forward propagation.

Original: In contrast, the computation graphs for implicit models are "ﬂexible" or can be assumed as having "inﬁnite" depth.
Revised: In contrast, the computation graphs for implicit models are "ﬂexible" or can be assumed as having "inﬁnite" depth.

Original: For example, Neural ODEs (Chen et al., 2018; Massaroli et al., 2020) encode their neural architectures by a diﬀerential system with learnable parameters.
Revised: For example, Neural ODEs (Chen et al., 2018; Massaroli et al., 2020) encode their neural architectures by a diﬀerential system with learnable parameters.

Original: Then the implicit ODE solvers they used is equivalent to a continuous ResNet taking inﬁnitesimal steps.
Revised: Then the implicit ODE solvers they used is equivalent to a continuous ResNet taking inﬁnitesimal steps.

Original: Furthermore, the training process of Neural ODEs can be depicted as ﬁnding a diﬀerential system of a certain type (like heat equations) by updating its learnable parameters which demonstrate that Neural ODE also enjoy interpretability to a certain extent.
Revised: Furthermore, the training process of Neural ODEs can be depicted as ﬁnding a diﬀerential system of a certain type (like heat equations) by updating its learnable parameters which demonstrate that Neural ODE also enjoy interpretability to a certain extent.

Original: Moreover,the computation of the forward propagation of DEQ (Bai et al., 2019; Winston & Kolter, 2020) can be depicted as solving thefollowing equilibrium state:
Revised: Moreover, DEQ (Bai et al., 2019; Winston & Kolter, 2020) is another class of implicit models. The central part of DEQ is the designation of the equilibrium equations z ∗ =

Original: [Equation]
Revised: F ( z ∗ ; x ) and block F . Its forward procedure is trying to solve the equilibrium equations with input x to get the equilibrium state z ∗ as output by accelerated algorithms.

Original: where x ∈ R d in , z ∗ ∈ R
Revised: Since the z ∗ is ﬁxed given

Original: d are the inputs and equilibrium points of DEQ, W , U , W c are learnable weights, y are ﬁnal outputs and g is a small explicit model to project the input x from real-world domain to the feature space, which is widely used in neural architectures.
Revised: F and x , its inference can be regarded as forwarding an explicit network stacked by the same block F for inﬁnite times. For example, DEQ chooses their F by one set of Conv+ReLU while MDEQ constructs an HRNet-like block as theirs. However, no evidence shows that the best block constructions in explicit models can perform the best in the DEQ scheme.

Original: g ( · ) usually consists of one or two convolution layers. Satisfying some analytical conditions such as norm constraints for weights or outputs, the computation graphs for solving z ∗ can also be viewed as forwarding a proper neural network with inﬁnite depth.
Revised: The construction of DEQ blocks or equilibrium equations is an open question worth exploring.

Original: But unlike DNNs, implicit models can use shallower computation graphs computed by some accelerated root-ﬁnding problems for their forward  and backward propagation. Due to the above reasons, such simple models can achieve comparable results with more eﬃcientparameters and memory costs.
Revised: Furthermore, most DEQ blocks do not have any mathematical insights (like the diﬀusion process in Neural ODE) and perform totally a black box with limited interpretability. Since implicit models usually adopt accelerated root-ﬁnding algorithms for their forward outputs and backward gradients, they enjoy the advantages of constant and more eﬃcient memory costs compared with DNNs.

Original: Because of the ﬂexibility and interpretability for implicit models, the design of implicit models draws much attention these days (Ghaoui et al., 2019; Gould et al., 2019).
Revised: Due to the above advantages, the design of implicit models draws much attention these days (Ghaoui et al., 2019; Gould et al., 2019).

Original: Apart from the models illustrated above, many kinds of other implicit models have been proposed, including diﬀerentiable physics engines (Qiao et al., 2020; de Avila Belbute-Peres et al., 2018), logical structure learning (Wang et al., 2019b) and implicit neural blocks (Li et al., 2020).
Revised: Apart from the models illustrated above, many kinds of other implicit models have been proposed, including diﬀerentiable physics engines (Qiao et al., 2020; de Avila Belbute-Peres et al., 2018), logical structure learning (Wang et al., 2019b) and implicit neural blocks (Li et al., 2020).

Original: Model Interpretaiblity.
Revised: Model Interpretaiblity.

Original: Many researchers nowadays are trying to make their proposed method more interpretable.
Revised: Many researchers nowadays are trying to make their proposed method more interpretable.

Original: Although there are many kinds of approaches to achieve this goal, we marginally divide them into two parts: "Empirical" and "Mathematical" interpretability.
Revised: Although there are many kinds of approaches to achieve this goal, we marginally divide them into two parts: "Empirical" and "Mathematical" interpretability.

Original: Many works, such as (Zhang et al., 2018a; Zhang & Zhu, 2018;
Revised: Many works, such as (Zhang et al., 2018a; Zhang & Zhu, 2018;

Original: Bau et al., 2018), attempt to empirically disassemble the black box by characterizing some statistical or structural information like the outputs and gradients of neural networks’ middle layers.
Revised: Bau et al., 2018), attempt to empirically disassemble the black box by characterizing some statistical or structural information like the outputs and gradients of neural networks’ middle layers.

Revised: However, these works cannot be directly implemented on DEQ models since implicit ones do not have explicit depths like DNNs, which makes analyzing such models by dissecting each layer’s behavior almost impossible.

Original: Apart from that, researchers (Djolonga & Krause, 2017; Amos & Kolter, 2017; Xie et al., 2019; Chan et al., 2020) also managed to understand the neural architecture by linking it to a mathematical problem.
Revised: Apart from that, researchers (Djolonga & Krause, 2017; Amos & Kolter, 2017; Xie et al., 2019; Chan et al., 2020) also managed to understand the neural architecture by linking it to a mathematical problem.

Original: In this way, researchers can analyze the black box by its corresponding mathematical problem.
Revised: In this way, researchers can analyze the black box by its corresponding mathematical problem.

Original: Furthermore, new components can be proposed due to the analysis of these problems.
Revised: Furthermore, new components can be proposed due to the analysis of these problems.

Original: Our model aims to design a new DEQ architecture with proper mathematical interpretability.
Revised: Our model aims to design a new DEQ architecture with proper mathematical interpretability.

Original: Compared with the former works, our model can deal with the classiﬁcation tasks with better performances with novel components.
Revised: Compared with the former works, our model can deal with the classiﬁcation tasks with better performances with novel components.

Original:  Multi-Branch Optimized Induced Equilirium Models 2. The proposed architecture for the Multi-Branch OptEqs
Revised:  Multi-Branch Optimized Induced Equilirium Models 2. The proposed architecture for the Multi-Branch OptEqs

Original: Although OptEqs can maintain its ability to reformulate the whole architecture as a solution of minimizing an objective function they designed (i.e. the mathematical interpretability), its performance is still weak. Motivated by the huge improvement gained from processing multi-scale inputs in MDEQ, we’d like to design a multi-scale OptEqs architecture with state-of-the-art performance and preserve mathematical interpretability.
Revised: Inspired by OptEqs, we ﬁrst design an objective function for our tasks to solve and then use its ﬁrst-order stationary conditions as MOptEqs’ equilibrium equation.

Original: To achieve this goal, we ﬁrst parallel several  OptEqslike MOptEqs (Details can be found in Appendix.A.1).
Revised: The base of our function is formed by summarizing several objective functions of diﬀerent scales deﬁned for OptEqs (Details are stated in Appendix.A.1).

Original: The most important problem is how to design  the fusion method of each branch that suits our purpose. However, the existing fusion method can not satisfy our target. Simply using non-parametric operators (like addition) can ensure the interpretability, but the performance is poor.
Revised: However, such a model can not obtain satisfying results since these branches are independent. For this count, we need to design some terms describing the dependencies of each branch in the hidden objective function, as shown in the following paragraphs.

Original: Utilizing convolutional blocks like those used in MDEQ and other works (Ronneberger et al., 2015; Wang et al., 2019a) can improve the performance. Nevertheless, they cannot be reformulated as optimization problems, which means their mathematical interpretability is poor.
Revised: Then we can obtain the equilibrium equations for our

Original: For this account, we’d like to design our fusion method by modelling the architecture’s hidden problems and convert them to novel fusion components.
Revised: MOptEqs and our MOptEqs’ block F by analyzing the ﬁrst-order stationary condition of our designed optimization problem.

Original: Hierarchical Heritage Modeling.
Revised: Hierarchical Heritage Modeling.

Original: State-of-the-art explicit models for image tasks are explicitly structured into sequential stages and process diﬀerent resolutions hierarchically (He et al., 2016;
Revised: State-of-the-art explicit models for image tasks are explicitly structured into sequential stages and process diﬀerent resolutions hierarchically (He et al., 2016;

Original: Shelhamer et al., 2017; Lee et al., 2009a), which implies that features should be extracted hierarchically.
Revised: Shelhamer et al., 2017; Lee et al., 2009a), which implies that features should be extracted hierarchically.

Original: In other words, the posterior branches should inherit from their prior ones, which we call such property hierarchical heritage.
Revised: In other words, the posterior branches should inherit from their prior ones, which we call such property hierarchical heritage.

Original: In our work, we are going to formulate such correlation of neighbouring branches in the hidden objective function by adding the following inner-product term into its origin:
Revised: In our work, we are going to formulate such correlation of neighbouring branches in the hidden objective function by adding the following inner-product term into its origin:

Original: [Equation]
Revised: [Equation]

Original: where z i ∈ R
Revised: where z i ∈ R

Original: d
Revised: d

Original: i , z j ∈ R d j , P i : j ∈ R s i × s j denotes the Average Downsample (its transpose can be regarded as a weighted nearest upsample) or Identity matrix suited to the shape of z i and z j
Revised: i , z j ∈ R d j , P i : j ∈ R s i × s j denotes the Average Downsample (its transpose can be regarded as a weighted nearest upsample) or Identity matrix suited to the shape of z i and z j

Original: ( i < j or j = 1 , i = L ).
Revised: ( i < j or j = 1 , i = L ).

Original: This term only estimate the summation for the similarity of i -th and j -th branch’s channels with the same channel index, which we call them corresponding channels , as shown in the above equation.
Revised: This term only estimate the summation for the similarity of i -th and j -th branch’s channels with the same channel index, which we call them corresponding channels , as shown in the above equation.

Original: When z i and z j are similar, the inner product will be large.
Revised: When z i and z j are similar, the inner product will be large.

Original: Otherwise, the result will become small, which implies that the relationship of corresponding channels is weak.
Revised: Otherwise, the result will become small, which implies that the relationship of corresponding channels is weak.

Original: In this way, we can ensure the similarity of corresponding channels of the near branches.
Revised: In this way, we can ensure the similarity of corresponding channels of the near branches.

Original: Except for the relations between diﬀerent branches’ corresponding channels, the relationships between dis-corresponding channels are explored in the following section.
Revised: Except for the relations between diﬀerent branches’ corresponding channels, the relationships between dis-corresponding channels are explored in the following section.

Original: Diversity Modeling.
Revised: Diversity Modeling.

Original: In addition to the hierarchical relations for corresponding channels, works (Pang et al., 2019; Amada et al., 2021) have shown enhancing the diversities between branches can also improve the model’s improvements.
Revised: In addition to the hierarchical relations for corresponding channels, works (Pang et al., 2019; Amada et al., 2021) have shown enhancing the diversities between branches can also improve the model’s improvements.

Original: For this account, we also consider making diﬀerent branches extract various features to improve the representation abilities of diﬀerent branches.
Revised: For this account, we also consider making diﬀerent branches extract various features to improve the representation abilities of diﬀerent branches.

Original: To achieve such a goal, we add a diversity term in the objective function
Revised: To achieve such a goal, we add a diversity term in the objective function

Original: Eqn.(3) if we assume that i < j ,
Revised: Eqn.(4) if we assume that i < j ,

Original: [Equation]
Revised: [Equation]

Original: where vec − 1 is the inverse vectorization operator convert the vectorized feature
Revised: where vec − 1 is the inverse vectorization operator convert the vectorized feature

Original: z i ∈ R
Revised: z i ∈ R

Original: d i ×to a matrix in R diC × C with C channels and vec − 1 ( z i ) ( k ) ∈ R diC × 1 denotes the k -th channel of i -th branch in practice.
Revised: d i ×to a matrix in R diC × C with C channels and vec − 1 ( z i ) ( k ) ∈ R diC × 1 denotes the k -th channel of i -th branch in practice.

Original: Since we have already considered the relationships between the corresponding channels in the hierarchical term, we only tries to estimate the diversities of dis-corresponding channels of the i -th and j -th branches in this term.
Revised: Since we have already considered the relationships between the corresponding channels in the hierarchical term, we only tries to estimate the diversities of dis-corresponding channels of the i -th and j -th branches in this term.

Original: As the diversity term goes small, the diversities between the branches become stronger.
Revised: As the diversity term goes small, the diversities between the branches become stronger.

Original: The architecture of MOptEqs and its hidden problem.
Revised: The architecture of MOptEqs and its hidden problem.

Original: With the two terms we proposed, we can reformulate the hidden objective for our problem by replacing the L 2 norm with our new modelling as follows:
Revised: With the two terms we proposed, we can reformulate the hidden objective for our problem by replacing the L 2 norm with our new modelling as follows:

Original: [Equation]
Revised: [Equation]

Original: where g ( x ) is the input feature with raw input x ,hyperparamter λ  is a positive constant, z i ∈ 
Revised: g ( x ) is the input feature for the raw input x , λ > 0 is a hyperparameter and z i ∈ R d i × 1 are the ﬁnal outputs of MOptEqs.

Original: R d i [-× 
-]
Revised: We can choose f to constrain z i [-× 
-]on our demand and will inﬂuence model’s activation function.

Original: and f is the positive orthant f ( x )
Revised: If we choose f ( x )

Original: =
Revised: =

Original: I { x ≥ 0 } and ( I + ∂f ) − 1 is theReLU activation function.
Revised: I { x ≥ 0 } to ensure the outputs to be positive, then the activation function is ReLU.

Original: We set D ( z L , z L +1 )
Revised: We set D ( z L , z L +1 )

Original: =
Revised: =

Original: D ( z L , z 1 ) and H ( z L , z L +1 ) =
Revised: D ( z L , z 1 ) and H ( z L , z L +1 ) =

Original: H ( z L , z 1 ) when i
Revised: H ( z L , z 1 ) when i

Original: =
Revised: =

Original: L to complete the loop of relationship.
Revised: L to complete the loop.

Original: The two terms we added tries to make the corresponding channels of diﬀerent branches correlated by maximizing the H term and enhancing the diversities of diﬀerent branches by minimizing D term.
Revised: The two terms we added tries to make the corresponding channels of diﬀerent branches correlated by maximizing the H term and enhancing the diversities of diﬀerent branches by minimizing D term.

Original: We note that H and D are not conﬂict since they are handling diﬀerent pairs of channels for branches.
Revised: We note that H and D are not conﬂict since they are handling diﬀerent pairs of channels for branches.

Original: In the following proposition, we conclude that the calculation of our MOptEqs is equivalent to solving the  stationary points ∇ z i G = 0  .
Revised: As the following proposition shows, we ﬁnally get the equilibrium equations (Eqn.(6)) for our MOptEqs by calculating the ﬁrst-order stationary conditions ∇ z i G = 0 for problem G .

Original: Proposition 

Revised: Proposition

Original: The proposed multi-branch structure induced by Eqn(4) can be depicted as solving the equilibrium points (cid:101) z ∗ :=
Revised: The proposed multi-branch structure induced by Eqn(5) can be depicted as solving the equilibrium points (cid:101) z ∗ :=

Original: [ z (cid:62)∗ 1 , ..., z (cid:62)∗ L ] (cid:62) ∈ R (cid:80)
Revised: [ z (cid:62)∗ 1 , ..., z (cid:62)∗ L ] (cid:62) ∈ R (cid:80)

Original: Li =1 d i for the following equations:
Revised: Li =1 d i for the following equations:

Original: [Equation]
Revised: [Equation]

Original: where (cid:102) W =  
Revised: where (cid:102) W =  

Original: [Equation] [Equation] W L 
Revised: W.

Original: [Equation]
Revised: [Equation]

Original: [Equation] [ U [Equation]
Revised: W

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: And h (˜ z ) =
Revised: And h (˜ z ) =

Original: [ h 1 ( z 1 , z L , z 2 )(cid:62) , .., h i
Revised: [ h 1 ( z 1 , z L , z 2 )

Original: [Equation]
Revised: [Equation]

Original:  z i , z i
Revised: , .., h i ( z i , z i

Original: [Equation]
Revised: [Equation]

Original: 1 , z i +1 ) (cid:62) , .., h L 
Revised: 1 , z i +1 ) (cid:62) , .., h L ( z L , z L

Original: [Equation]
Revised: [Equation]

Original: z L , z L − 1 , z 1 ) (cid:62) ] (cid:62) and each h i is deﬁned as the mapping from R d i × R d
Revised: 1 , z 1 ) (cid:62) ] (cid:62) and each h i is deﬁned as the mapping from R d i × R d

Original: i +1 ×
Revised: i +1 ×

Original: R d i − 1 to R d i ,
Revised: R d i − 1 to R d i ,

Original: [Equation]
Revised: [Equation]

Original: The equilibrium points for the above formula are also the ﬁrst-order stationary points for the optimization problem Eqn(4).

Original: [Table]
Revised: [Table]

Original: ∈ ℝ (cid:3031) (cid:3284) 𝑧 (cid:2919)(cid:2878)(cid:2869) ′ 𝑧 (cid:2919) ′𝑧 (cid:2919)(cid:2879)(cid:2869) ′ (cid:2883)
Revised: ∈ ℝ (cid:3031) (cid:3284) 𝑧 (cid:2919)(cid:2878)(cid:2869) ′ 𝑧 (cid:2919) ′𝑧 (cid:2919)(cid:2879)(cid:2869) ′ (cid:2883)

Original: SIGN 𝑧 (cid:2919) ′𝑧 (cid:2919)(cid:2878)(cid:2869) ′ (cid:2883)
Revised: SIGN 𝑧 (cid:2919) ′𝑧 (cid:2919)(cid:2878)(cid:2869) ′ (cid:2883)

Original: SIGN
Revised: SIGN

Original: RD
Revised: RD

Original: RD 𝑧 (cid:2919) ′
Revised: RD 𝑧 (cid:2919) ′

Original: 𝑧 (cid:3043)(cid:3036) ∈ ℝ (cid:3031) (cid:3284) (
Revised: 𝑧 (cid:3043)(cid:3036) ∈ ℝ (cid:3031) (cid:3284) (

Original: [Figure]
Revised: [Figure]

Original: (cid:3004)×(cid:3009) (cid:3284)(cid:3126)(cid:3117) (cid:3024) (cid:3284)(cid:3126)(cid:3117) )
Revised: (cid:3004)×(cid:3009) (cid:3284)(cid:3126)(cid:3117) (cid:3024) (cid:3284)(cid:3126)(cid:3117) )

Original: [Figure]

Original:  (cid:3004)×(cid:3009) (cid:3284)(cid:3127)(cid:3117) (cid:3024) (cid:3284)(cid:3127)(cid:3117) )
Revised: (ℝ (cid:3004)×(cid:3009) (cid:3284)(cid:3127)(cid:3117) (cid:3024) (cid:3284)(cid:3127)(cid:3117) )

Original: [Figure]
Revised: [Figure]

Original: −λ 
Revised: −λ −λ

Original: [Figure]
Revised: [Figure]

Revised: The equilibrium points for the above formula are also the ﬁrst-order stationary points for the optimization problem Eqn(5).

Original: The proof of the proposition is listed in Appendix.
Revised: The proof of the proposition is listed in Appendix.

Original: A.3.
Revised: A.3.

Original: In this manner, we obtain the proposed architecture MOptEqs, its forward propagation is equivalent to ﬁnd the equilibrium points of Eqn(5), and such process can also be regarded as solving the stationary points of the optimization problem (4).
Revised: In this manner, we obtain the proposed architecture MOptEqs, its forward propagation is equivalent to ﬁnd the equilibrium points of Eqn(6), and such process can also be regarded as solving the stationary points of the optimization problem (5).

Original: The whole structure is also shown in Figure.1 (a) and we also draw ﬁgures to illustrate the practical process of h i (HH & D) on its right (Figure.1 (b)).
Revised: The whole structure is also shown in Figure.1 (a) and we also draw ﬁgures to illustrate the practical process of h i (HH & D) on its right (Figure.1 (b)).

Original: We also evaluate the eﬀectiveness of such modelling in Section.
Revised: We also evaluate the eﬀectiveness of such modelling in Section.

Original: 3.2’s experiments.
Revised: 3.2’s experiments.

Original: 2. The proposed Training Strategy for MOptEqs
Revised: 2. The proposed Training Strategy for MOptEqs

Original: Researches (Kong et al., 2020; Xie et al., 2020; Tang et al., 2021) have discovered that small perturbations can enhance the generalization abilities  of  neural architectures in various tasks and models.
Revised: Researches (Kong et al., 2020; Xie et al., 2020; Tang et al., 2021) have discovered that small perturbations can enhance the generalization abilities and varieties of branches for neural architectures.

Original: Most eﬃcient works use adversarial perturbations based on gradients (Xie & Yuille, 2020) for such a trick.
Revised: Most eﬃcient works use adversarial perturbations based on gradients (Xie & Yuille, 2020) for such a trick.

Original: However, such a strategy will slow the training process since they need at least one more back-propagation in each training step for perturbations, which is time-consuming.
Revised: However, such a strategy will slow the training process since they need at least one more back-propagation in each training step for perturbations, which is time-consuming.

Original: Unlike these methods, we can acquire more computation eﬃcient perturbations because our models can be formulated as minimizing an objective function we designed.
Revised: Unlike these methods, we can acquire more computation eﬃcient perturbations because our models can be formulated as minimizing an objective function we designed.

Original: Then we can replace the maximum problem (details in Appendix.
Revised: Then we can replace the maximum problem (details in Appendix.

Original: A.2) for adversarial perturbation with maximizing our architecture’s hidden objective function G , since we assume MOptEq’s performance will go worse if G ( z ; g ( x )) changes a lot at g ( x ) ’s neighborhoods since the forward propagation of MOptEqs is minimization G ( z ; g ( x )) .
Revised: A.2) for adversarial perturbation with maximizing our architecture’s hidden objective function G , since we assume MOptEq’s performance will go worse if G ( z ; g ( x )) changes a lot at g ( x ) ’s neighborhoods since the forward propagation of MOptEqs is minimization G ( z ; g ( x )) .

Original: Assumption 1 For a well trained OptEqs with its objective function denoted as G ( z ; g ( x )) and a natural sample x which can be correctly classiﬁed.
Revised: Assumption 1 For a well trained OptEqs with its objective function denoted as G ( z ; g ( x )) and a natural sample x which can be correctly classiﬁed.

Original: If an input perturbation (cid:107) δ (cid:107) ∞ ≤ (cid:15) can cause one of the following changes:
Revised: If an input perturbation (cid:107) δ (cid:107) ∞ ≤ (cid:15) can cause one of the following changes:

Original: [Equation]
Revised: [Equation]

Original: with L 1 , L 2 > 1 , it may lead the OptEqs to the wrong results with high probability.
Revised: with L 1 , L 2 > 1 , it may lead the OptEqs to the wrong results with high probability.

Original: For the second part of the assumption, it’s common sense and widely used in the analysis of adversarial robustness (Zhang et al., 2021; Li et al., 2020).
Revised: For the second part of the assumption, it’s common sense and widely used in the analysis of adversarial robustness (Zhang et al., 2021; Li et al., 2020).

Original: With the above assumption, we can generate the perturbations by maximizing the objective function G ( z ; g ( x )) for our
Revised: With the above assumption, we can generate the perturbations by maximizing the objective function G ( z ; g ( x )) for our

Original: MOptEqs.
Revised: MOptEqs.

Original: Since our MOptEqs can be regarded as a kind of "implicit" ensembling that parallels several equilibrium models, we decide to inject the perturbations acquired for the prior branch to the posterior one, like boosting to enhance the performance.
Revised: Since our MOptEqs can be regarded as a kind of "implicit" ensembling that parallels several equilibrium models, we decide to inject the perturbations acquired for the prior branch to the posterior one, like boosting to enhance the performance.

Original: Then the hidden objective function of MOptEqs with G ( z ; g ( x )) is shown as follows,
Revised: Then the hidden objective function of MOptEqs with G ( z ; g ( x )) is shown as follows,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where δ 0 = 

Revised: where δ 0 =

Original: And the problem becomes a bilevel optimization problem.
Revised: And the problem becomes a bilevel optimization problem.

Original: The solution to the lower-level problem is δ i
Revised: The solution to the lower-level problem is δ i

Original: =
Revised: =

Original: (cid:15) sign ( U (cid:62)
Revised: (cid:15) sign ( U (cid:62)

Original: i W − 1 (cid:62) i z i ) .
Revised: i W − 1 (cid:62) i z i ) .

Original: Since the perturbation can be obtained by feeding the output of the activation layer σ ( Wz + U g
Revised: Since the perturbation can be obtained by feeding the output of the activation layer σ ( Wz + U g

Original: ( x ) + b ) to the transposed convolution layer with weight
Revised: ( x ) + b ) to the transposed convolution layer with weight

Original: U i , we call it reconstructed perturbation.
Revised: U i , we call it reconstructed perturbation.

Original: Compared with the adversarial perturbations based on gradients, our perturbations can be acquired directly by matrix-vector multiplication instead of iteratively built by gradients.
Revised: Compared with the adversarial perturbations based on gradients, our perturbations can be acquired directly by matrix-vector multiplication instead of iteratively built by gradients.

Original: For this account, such a process does not require much computation cost.
Revised: For this account, such a process does not require much computation cost.

Original: Following the above steps, we obtain a harmful perturbed direction for the prior branches and then added them to their posterior branches.
Revised: Following the above steps, we obtain a harmful perturbed direction for the prior branches and then added them to their posterior branches.

Original: We note that our perturbation is added to the input feature g ( x ) instead of the raw input.
Revised: We note that our perturbation is added to the input feature g ( x ) instead of the raw input.

Original: Proposition 2 For a natural sample x and a well trained OptEqs
Revised: Proposition 2 For a natural sample x and a well trained OptEqs

Original: [Equation]
Revised: [Equation]

Original: with its hidden objective function denoted as G ( z ; g ( x )) , for g (cid:48)
Revised: with its hidden objective function denoted as G ( z ; g ( x )) , for g (cid:48)

Original: ( x ) = g ( x ) − δ and δ = (cid:15) sign ( U (cid:62)
Revised: ( x ) = g ( x ) − δ and δ = (cid:15) sign ( U (cid:62)

Original: i W − 1 (cid:62) i z i ) , which obeys the constraint (cid:107) δ (cid:107) ∞ ≤ (cid:15) .
Revised: i W − 1 (cid:62) i z i ) , which obeys the constraint (cid:107) δ (cid:107) ∞ ≤ (cid:15) .

Original: z (cid:48)∗ and z ∗ are deﬁned as follows:
Revised: z (cid:48)∗ and z ∗ are deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: If the (cid:107) W (cid:107) 2 , (cid:107) U (cid:107) 2 ≤ 1 and 1 N (cid:107) U (cid:62) W − 1 (cid:62) z ∗ (cid:107) 1 (cid:29) (cid:15) , where N is the element number of g ( x ) , then
Revised: If the (cid:107) W (cid:107) 2 , (cid:107) U (cid:107) 2 ≤ 1 and 1 N (cid:107) U (cid:62) W − 1 (cid:62) z ∗ (cid:107) 1 (cid:29) (cid:15) , where N is the element number of g ( x ) , then

Original: [Equation]
Revised: [Equation]

Original: The proposition demonstrates if we choose the perturbed direction to be (cid:15) sign ( U (cid:62) W − 1 (cid:62) z ) , at least one of the changes for the G or z ∗ will be around (cid:15) (cid:107) U (cid:62) W − 1 (cid:62) z ∗ (cid:107) 1 and implies each branch may perform bad on the perturbed data according to our assumptions.
Revised: The proposition demonstrates if we choose the perturbed direction to be (cid:15) sign ( U (cid:62) W − 1 (cid:62) z ) , at least one of the changes for the G or z ∗ will be around (cid:15) (cid:107) U (cid:62) W − 1 (cid:62) z ∗ (cid:107) 1 and implies each branch may perform bad on the perturbed data according to our assumptions.

Original: With the experiments in Section.3.2, we can conclude that our reconstructed perturbation is useful and our assumption is reasonable.
Revised: With the experiments in Section.3.2, we can conclude that our reconstructed perturbation is useful and our assumption is reasonable.

Original: Like boosting strategy, we can feed the perturbed data for the prior branch to its posterior branchduring training to enhance the performance; we call such method as the Perturbation Enhanced training strategy (PE).
Revised: Like boosting strategy, we can feed the perturbed data for the prior branch to its posterior branch to enhance the performance; we call such method as the Perturbation Enhanced training strategy (PE).

Original: Following experiments also show that PE can indeed enhance the performance of our MOptEqs.
Revised: Following experiments also show that PE can indeed enhance the performance of our MOptEqs.

Original: 2. Model Optimization and Forward Convergence
Revised: 2. Model Optimization and Forward Convergence

Original: Forward Propagation and Implicit Diﬀerentiation.
Revised: Forward Propagation and Implicit Diﬀerentiation.

Revised: Like other equilibrium models, the forward propagation procedure is solving the given equilibrium functions Eqn. 7.

Revised: For this count, our model also enjoys the constant memory cost advantages as other DEQs.

Original: We can run root-ﬁnding algorithms to solve the roots ˜
Revised: In our work, run root-ﬁnding algorithms to solve the roots ˜

Original: z ∗ =
Revised: z ∗ =

Original: [ z ∗ 1 , ...
Revised: [ z ∗ 1 , ...

Original: z ∗ L ] of the following problem to reach the equilibrium states for the forward propagation (shown in Appendix.
Revised: z ∗ L ] of the following problem to reach the equilibrium states, which is the same as MDEQ (shown in Appendix.

Original: A.5):
Revised: A.5):

Original: [Equation]
Revised: [Equation]

Original: As for the backward propagation, we also adopt the implicit diﬀerentiation method widely used in (Bai et al., 2020; 2019; Chen et al., 2018).
Revised: As for the backward propagation, we also adopt the implicit diﬀerentiation method widely used in (Bai et al., 2020; 2019; Chen et al., 2018).

Original: Instead of tracing the gradients during the forward propagation, the implicit diﬀerentiation method directly backpropagates through the equilibrium state using the Jacobian of T θ = ∇ ˜
Revised: Instead of tracing the gradients during the forward propagation, the implicit diﬀerentiation method directly backpropagates through the equilibrium state using the Jacobian of T θ = ∇ ˜

Original: z G + ˜
Revised: z G + ˜

Original: z
Revised: z

Original: at ˜ z ∗ .
Revised: at ˜ z ∗ .

Original: For a given loss l = L (˜ z ∗ , y ) (where y is the target) and the gradients can be written as
Revised: For a given loss l = L (˜ z ∗ , y ) (where y is the target) and the gradients can be written as

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: And the calculation of ∂l∂ ˜ z ∗ ( I − J T θ | ˜
Revised: And the calculation of ∂l∂ ˜ z ∗ ( I − J T θ | ˜

Original: z ∗ ) − 1 is equivalent to solve root m for the following equation:
Revised: z ∗ ) − 1 is equivalent to solve root m for the following equation:

Original: [Equation]
Revised: [Equation]

Original: As for the root-ﬁnding algorithm
Revised: As for the root-ﬁnding algorithm

Original: [Table]
Revised: [Table]

Original: we can use Broyden method (Broyden, 1965), Anderson
Revised: we can use Broyden method (Broyden, 1965), Anderson

Original: Method (Anderson, 1965; Bai et al., 2021) or other root-ﬁnding methods to solve Eqn(6) for the equilibrium state and Eqn(8) for the backward gradient.
Revised: Method (Anderson, 1965; Bai et al., 2021) or other root-ﬁnding methods to solve Eqn(7) for the equilibrium state and Eqn(9) for the backward gradient.

Original: Forward Convergence.
Revised: Forward Convergence.

Original: Like other implicit models, we need to make some constraints on parameters in order to make the whole MOptEqs T Θ (˜ z ; x ) ( ˜ z here is { z i } Li =1 ) be a contractive mapping.
Revised: Like other implicit models, we need to make some constraints on parameters in order to make the whole MOptEqs T Θ (˜ z ; x ) ( ˜ z here is { z i } Li =1 ) be a contractive mapping.

Original: For the MOptEqs without considering HH & D (Eqn.9), the models can easily converge with (cid:107) W i (cid:107) 2 ≤ ζ < 1 .
Revised: For the MOptEqs without considering HH & D (Eqn.10), the models can easily converge with (cid:107) W i (cid:107) 2 ≤ ζ < 1 .

Original: But since we use the HH & D module, we need to choose proper λ to ensure the convergence, around or less than 1 C will be appropriate, which does not need ζ to be too small.
Revised: But since we use the HH & D module, we need to choose proper λ to ensure the convergence, around or less than 1 C will be appropriate, which does not need ζ to be too small.

Original: We also conduct experiments to verify the convergence in Fig.
Revised: We also conduct experiments to verify the convergence in Fig.

Original: in Appendix.
Revised: in Appendix.

Original: A.7.
Revised: A.7.

Original:  Experimental Results
Revised:  Experimental Results

Original: In this section, we conduct experiments for the image classiﬁcation tasks on CIFAR- 10 ,CIFAR100 Krizhevsky et al.
Revised: In this section, we conduct experiments for the image classiﬁcation tasks on CIFAR- 10 ,CIFAR100 Krizhevsky et al.

Original: and ImageNette implemented on the PyTorch (Paszke et al., 2017) platform to demonstrate the eﬀectiveness of our model.
Revised: and ImageNette implemented on the PyTorch (Paszke et al., 2017) platform to demonstrate the eﬀectiveness of our model.

Original: Details are listed in Appendix..A.6.
Revised: Details are listed in Appendix..A.6.

Original: 3. Comparison of Prior Implicit Models
Revised: 3. Comparison of Prior Implicit Models

Original: In this part, we decide to verify the superiority of MOptEqs in two aspects.
Revised: In this part, we decide to verify the superiority of MOptEqs in two aspects.

Original: First, we build the single-scale MOptEqs and compare the empirical results with other single-scale implicit models.
Revised: First, we build the single-scale MOptEqs and compare the empirical results with other single-scale implicit models.

Original: And then we compare the experimental results with prior implicit models that utilize the multi-scale inputs like MDEQs, the state-of-the-art implicit model with multi-resolution.
Revised: And then we compare the experimental results with prior implicit models that utilize the multi-scale inputs like MDEQs, the state-of-the-art implicit model with multi-resolution.

Original: Implicit Models with Single View.
Revised: Implicit Models with Single View.

Original: In this part, we compare our MOptEqs with other single-view implicit models with comparable model sizes.
Revised: In this part, we compare our MOptEqs with other single-view implicit models with comparable model sizes.

Original: Like OptEqs who uses three blocks for their experiment, we ﬁrst construct our small MOptEqs with three branches whose outputs share the same size and the channel number C = 32 .
Revised: Like OptEqs who uses three blocks for their experiment, we ﬁrst construct our small MOptEqs with three branches whose outputs share the same size and the channel number C = 32 .

Original: Results in Table.1 (a) demonstrates that our MOptEqs structure can even enhance the performance for recognition tasks under the single-view cases.
Revised: Results in Table.1 (a) demonstrates that our MOptEqs structure can even enhance the performance for recognition tasks under the single-view cases.

Original: Compared to the multi-branch model without fusion (POptEqs), our
Revised: Compared to the multi-branch model without fusion (POptEqs), our

Original: MOptEqs’ HH & D can eﬃciently utilize the relationships between diﬀerent branches and lead the model to better performance.
Revised: MOptEqs’ HH & D can eﬃciently utilize the relationships between diﬀerent branches and lead the model to better performance.

Original: Our superiority also holds compared with former multi branches fusion methods: "sum" (non-parametric module) and "conv" (parametric module).
Revised: Our superiority also holds compared with former multi branches fusion methods: "sum" (non-parametric module) and "conv" (parametric module).

Original: Meanwhile, we can also conclude the eﬀectiveness of our perturbation enhanced training strategy from the table since its performance is the best.
Revised: Meanwhile, we can also conclude the eﬀectiveness of our perturbation enhanced training strategy from the table since its performance is the best.

Original: [Table]
Revised: [Table]

Original: Implicit Models with Multiple Scales.
Revised: Implicit Models with Multiple Scales.

Original: Moreover, we conduct experiments compared with other models which handling multi-scale inputs.
Revised: Moreover, we conduct experiments compared with other models which handling multi-scale inputs.

Original: Like MDEQ, we construct our MOptEqs with four branches with resolution size equals to 32 , 16 , 8 , 4 .
Revised: Like MDEQ, we construct our MOptEqs with four branches with resolution size equals to 32 , 16 , 8 , 4 .

Original: Other details can be found in
Revised: Other details can be found in

Original: Appendix.
Revised: Appendix.

Original: A.6.
Revised: A.6.

Original: From Table.1 (b) and as shown in Table.2 (a), one can see that our MOptEqs not only outperforms the widely used explicit model
Revised: From Table.1 (b) and as shown in Table.2 (a), one can see that our MOptEqs not only outperforms the widely used explicit model

Original: ResNet- 

Revised: ResNet-

Original: (He et al., 2016), but also shows better performance compared with MDEQ with fewer parameters on CIFAR, which is one of the best models.
Revised: (He et al., 2016), but also shows better performance compared with MDEQ with fewer parameters on CIFAR, which is one of the best models.

Original: The empirical results for the multi-scale models further verify the superiority of MOptEqs and its training strategy.
Revised: The empirical results for the multi-scale models further verify the superiority of MOptEqs and its training strategy.

Original: Apart from experiments on small images, we also conduct experiments on ImageNette, which is a subset of 10 classes from ImageNet.
Revised: Apart from experiments on small images, we also conduct experiments on ImageNette, which is a subset of 10 classes from ImageNet.

Original: Compared with MDEQ, our MOptEqs consistently perform better shown in Table.2 (b).
Revised: Compared with MDEQ, our MOptEqs consistently perform better shown in Table.2 (b).

Original: Nevertheless, as one can see from the results, the diﬀerence between MDEQ and our model becomes much bigger in the CIFAR- 100 and
Revised: Nevertheless, as one can see from the results, the diﬀerence between MDEQ and our model becomes much bigger in the CIFAR- 100 and

Original: Imagenette with the same training hyper-parameters.
Revised: Imagenette with the same training hyper-parameters.

Original: Such a phenomenon demonstrates that the performance of our model is much more stable than MDEQ.
Revised: Such a phenomenon demonstrates that the performance of our model is much more stable than MDEQ.

Original: Apart from these experiments, we also conducted ablation studies for our models in the following section.
Revised: Apart from these experiments, we also conducted ablation studies for our models in the following section.

Original: 3.  The comprehensive understanding of MOptEqs
Revised:  The comprehensive understanding of MOptEqs

Original: [Figure] 
Revised: [Figure] [Figure]

Revised: [Figure]

Revised: [Figure]

Original:  Visualization of the impact for the HH & D module.
Revised: [Figure] [Figure] [Figure] Visualization of the impact for the HH & D module.

Original: In this section, we try to estimate the eﬀect of our HH & D module.
Revised: In this section, we try to estimate the eﬀect of our HH & D module.

Original: We ﬁnish experiments in this part using three-branch
Revised: We ﬁnish experiments in this part using three-branch

Original: MOptEqs with 16 channels for each branch to make the visualization more clear.
Revised: MOptEqs with 16 channels for each branch to make the visualization more clear.

Original: We tries to verify our HH & D module’s eﬀect in two aspects.
Revised: We tries to verify our HH & D module’s eﬀect in two aspects.

Original: First, we plot scatter ﬁgures for the values of (cid:80) Li 
Revised: First, we plot scatter ﬁgures for the values of (cid:80) Li [Equation] i

Original: [Equation] 
Revised: [Equation] [Equation]

Original:  (denoted as Average D ) and(cid:80) Li
Revised: j [Equation] ( C 2 − C ) (denoted as Average D ) and

Original:  =1 (cid:80) j ∈N ( i ) H
Revised: (cid:80) Li =1 (cid:80) j ∈N ( i ) H

Original: ( z i , z j ) LC (denoted as Average H ) for 100 randomly chosen samples ( N ( i ) denotes neighboring branch of the i -th branch).
Revised: ( z i , z j ) LC (denoted as Average H ) for 100 randomly chosen samples ( N ( i ) denotes neighboring branch of the i -th branch).

Original: Since the hidden optimization problem is to maximizing H while minimizing
Revised: Since the hidden optimization problem is to maximizing H while minimizing

Original: D , we ﬁnd that our MOptEqs (Figure.2(a)) can induce the output features to reach such goal compared with POptEqs (MOptEqs without HH & D) (Figure.2(b)).
Revised: D , we ﬁnd that our MOptEqs (Figure.2(a)) can induce the output features to reach such goal compared with POptEqs (MOptEqs without HH & D) (Figure.2(b)).

Original: Furthermore, we also plot heatmaps of the ﬁrst two branches’ correlation score ( | vec − 1 ( z 2 ) vec − 1 ( z 1 ) (cid:62) | ∈
Revised: Furthermore, we also plot heatmaps of the ﬁrst two branches’ correlation score ( | vec − 1 ( z 2 ) vec − 1 ( z 1 ) (cid:62) | ∈

Original: R 16 × 16 ) for a randomly chosen sample shown in Figure.2(c)(d).
Revised: R 16 × 16 ) for a randomly chosen sample shown in Figure.2(c)(d).

Original: One can see that MOptEqs’ heatmap (Fig.2(c)) is more likely to be a diagonal matrix while heatmap Fig.2(d) for POptEqs looks random.
Revised: One can see that MOptEqs’ heatmap (Fig.2(c)) is more likely to be a diagonal matrix while heatmap Fig.2(d) for POptEqs looks random.

Original: The heatmap shows that our HH & D can induce the model to perform as our demand, which means corresponding channels for adjacent branches can be more correlated, while the dis-corresponding channels are unrelated due to our architecture design.
Revised: The heatmap shows that our HH & D can induce the model to perform as our demand, which means corresponding channels for adjacent branches can be more correlated, while the dis-corresponding channels are unrelated due to our architecture design.

Original: The visualizations and former results on diﬀerent datasets verify the eﬀectiveness of our modelling and our HH & D modules’ design.
Revised: The visualizations and former results on diﬀerent datasets verify the eﬀectiveness of our modelling and our HH & D modules’ design.

Original: [Figure]
Revised: [Figure]

Original: Evaluation of the reconstructed perturbation.
Revised: Evaluation of the reconstructed perturbation.

Original: We compare our perturbation with randomly generated ones in a one-branch well-trained MOptEqs to validate its eﬀectiveness.
Revised: We compare our perturbation with randomly generated ones in a one-branch well-trained MOptEqs to validate its eﬀectiveness.

Original: We ﬁrst feed a natural sample x to the model and acquire output z for x and generate our reconstructed perturbation using such output and then feed it to the model.
Revised: We ﬁrst feed a natural sample x to the model and acquire output z for x and generate our reconstructed perturbation using such output and then feed it to the model.

Original: Furthermore, we add the perturbations generated by Binomial distribution ( P ( δ i = (cid:15) ) =
Revised: Furthermore, we add the perturbations generated by Binomial distribution ( P ( δ i = (cid:15) ) =

Original: P ( δ i = − (cid:15) )
Revised: P ( δ i = − (cid:15) )

Original: = 0 . 5 ) and Uniform distribution
Revised: = 0 . 5 ) and Uniform distribution

Original: U [ − (cid:15), (cid:15) ] to input features g ( x ) for comparison.
Revised: U [ − (cid:15), (cid:15) ] to input features g ( x ) for comparison.

Original: The result of each distribution is averaged for ﬁve trials.
Revised: The result of each distribution is averaged for ﬁve trials.

Original: Figure.3(a) shows our perturbation generated by maximizing G is much more eﬀective, which validates the eﬀectiveness of our reconstructed perturbations and the rationality of our assumptions and analysis in Section.2.2.
Revised: Figure.3(a) shows our perturbation generated by maximizing G is much more eﬀective, which validates the eﬀectiveness of our reconstructed perturbations and the rationality of our assumptions and analysis in Section.2.2.

Original: Robustness of our MOptEqs trained by PE strategy.
Revised: Robustness of our MOptEqs trained by PE strategy.

Original: In addition to improving the generalization abilities for models as shown in Sec.3.1, perturbations with proper size may also enhance the robustness of our MOptEqs, shown in Figure.3(b).
Revised: In addition to improving the generalization abilities for models as shown in Sec.3.1, perturbations with proper size may also enhance the robustness of our MOptEqs, shown in Figure.3(b).

Original: We conduct experiments on small MOptEqs (the same setting as Sec.3.1’s single-view model).
Revised: We conduct experiments on small MOptEqs (the same setting as Sec.3.1’s single-view model).

Original: Our perturbation enhanced training strategy goes stronger as (cid:15) increases.
Revised: Our perturbation enhanced training strategy goes stronger as (cid:15) increases.

Original: The ﬁgure shows that accuracy for the model trained naturally drops quicker than trained by PE strategy.
Revised: The ﬁgure shows that accuracy for the model trained naturally drops quicker than trained by PE strategy.

Original: Overall, trained with appropriate reconstructed perturbation can partly improve the robustness of our MOptEqs.
Revised: Overall, trained with appropriate reconstructed perturbation can partly improve the robustness of our MOptEqs.

Original: MOptEqs vs. adding Regularizers in training loss.
Revised: MOptEqs vs. adding Regularizers in training loss.

Original: Apart from our HH & D modules, adding
Revised: Apart from our HH & D modules, adding

Original: D and H to the training loss is one of the most popular methods for obtaining outputs with certain properties.
Revised: D and H to the training loss is one of the most popular methods for obtaining outputs with certain properties.

Original: In order to demonstrate the superiority of our module empirically, we conduct experiments for our MOptEqs trained by cross-entropy and POptEqs (MOptEqs (w/o.
Revised: In order to demonstrate the superiority of our module empirically, we conduct experiments for our MOptEqs trained by cross-entropy and POptEqs (MOptEqs (w/o.

Original: HH & D)) trained by the cross-entropy loss adding γ (cid:80) Li =

Revised: HH & D)) trained by the cross-entropy loss adding γ (cid:80) Li =

Original: ( H ( z i , z i +1 )
Revised: ( H ( z i , z i +1 )

Original: − λ D ( z i , z i +1 ))
Revised: − λ D ( z i , z i +1 ))

Original: as regularizers (we call it augmented loss).
Revised: as regularizers (we call it augmented loss).

Original: Then we drew its test accuracy for diﬀerent γ trained by the augmented loss ( λ is the same as ours) shown in Figure.3(c).
Revised: Then we drew its test accuracy for diﬀerent γ trained by the augmented loss ( λ is the same as ours) shown in Figure.3(c).

Original: With proper γ , POptEqs trained by additional loss can perform better than γ = 0 .
Revised: With proper γ , POptEqs trained by additional loss can perform better than γ = 0 .

Original: Such phenomenon demonstrates the eﬀectiveness of our HH & D Modelling.
Revised: Such phenomenon demonstrates the eﬀectiveness of our HH & D Modelling.

Original: Furthermore, the ﬁgure demonstrates the superiority of our model since the traditional method is consistently worse than ours.
Revised: Furthermore, the ﬁgure demonstrates the superiority of our model since the traditional method is consistently worse than ours.

Original: We left some other explorations for our model in the Appendix due to the space limit.
Revised: We left some other explorations for our model in the Appendix due to the space limit.

Original:  Conclusion
Revised:  Conclusion

Original: We introduce the multi-branch optimization induced equilibrium models (MOptEqs), a new extension of OptEqs that can utilize multiscale information for recognition tasks and retain its ability to recover to an optimization problem whose solution is equivalent to the equilibrium states of our model.
Revised: We introduce the multi-branch optimization induced equilibrium models (MOptEqs), a new extension of OptEqs that can utilize multiscale information for recognition tasks and retain its ability to recover to an optimization problem whose solution is equivalent to the equilibrium states of our model.

Original: The model architecture is designed based on modelling the hidden objective function for the multi- resolution recognition task.
Revised: The model architecture is designed based on modelling the hidden objective function for the multi- resolution recognition task.

Original: Furthermore, we also propose a new training strategy inspired by our understandings of the hidden optimization problem.
Revised: Furthermore, we also propose a new training strategy inspired by our understandings of the hidden optimization problem.

Original: The empirical results show the advantages of our proposed methods
Revised: The empirical results show the advantages of our proposed methods

Original: The success of our HH
Revised: The success of our HH

Original: & D module and
Revised: & D module and

Original: PE training strategy demonstrates the deep link between the optimization problem and neural architecture and may motivate further explorations. 
Revised: PE training strategy demonstrates the deep link between the optimization problem and neural architecture and may motivate further explorations. A Appendix A. Paralleling Multi-Branch OptEqs

Revised: Experiments show that even a simple one-layer OptEqs module enjoys impressive generalization abilities.

Revised: If we view a simple one-layer OptEqs module as a powerful feature generator, we can use an "ensemble" scheme on the DEQs by paralleling several DEQ modules.

Revised: Then we get the multi-branch implicit structure (formulated as Eqn.10), which is capable of utilizing samples of multi-resolution but also retains the capability of recovering to an optimization problem.

Revised: [Equation]

Revised: where z ∗ i denotes the equilibrium outputs of i -th branch, we call this structure ParallelOptEqs (POptEqs).

Revised: Like (Xie et al., 2021) stated for OptEqs, the POptEqs can also be depicted as solving the following problem (Eqn.11) if W i is invertible and σ ( · ) is activation function,

Revised: [Equation]

Revised: [Equation]

Revised: where g ( x ) is the input feature with raw input x , z i ∈

Revised: R d i × 1 and f added for the activation function.

Revised: For example f is the positive orthant f ( x )

Revised: =

Revised: I { x ≥ 0 } if we use ReLU activation, since ( I + ∂f ) − 1 is the ReLU activation function, other activation functions can be found in

Revised: Li et al.

Revised: (2019).The calculation of the above structure (Eqn.10) is equivalent to solving the stationary points ∇ z i G = 0 .

Revised: A. Adversarial Perturbations.

Revised: Adversarial perturbations aim to cheat well-trained neural networks with small or unnoticeable changes on natural images.

Revised: The adversarial perturbations can be obtained by solving the following maximum problem for input x nat and label y :

Revised: [Equation]

Revised: where C is the feasible set of δ in case the change is too large and usually chosen to be an l-inﬁnite ball.

Revised: Due to the neural network is a non-convex function for δ , obtaining the perturbation is not easy.

Revised: The most popular method is the PGD- k method which solves the above problem by implementing the projected gradient method by k times.

Revised: Although the performance of such a method is satisfactory in most cases, it makes the whole training procedure k + 1 times slower than the origin.

Revised: In our work, we replace the above objective function with our model’s hidden objective function.

Revised: Since the hidden optimization problem is convex with respect to the perturbation, we can get the perturbation much easier.

Revised: Moreover, such perturbation can also enhance the generalization abilities, as shown in the experiments.

Revised: A. Proofs for Proposition.

Revised: In this section, we are going to prove that the equilibrium equation Eqn.1 is also the ﬁrst order stationary points for the optimization problem 5.

Revised: Proof 1 The equilibrium equations Eqn.1 in the proposition can be separated into diﬀerent branches as follows:

Revised: [Equation]

Revised: where h i : R d i × R d i +1 ×

Revised: R d i − 1 → R d i can be deﬁned as:

Revised: [Equation]

Revised: where

Revised: [Equation]

Revised: First

Revised: [Equation]

Revised: we need to prove that

Revised: [Equation]

Revised: The derivatives for the right term can be divided as the derivatives for H and D :

Revised: [Equation]

Revised: which is the ﬁrst term of h i .

Revised: Then we are going to calculate the derivatives for D : ∇ z i

Revised: [Equation]

Revised: with the following equation holds for the sampling matrix P i − 1: i and sampling is done for each channel independently:

Revised: [Equation]

Revised: The equation can be formulated as:

Revised: [Equation]

Revised: Since

Revised: [Equation]

Revised: In the same manner

Revised: [Equation]

Revised: we can get,

Revised: [Equation]

Revised: Taking all the terms together

Revised: [Equation]

Revised: we’ve proved the correctness of Eqn.14.

Revised: Then we can proved the relationships between Eqn.12 and the ﬁrst-order stationary points for Eqn.5.

Revised: Take z i for an example:

Revised: [Equation]

Revised: Then we can get the equilibrium equation:

Revised: [Equation]

Revised: the proof is the same for all i .

Revised: In this manner, we proved the relations between the structure and the optimization problems.

Revised: The proof for the proposition is complete

Revised: [Equation]

Revised: A. Proofs for Proposition.

Revised: Proof 2 From the preliminaries above, the diﬀerence between G ( z (cid:48)∗ ; g ( x ) − δ ) and G ( z ∗ ; g ( x )) with δ =

Revised: (cid:15)SIGN ( U (cid:62) W − 1 (cid:62) z ∗ ) is as follows:

Revised: [Equation]

Revised: Since z ∗ is the minimizer of G at g ( x )

Revised: [Equation]

Revised: the above function can be converted as follows:

Revised: [Equation]

Revised: [Table]

Revised: From the structure of OptEqs

Revised: [Equation]

Revised: we can get:

Revised: [Equation]

Revised: The inequality is acquired because of the Lipshitzness of common activation function (ReLU and Leacky ReLU) and the bounds on weight matrices.

Revised: With the above results, we can get:

Revised: [Equation]

Revised: Then we can obtain the proposition:

Revised: [Equation]

Revised: The proofs are similar when considering the HH & D modules.

Revised: A. Forward Process of our MOptEqs’ implicit part.

Revised: The forward propagation is illustrated as follows:

Revised: Algorithm 1: Forward Process of our MOptEqs’ implicit part.

Revised: Require: Input x , initial points { z i } Li =1 , perturbation size (cid:15) , δ 0 = 0 , maximum perturbation size (cid:15) .

Original: {+Ensure: Equilibrium points { z i } Li =
+}
Revised: {+Ensure: Equilibrium points { z i } Li =
+}

Revised: Calculate the input feature g ( x ) .

Revised: The calculation of ∇ ˜

Revised: z G (˜ z ; g ( x ))

Original: {+= 
+}
Revised: {+= 
+}

Revised: is the same as ﬁnding the equilibrium points

Revised: T θ = ∇ ˜

Revised: z G + ˜

Revised: z

Revised: =

Revised: ˜

Revised: z .

Revised: Use Broyden (Broyden, 1965) or Anderson Method (Anderson, 1965) or others to ﬁnd roots of the following equation like MDEQ:

Revised: [Equation]

Revised: z return ˜

Revised: z = { z ∗ i } Li =

Revised: And the calculation of T θ is shown as follows:

Revised: Require:

Revised: Input feature g ( x ) , initial points { z i } Li =1 , perturbation size (cid:15) , δ 0 = 0 , maximum perturbation size (cid:15) .

Revised: Ensure: Output T θ = ∇ ˜

Revised: z G (˜ z ; g ( x ))

Revised: + ˜ z for i ∈ RANGE (1 , L ) do k 1 , k 2 ← neighbor of i in L -loop Calculate z pi by h

Revised: i (Shown in Fig.1 (b)): z pi ← h

Revised: i ( z i , z i − 1 , z i +1 ) if Use PE Method then y i ← σ ( W i z

Revised: pi

Revised: + U

Revised: i ( g ( x ) − δ

Revised: i − 1 )

Revised: + b i ) δ

Revised: i ← (cid:15) SIGN ( U (cid:62) i y i ) else

Revised: y i ← σ ( W i z pi

Revised: + U

Revised: i g ( x ) + b i ) end if z i ← W (cid:62) i

Revised: y i end for return { z ∗ i } Li =

Revised: A. Experiments details of MOptEqs for the experiments Classification. A.6. CIFAR-

Revised: For classiﬁcation tasks of single-view models, we set the channel number of each branch to 32 , λ = 0 .

Revised: 01 and perturbation size (cid:15) = 0 .

Revised: 1 for the small MOptEqs with three branches for the single-view comparison as OptEqs.

Revised: The batch size is set to be 128 for all the experiments.

Revised: As for the multi-scale models’ comparison, we use four branches with each branch take inputs with resolutions equals 32 , 16 , 8 , 4 like MDEQ, the output channel number for each branch is 256 for MOptEqs with 1 . 9 M learnable parameters.

Revised: At the same time, we set the channel number to be 128 for MOptEqs with 0 .

Revised: 48 M learnable parameters.

Revised: And we set λ = 0 .

Revised: 001 and perturbation size (cid:15) = 0 .

Revised: 1 for the experiments.

Revised: We use the widely used SGD algorithm for training procedure.

Revised: We set weight decay to be 0 .

Revised: 001 and the initial learning rate start from 0 .

Revised: 1 and decay by 0 .

Revised: 1 at 100 , 150 , 175 -th epoch with 200 epochs in total like others.

Revised: We use the standard data augmentation for all the experiments.

Revised: Our implementation for small MDEQ only changes the channel number of MDEQ’s 10 M model to obtain the comparable size with our model, without changing the training settings and its hyper-parameters.

Revised: A.6. CIFAR-

Revised: Except for CIFAR- 10 , we also ﬁnish the experiments on CIFAR- 100 classiﬁcation 2 to further verify our model’s eﬀectiveness.

Revised: The model we used is the same as the models used in the multi-scale comparison, only changing the output from 10 to 100 for classiﬁcation.

Revised: MDEQ with 11 M parameter is the same model as (Bai et al., 2020) proposed for CIFARexperiment with only changing the output from 10 to 100 for CIFAR- 100 .

Revised: Moreover, the hyper-parameter setting is also the same as (Bai et al., 2020) proposed in the paper.

Revised: We use the widely used SGD algorithm for training procedure.

Revised: We set weight decay to be 0 .

Revised: 001 and the initial learning rate start from 0 .

Revised: 1 and decay by 0 .

Revised: 1 at 100 , 150 , 175 -th epoch with 200 epochs in total like others.

Revised: We use the standard data augmentation for all the experiments.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: A.6. ImageNette

Revised: Besides the classiﬁcation for images with small size, we also conduct the experiments on the

Revised: Imagenette 3 , which is a subset of 10 classes from ImageNet 4 with 9469 training photos and its test set consists of 3925 images.

Revised: Furthermore, we use the full-size version of Imagenettefurther to verify the superiority of our models on large-scale images.

Revised: The model architecture for our MOptEqs and MDEQ (for comparison) is the same as the ones used in the multi-scale models’ comparison for CIFAR- 10 except adding two downsampling layers in the head of models to downsample the input size from 256 to 64 .

Revised: As for the hyper-parameters, we only change the weight decay to be 5 e − 5 , batch size to be 32 and extend the whole training epochs to 200 with 100 -th, 150 -th, 175 -th for the learning rate decay for all the models.

Revised: All the hyper-parameters for both MOptEqs and MDEQs are hardly changed during all the datasets.

Revised: A. Convergence validation for our MOptEqs.

Revised: As shown in Figure.4(a), the Anderson method can quickly ﬁnd the equilibrium points for our small MOptEq used for single-view comparison.

Revised: From the ﬁgure, one can see that MOptEqs with proper λ (around or smaller than 1 C , around 0 .

Revised: 03 in this circumstance) can ensure the whole mappings be contractive.

Revised: But too large λ will make the model hard to converge or even fail.

Revised: Apart from plotting the convergent iterations’ of our forward root-ﬁnding procedure for the small MOptEqs (shown in Figure.4 (a)), we also draw ﬁgures to show the convergence of the forward propagation for our multi-scale MOptEqs ( λ = 0 .

Revised: 001 ) used in the imagenette and

Revised: CIFAR experiments as Figure.4 (b) shown.

Revised: We use the pre-trained model on Imagenette whose prediction accuracy is reported in the above sections.

Revised: One can see that our multi-scale

Revised: MOptEqs can also quickly converge to the equilibrium points as we expected.

Revised: A. Ablation Studies on the impact of D and H part

Revised: In this section, we analyzed the inﬂuence of our D and H term for a single-branch MOptEq model trained without PE method shown in Table.

Revised: We set λ = 0 .

Revised: 02 for the experiment.

Revised: From the table, one can see that the contribution of the H term is slightly higher than the diversity term.

Revised: But considering both two terms can signiﬁcantly improve the results comparing with MOptEqs without considering these two terms.

Revised: [Table]

Revised: A. Experiments on the ImageNet

Revised: Except CIFAR and Imagenette, we also conducted experiments on ImageNet with 13 M parameters shown as below:

Revised: [Table]

Revised: From the results, one can see that our model can achieve the satisfying performance on

Revised: ImageNet compared with the state-of-the-art models.

Revised: A. PE method and robustness

Revised: From the above experiments, one can see that the PE method may also improve the robustness of our MOptEqs.

Revised: In this section, we conduct experiments to see whether the PE methods can further enhance the robustness of MOptEqs when we use adversarial training methods on these models.

Revised: We ﬁnish comparisons of PGD- 3 training for the single-view MOptEq with or without PE, and the result is shown as follows:

Revised: From the table, we can conclude

Revised: [Table]

Revised: that our PE method can improve the robustness of our MOptEqs model trained by the adversarial training method.

Revised: Such advantages also demonstrate the superiority of our models’ mathematical interpretability since the PE method is acquired based on the model’s hidden optimization problem.

Revised: A. Other Training methods: Fixed Point method and One-step gradients A.11. Methods

Revised: Since our model is not so complicated, we can use the ﬁxed point method for L iteration for the forward propagation.

Revised: While for the back propagation, we can use BPTT (Mozer, 1989) and (Geng et al., 2021) proposed method, which only backward the ﬁnal forward iterations using the chain rule:

Revised: [Equation]

Revised: where θ ∈ Θ are the learnable parameters for the implicit models, T L Θ denotes the output z for the L -th ﬁxed-point iteration.

Revised: Since the back-propagation of such a strategy is similar to backwards only a single layer of DNN module without the calculation of root-ﬁnding algorithm for the Jacobian in the implicit diﬀerentiation, the computational cost is much less than the implicit method.

Revised: However, the gradient approximated by such a method is not accurate and may cause the performance drop.

Revised: However, as we illustrated below, such a method can be regarded as a trade-oﬀ if the computational resources of training are not enough.

Revised: A.11. Empirical Results and Computational Efficiency of One-stpe Gradient

Revised: We take the ImageNette dataset as an example, the training memory cost, forward time, and test accuracy are shown as follows (each batch contains 32 images and is ﬁnished on 2 x

Revised: GTX 1080Ti for 200 epochs):

Revised: Model

Revised: Memory/Batch

Revised: Infer

Revised: [Table]

Revised: /Batch

Revised: [Table]

Revised: MOptEq

Revised: [Table]

Revised: [Table]

Revised: The results shows that the one-step gradient methods can save a lot computation resources while with slightly drop on the performance.