Original: R ESIDUAL NETWORKS CLASSIFY INPUTS BASED ON THEIR NEURAL TRANSIENT DYNAMICS
Revised: R ESIDUAL NETWORKS CLASSIFY INPUTS BASED ON THEIR NEURAL TRANSIENT DYNAMICS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this study, we analyze the input-output behavior of residual networks from a dynamical system point of view by disentangling the residual dynamics from the output activities before the classiﬁcation stage.
Revised: In this study, we analyze the input-output behavior of residual networks from a dynamical system point of view by disentangling the residual dynamics from the output activities before the classiﬁcation stage.

Original: For a network with simple skip connections between every successive layer, and for logistic activation function and shared weights between layers, we show analytically that there is a cooperation and competition dynamics between residuals corresponding to each input dimension.
Revised: For a network with simple skip connections between every successive layer, and for logistic activation function and shared weights between layers, we show analytically that there is a cooperation and competition dynamics between residuals corresponding to each input dimension.

Original: Interpreting these kind of networks as nonlinear ﬁlters, the steady state value of the residuals in the case of attractor networks are indicative of the common features between different input dimensions that the network has observed during training and has encoded in those components.
Revised: Interpreting these kind of networks as nonlinear ﬁlters, the steady state value of the residuals in the case of attractor networks are indicative of the common features between different input dimensions that the network has observed during training and has encoded in those components.

Original: In cases where residuals do not converge to an attractor state, their internal dynamics are separable for each input class, and the network can reliably approximate the output.
Revised: In cases where residuals do not converge to an attractor state, their internal dynamics are separable for each input class, and the network can reliably approximate the output.

Original: We bring analytical and empirical evidence that residual networks classify inputs based on the integration of the transient dynamics of the residuals, and will show how the network responds to input perturbations.
Revised: We bring analytical and empirical evidence that residual networks classify inputs based on the integration of the transient dynamics of the residuals, and will show how the network responds to input perturbations.

Original: We compare the network dynamics for a ResNet and a MultiLayer Perceptron and show that the internal dynamics, and the noise evolution are fundamentally different in these networks, and ResNets are more robust to noisy inputs.
Revised: We compare the network dynamics for a ResNet and a MultiLayer Perceptron and show that the internal dynamics, and the noise evolution are fundamentally different in these networks, and ResNets are more robust to noisy inputs.

Original: Based on these ﬁndings, we also develop a new method to adjust the depth for residual networks during training.
Revised: Based on these ﬁndings, we also develop a new method to adjust the depth for residual networks during training.

Original: As it turns out, after pruning the depth of a
Revised: As it turns out, after pruning the depth of a

Original: ResNet using this algorithm,the network is still capable of classifying inputs with a high accuracy.
Revised: ResNet using this algorithm,the network is still capable of classifying inputs with a high accuracy.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Residual networks (ResNets), ﬁrst introduced in (He et al., 2016), have been more successful in classiﬁcation tasks in comparison with many other standard methods.
Revised: Residual networks (ResNets), ﬁrst introduced in (He et al., 2016), have been more successful in classiﬁcation tasks in comparison with many other standard methods.

Original: This success is attributed to the skip connections between layers that facilitate the propagation of the gradient throughout the network, and in practice allow very deep networks to undergo a successful training.
Revised: This success is attributed to the skip connections between layers that facilitate the propagation of the gradient throughout the network, and in practice allow very deep networks to undergo a successful training.

Original: Apart from mitigating the gradient problem in deep networks, the skip connections introduce a dependency between variables in different layers that can be seen as a system state.
Revised: Apart from mitigating the gradient problem in deep networks, the skip connections introduce a dependency between variables in different layers that can be seen as a system state.

Original: This novelty provides an opportunity for interesting theoretical analysis of their functioning, and has been the underlying pillar for some interesting analysis of such networks from dynamical system point of view (Ciccone et al., 2018;
Revised: This novelty provides an opportunity for interesting theoretical analysis of their functioning, and has been the underlying pillar for some interesting analysis of such networks from dynamical system point of view (Ciccone et al., 2018;

Original: Chang et al., 2018; Haber & Ruthotto, 2017; Lu et al., 2017; Liao & Poggio, 2016; Ruthotto &
Revised: Chang et al., 2018; Haber & Ruthotto, 2017; Lu et al., 2017; Liao & Poggio, 2016; Ruthotto &

Original: Haber, 2018; Chaudhari et al., 2017).
Revised: Haber, 2018; Chaudhari et al., 2017).

Original: This study mainly emphasizes the importance of internal transient dynamics in ResNets on classiﬁcation performance.
Revised: This study mainly emphasizes the importance of internal transient dynamics in ResNets on classiﬁcation performance.

Original: Using a dynamical system approach, for a general ResNet, we derive dynamics of state evolutions of the residuals in different layers.
Revised: Using a dynamical system approach, for a general ResNet, we derive dynamics of state evolutions of the residuals in different layers.

Original: It is well-known that very deep residual networks with weight sharing are equivalent to shallow recurrent neural networks, with similar performance to ResNets with variable weights between layers (Liao & Poggio, 2016).
Revised: It is well-known that very deep residual networks with weight sharing are equivalent to shallow recurrent neural networks, with similar performance to ResNets with variable weights between layers (Liao & Poggio, 2016).

Original: Inspired by this work, we study ResNets with shared weights and sigmoid activation functions, which provide a more tractable mathematical analysis.
Revised: Inspired by this work, we study ResNets with shared weights and sigmoid activation functions, which provide a more tractable mathematical analysis.

Original: Then, we show empirically that those dynamics are observed in ResNets with variable weights as well.
Revised: Then, we show empirically that those dynamics are observed in ResNets with variable weights as well.

Original: To study the classiﬁcation performance of ResNets on noisy inputs, we compare the dynamics of noise evolution in these networks with that of Multi-Layer Perceptrons (MLP), and ﬁnd that for small noise amplitudes, MLP has a higher value of signal to noise ratio.
Revised: To study the classiﬁcation performance of ResNets on noisy inputs, we compare the dynamics of noise evolution in these networks with that of Multi-Layer Perceptrons (MLP), and ﬁnd that for small noise amplitudes, MLP has a higher value of signal to noise ratio.

Original: However, for increased noise amplitude, ResNets are more robust.
Revised: However, for increased noise amplitude, ResNets are more robust.

Original: The main contributions of the paper are:
Revised: The main contributions of the paper are:

Original: We show that network internal dynamics that are shaped by neural transient dynamics are well separated for each input class.
Revised: We show that network internal dynamics that are shaped by neural transient dynamics are well separated for each input class.

Original: We compare the transient dynamics of MLP and ResNets, and show how a noise term in the input evolves in the network, and how it affects the classiﬁcation robustness.
Revised: We compare the transient dynamics of MLP and ResNets, and show how a noise term in the input evolves in the network, and how it affects the classiﬁcation robustness.

Original: Based on residual dynamics, we develop a new method to obtain an adaptive depth for ResNets, during training, for input classiﬁcation.
Revised: Based on residual dynamics, we develop a new method to obtain an adaptive depth for ResNets, during training, for input classiﬁcation.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: In (Haber & Ruthotto, 2017), the authors have studied residual networks using difference equations, and analyzed the stability of the forward propagation of input data, and have linked the inverse problem to the well-posedness of the learning problem.
Revised: In (Haber & Ruthotto, 2017), the authors have studied residual networks using difference equations, and analyzed the stability of the forward propagation of input data, and have linked the inverse problem to the well-posedness of the learning problem.

Original: To circumvent the vanishing or exploding gradient problem, it is suggested in (Haber & Ruthotto, 2017) to design the eigenvalues of the feedforward propagation close to the edge of stability, so that the inverse problem is not ill-posed.
Revised: To circumvent the vanishing or exploding gradient problem, it is suggested in (Haber & Ruthotto, 2017) to design the eigenvalues of the feedforward propagation close to the edge of stability, so that the inverse problem is not ill-posed.

Original: It is however not clear whether having the eigenvalues set close to the edge of stability is beneﬁcial for the network performance, because it depends on the dynamics required by the actual task.
Revised: It is however not clear whether having the eigenvalues set close to the edge of stability is beneﬁcial for the network performance, because it depends on the dynamics required by the actual task.

Original: Employing this idea, the authors in (Chang et al., 2018) suggest a new reversible architecture for neural networks based on Hamiltonian systems.
Revised: Employing this idea, the authors in (Chang et al., 2018) suggest a new reversible architecture for neural networks based on Hamiltonian systems.

Original: Also, in a recent study, ResNets have been employed as an unrolled non-autonomous time-invariant (with weight sharing) system of differential equations (Ciccone et al., 2018), wherein, each ResNet block receives an external input, which depends on the previous block.
Revised: Also, in a recent study, ResNets have been employed as an unrolled non-autonomous time-invariant (with weight sharing) system of differential equations (Ciccone et al., 2018), wherein, each ResNet block receives an external input, which depends on the previous block.

Original: This successive process of feeding the following block by the output of the previous block continues until the latent space converges.
Revised: This successive process of feeding the following block by the output of the previous block continues until the latent space converges.

Original: Our approach in this paper is similar to the aforementioned studies, however, to understand the classiﬁcation mechanism in ResNets, we focus on the role of the intrinsic transient dynamics of the residuals over different layers of the network.
Revised: Our approach in this paper is similar to the aforementioned studies, however, to understand the classiﬁcation mechanism in ResNets, we focus on the role of the intrinsic transient dynamics of the residuals over different layers of the network.

Original: Some studies on ResNets have focused on tracking the features layer by layer (Greff et al., 2017; Chu et al., 2017), and have challenged the idea that deeper layers in neural networks build up abstract features that are different than those formed in lower layers.
Revised: Some studies on ResNets have focused on tracking the features layer by layer (Greff et al., 2017; Chu et al., 2017), and have challenged the idea that deeper layers in neural networks build up abstract features that are different than those formed in lower layers.

Original: One supporting evidence for this challenge comes from lesion studies on ResNets (Veit et al., 2016) and Highway networks (Srivastava et al., 2015) which show that after the network is trained, perturbing the weights in the deep layers does not have a fundamental effect on the network performance, and therefore, does not bring the performance to chance level.
Revised: One supporting evidence for this challenge comes from lesion studies on ResNets (Veit et al., 2016) and Highway networks (Srivastava et al., 2015) which show that after the network is trained, perturbing the weights in the deep layers does not have a fundamental effect on the network performance, and therefore, does not bring the performance to chance level.

Original: However, changing the weights which are closer to initial layers, have more damaging effect.
Revised: However, changing the weights which are closer to initial layers, have more damaging effect.

Original: Empirical studies in (Greff et al., 2017; Chu et al., 2017) suggest an alternative explanation for feature formation in deep layers; that is, successive layers estimate the same features which, along the depth of the network, are more reﬁned, and yield an estimate with smaller standard deviation than earlier layers.
Revised: Empirical studies in (Greff et al., 2017; Chu et al., 2017) suggest an alternative explanation for feature formation in deep layers; that is, successive layers estimate the same features which, along the depth of the network, are more reﬁned, and yield an estimate with smaller standard deviation than earlier layers.

Original: Our study supports this idea by showing that features in different layers of a ResNet with shared weights are formed by the transient dynamics of residuals that may converge towards their steady state values if they are stable.
Revised: Our study supports this idea by showing that features in different layers of a ResNet with shared weights are formed by the transient dynamics of residuals that may converge towards their steady state values if they are stable.

Original: In attractor networks, perturbing the initial layers changes those dynamics more drastically compared to perturbation of deeper layers, because the residuals in the deep layers are either very close to their stable ﬁxed point, or have already converged.
Revised: In attractor networks, perturbing the initial layers changes those dynamics more drastically compared to perturbation of deeper layers, because the residuals in the deep layers are either very close to their stable ﬁxed point, or have already converged.

Original: If there are no attractors for the residuals, sensitivity to initial conditions and the internal dynamics of the residuals play an important role in classiﬁcation.
Revised: If there are no attractors for the residuals, sensitivity to initial conditions and the internal dynamics of the residuals play an important role in classiﬁcation.

Original: In this case, perturbations of the network at initial layers can potentially change the dynamic evolution of the residuals completely, and this will have a more sever impact on the output.
Revised: In this case, perturbations of the network at initial layers can potentially change the dynamic evolution of the residuals completely, and this will have a more sever impact on the output.

Original: Classiﬁcation based on unstable internal states is similar to
Revised: Classiﬁcation based on unstable internal states is similar to

Original: Reservoir networks (Maass et al., 2002), where it has been shown that the high dimensionality of the neurons at the readout layer can compensate for the lack of stability of the neural activities.
Revised: Reservoir networks (Maass et al., 2002), where it has been shown that the high dimensionality of the neurons at the readout layer can compensate for the lack of stability of the neural activities.

Original: Moreover, one important topic in this domain is the depth of ResNets.
Revised: Moreover, one important topic in this domain is the depth of ResNets.

Original: On the one hand, the success of ResNets in classiﬁcation has been attributed to their deep architecture (He et al., 2016), on the other hand, there are studies that claim most of the training is accomplished in the initial layers, and having a very deep architecture is not necessary (Zagoruyko & Komodakis, 2016).
Revised: On the one hand, the success of ResNets in classiﬁcation has been attributed to their deep architecture (He et al., 2016), on the other hand, there are studies that claim most of the training is accomplished in the initial layers, and having a very deep architecture is not necessary (Zagoruyko & Komodakis, 2016).

Original: Another challenge is to understand the generalization property of ResNets (which may be related to its depth), because their power is correlated with their ability in recognizing unseen data that also belong to the classes that these networks have been trained on.
Revised: Another challenge is to understand the generalization property of ResNets (which may be related to its depth), because their power is correlated with their ability in recognizing unseen data that also belong to the classes that these networks have been trained on.

Original: In our analysis of the transient and steady state dynamics of the residuals, we discuss these issues.
Revised: In our analysis of the transient and steady state dynamics of the residuals, we discuss these issues.

Original: In fact, an important ﬁnding of this paper is that residual networks classify inputs based on summing over all the residual’s outputs throughout the network, meaning different transitions of residuals (convergence to their steady state, or their long wandering trajectories without convergence) can potentially change the classiﬁcation result.
Revised: In fact, an important ﬁnding of this paper is that residual networks classify inputs based on summing over all the residual’s outputs throughout the network, meaning different transitions of residuals (convergence to their steady state, or their long wandering trajectories without convergence) can potentially change the classiﬁcation result.

Original: Interestingly, in biological neuronal networks, it has been suggested that optimal stimulus separation in neurons that encode sensory information occurs during the transients rather than the ﬁxed points of the neuronal activity trajectories (Mazor & Laurent, 2005; Rabinovich et al., 2008).
Revised: Interestingly, in biological neuronal networks, it has been suggested that optimal stimulus separation in neurons that encode sensory information occurs during the transients rather than the ﬁxed points of the neuronal activity trajectories (Mazor & Laurent, 2005; Rabinovich et al., 2008).

Original: Also, it has been discussed that spatio-temporal processing in cortical circuits are state dependent, and the role of transients are crucial (Buonomano & Maass, 2009).
Revised: Also, it has been discussed that spatio-temporal processing in cortical circuits are state dependent, and the role of transients are crucial (Buonomano & Maass, 2009).

Original: In our study we show that also in ResNets, these transients are the decisive factors for classiﬁcation.
Revised: In our study we show that also in ResNets, these transients are the decisive factors for classiﬁcation.

Original: Based on this ﬁnding, we show how a noise term in the input is mapped to the output, and we develop a new method to control the depth of ResNets.
Revised: Based on this ﬁnding, we show how a noise term in the input is mapped to the output, and we develop a new method to control the depth of ResNets.

Original:  D YNAMICS OF INTERACTIONS BETWEEN RESIDUALS
Revised:  D YNAMICS OF INTERACTIONS BETWEEN RESIDUALS

Original: We consider a dense ResNet with N input dimensions, and arbitrary T layers with exactly N neurons at each layer.
Revised: We consider a dense ResNet with N input dimensions, and arbitrary T layers with exactly N neurons at each layer.

Original: A unique property of a ResNet that distinguishes it from conventional feedforward networks (such as MLP) is the skip connection between layers.
Revised: A unique property of a ResNet that distinguishes it from conventional feedforward networks (such as MLP) is the skip connection between layers.

Original: In the ResNet we consider here, the activity of neuron i at layer t , before the output of the previous layer ( t − 1) is added to it, is represented as y i ( t ) , and the activity of all neurons in the same layer is represented by the vector y ( t ) .
Revised: In the ResNet we consider here, the activity of neuron i at layer t , before the output of the previous layer ( t − 1) is added to it, is represented as y i ( t ) , and the activity of all neurons in the same layer is represented by the vector y ( t ) .

Original: After the integration of the output from layer t − 1 , the output of layer t is represented by x ( t ) .
Revised: After the integration of the output from layer t − 1 , the output of layer t is represented by x ( t ) .

Original: The components of these residuals y ( t ) are calculated based on a linear function of x ( t ) , i.e. z i ( t ) =
Revised: The components of these residuals y ( t ) are calculated based on a linear function of x ( t ) , i.e. z i ( t ) =

Original: (cid:80) Ni =1 w ij ( t ) x
Revised: (cid:80) Ni =1 w ij ( t ) x

Original: j ( t ) followed by a nonlinear function f
Revised: j ( t ) followed by a nonlinear function f

Original: ( z i ) .
Revised: ( z i ) .

Original: Figure 1 illustrates the relation between x and y .
Revised: Figure 1 illustrates the relation between x and y .

Original: Any hidden layer t represents a sample of the dynamical states x after t steps.
Revised: Any hidden layer t represents a sample of the dynamical states x after t steps.

Original: Input data is considered as the initial condition of the system, and is depicted by x (0) .
Revised: Input data is considered as the initial condition of the system, and is depicted by x (0) .

Original: Interpreting the network as a dynamical system which evolves throughout the layers, the dynamics of neural activations are x ( t + 1) =
Revised: Interpreting the network as a dynamical system which evolves throughout the layers, the dynamics of neural activations are x ( t + 1) =

Original: x ( t ) + y ( t + 1) , where y ( t ) is the output of the neurons, and in the rest of the paper, they are called ”residuals”.
Revised: x ( t ) + y ( t + 1) , where y ( t ) is the output of the neurons, and in the rest of the paper, they are called ”residuals”.

Original: This equation implies a difference equation for the variable x ( t ) , that is x ( t + 1) − x ( t ) =
Revised: This equation implies a difference equation for the variable x ( t ) , that is x ( t + 1) − x ( t ) =

Original: y ( t + 1) .
Revised: y ( t + 1) .

Original: The left side of this equation resembles the forward
Revised: The left side of this equation resembles the forward

Original: Euler method of derivative of a continuous system, when the discretization step is equal to 1 .
Revised: Euler method of derivative of a continuous system, when the discretization step is equal to 1 .

Original: This approximates a continuous system with dynamics that follow ˙ x i ( t ) =
Revised: This approximates a continuous system with dynamics that follow ˙ x i ( t ) =

Original: y i ( t ) .
Revised: y i ( t ) .

Original: We are interested in understanding how the neural activities evolve over layers in a feedforward fully connected ResNet.
Revised: We are interested in understanding how the neural activities evolve over layers in a feedforward fully connected ResNet.

Original: The dynamics of x ( t ) and inputs to the residuals, z ( t ) , follow
Revised: The dynamics of x ( t ) and inputs to the residuals, z ( t ) , follow

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst line of equation 1 indicates that x ( t ) stores the sum of the residuals and the input ( x (0) ) from the input layer up to layer t , meaning that x ( t ) is a cumulative signal for y ( t ) , as well as the input data.
Revised: The ﬁrst line of equation 1 indicates that x ( t ) stores the sum of the residuals and the input ( x (0) ) from the input layer up to layer t , meaning that x ( t ) is a cumulative signal for y ( t ) , as well as the input data.

Original: For a ResNet with shared weights between layers, ˙ w ij ( t ) = 0 ; because the weights do not change between layers.
Revised: For a ResNet with shared weights between layers, ˙ w ij ( t ) = 0 ; because the weights do not change between layers.

Original: This constraint makes the analysis simpler, and results in ˙
Revised: This constraint makes the analysis simpler, and results in ˙

Original: z i ( t ) =
Revised: z i ( t ) =

Original: (cid:80) Nj =1 w ij ˙ x j ( t ) .
Revised: (cid:80) Nj =1 w ij ˙ x j ( t ) .

Original: In this case, after replacing ˙ x i ( t ) by y i ( t ) in equation 1, the dynamics of z ( t ) and the residuals y ( t ) will be
Revised: In this case, after replacing ˙ x i ( t ) by y i ( t ) in equation 1, the dynamics of z ( t ) and the residuals y ( t ) will be

Original: [Equation]
Revised: [Equation]

Original: A steady state solution for y i , in a network with a time invariant weight w ij (shared weights across layers), is obtained from setting either of the two terms on the right hand side of equation 2 to zero.
Revised: A steady state solution for y i , in a network with a time invariant weight w ij (shared weights across layers), is obtained from setting either of the two terms on the right hand side of equation 2 to zero.

Original: For a general nonlinear f ( . ) , according to equation 2, the interactions between the residuals are either competitive or cooperative, depending on the positive or negative inﬂuence that they have on each other.
Revised: For a general nonlinear f ( . ) , according to equation 2, the interactions between the residuals are either competitive or cooperative, depending on the positive or negative inﬂuence that they have on each other.

Original: For a logistic function f ( . ) , the derivative is y i ( t )(1 − y
Revised: For a logistic function f ( . ) , the derivative is y i ( t )(1 − y

Original: i
Revised: i

Original: ( t )) , which results in a particular form of predator-prey equation, well-known in studying ecosystems.
Revised: ( t )) , which results in a particular form of predator-prey equation, well-known in studying ecosystems.

Original: In this case, equation 2 yields
Revised: In this case, equation 2 yields

Original: [Equation]
Revised: [Equation]

Original: For N residuals, the number of possible ﬁxed points (solutions of equation 3) is 3 N .
Revised: For N residuals, the number of possible ﬁxed points (solutions of equation 3) is 3 N .

Original: However, for a given weight W , only some of the ﬁxed points are stable.
Revised: However, for a given weight W , only some of the ﬁxed points are stable.

Original: The initial condition for the residuals is determined by the output of the ﬁrst layer.
Revised: The initial condition for the residuals is determined by the output of the ﬁrst layer.

Original: After some transients over the next layers, each residual converges to its stable solution, if there is one.
Revised: After some transients over the next layers, each residual converges to its stable solution, if there is one.

Original: Note that the derivatives at y i = 0 or 1 are equal to zero, and the system’s trajectories are conﬁned to this space.
Revised: Note that the derivatives at y i = 0 or 1 are equal to zero, and the system’s trajectories are conﬁned to this space.

Original: According to equation 1 and ﬁgure 1, the cumulative of the residuals over the entire network feeds the classiﬁer.
Revised: According to equation 1 and ﬁgure 1, the cumulative of the residuals over the entire network feeds the classiﬁer.

Original: Therefore, perturbations at initial layers of the network disrupt the output more severely than perturbations of the deeper layers (lesion studies have considered weight perturbations, which reﬂects on neural activity perturbations).
Revised: Therefore, perturbations at initial layers of the network disrupt the output more severely than perturbations of the deeper layers (lesion studies have considered weight perturbations, which reﬂects on neural activity perturbations).

Original: A discrete-time analysis of the effect of perturbations at different layers on the output value is given in the Appendix.
Revised: A discrete-time analysis of the effect of perturbations at different layers on the output value is given in the Appendix.

Original: To study the contribution of input noise ( ξ ) in the ﬁnal hidden layer, we derive the noise evolution for the ResNet considered above (a similar analysis applies to ResNets with continuous activation functions).
Revised: To study the contribution of input noise ( ξ ) in the ﬁnal hidden layer, we derive the noise evolution for the ResNet considered above (a similar analysis applies to ResNets with continuous activation functions).

Original: Represented in vector notations, the signal plus noise evolution is obtained by replacing y i in equation equation 3 by y + ξ .
Revised: Represented in vector notations, the signal plus noise evolution is obtained by replacing y i in equation equation 3 by y + ξ .

Original: [Equation]
Revised: [Equation]

Original: Assuming that y ( t ) is the solution of the neural activity at layer t for the input signal without noise (that is reﬂected in initial condition
Revised: Assuming that y ( t ) is the solution of the neural activity at layer t for the input signal without noise (that is reﬂected in initial condition

Original: y (1) ), subtracting equation 3 from equation 4 yields
Revised: y (1) ), subtracting equation 3 from equation 4 yields

Original: [Equation]
Revised: [Equation]

Original: In equation 5 and equation 4, the vectors y and ξ are functions of t .
Revised: In equation 5 and equation 4, the vectors y and ξ are functions of t .

Original: This equation indicates how the growth rate of the noise ( ˙ ξ ) differs from the growth rate of the residuals ( ˙ y ).
Revised: This equation indicates how the growth rate of the noise ( ˙ ξ ) differs from the growth rate of the residuals ( ˙ y ).

Original:  N EURAL DYNAMICS IN MLP
Revised:  N EURAL DYNAMICS IN MLP

Original: An MLP can also be interpreted as a dynamical system.
Revised: An MLP can also be interpreted as a dynamical system.

Original: Given that in a feedforward MLP, y ( t + 1) =
Revised: Given that in a feedforward MLP, y ( t + 1) =

Original: f ( z ) , to derive the neural state dynamics, a − y ( t ) term can be added to both sides of this equation.
Revised: f ( z ) , to derive the neural state dynamics, a − y ( t ) term can be added to both sides of this equation.

Original: Using the same argument as in ResNets, the internal dynamics of MLP can be written as
Revised: Using the same argument as in ResNets, the internal dynamics of MLP can be written as

Original: [Equation]
Revised: [Equation]

Original: For a small noise, the last term in the equation above can be approximated by a Taylor expansion.
Revised: For a small noise, the last term in the equation above can be approximated by a Taylor expansion.

Original: Keeping only up to ﬁrst order terms of this expression results in
Revised: Keeping only up to ﬁrst order terms of this expression results in

Original: [Equation]
Revised: [Equation]

Original: The last equation indicates that for small noise amplitudes, the noise is always suppressed as it propagates in the network.
Revised: The last equation indicates that for small noise amplitudes, the noise is always suppressed as it propagates in the network.

Original: The reason is that the elements of ˙
Revised: The reason is that the elements of ˙

Original: y
Revised: y

Original: + y are always negative (property of the sigmoid function).
Revised: + y are always negative (property of the sigmoid function).

Original: However, for larger noise amplitudes, higher order terms in the Taylor approximation will be needed.
Revised: However, for larger noise amplitudes, higher order terms in the Taylor approximation will be needed.

Original: This fact is nicely reﬂected in the noise to signal ratio of the MLP network for small noise perturbations in ﬁgure 4.
Revised: This fact is nicely reﬂected in the noise to signal ratio of the MLP network for small noise perturbations in ﬁgure 4.

Original: Including the second order perturbation terms to the Taylor expansion, results in
Revised: Including the second order perturbation terms to the Taylor expansion, results in

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS ON MNIST
Revised:  E XPERIMENTS ON MNIST

Original: To study the behavior of the network on large datasets, we considered a network with 1064 sigmoid neurons in each layer, and 15 layer deep.
Revised: To study the behavior of the network on large datasets, we considered a network with 1064 sigmoid neurons in each layer, and 15 layer deep.

Original: First, we analyzed a case where the weight matrix was
Revised: First, we analyzed a case where the weight matrix was

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: shared between all the layers.
Revised: shared between all the layers.

Original: The input data was chosen from MNIST, and the classiﬁcation was performed by using a softmax function.
Revised: The input data was chosen from MNIST, and the classiﬁcation was performed by using a softmax function.

Original: In this case, the classiﬁcation error on the test set was 1 .
Revised: In this case, the classiﬁcation error on the test set was 1 .

Original: Note that the network considered here is the simplest possible network architecture (so as to allow us to understand the classiﬁcation mechanism in ResNets) with shared weights; therefore, the results are not comparable to the state of the art performance on MNIST.
Revised: Note that the network considered here is the simplest possible network architecture (so as to allow us to understand the classiﬁcation mechanism in ResNets) with shared weights; therefore, the results are not comparable to the state of the art performance on MNIST.

Original: As illustrated in ﬁgure 2 top panel, the residuals in the ﬁrst few layers are still in the transition period (non-zero standard deviation for 200 random samples from 10 classes).
Revised: As illustrated in ﬁgure 2 top panel, the residuals in the ﬁrst few layers are still in the transition period (non-zero standard deviation for 200 random samples from 10 classes).

Original: We used the same weights in a network with 1000 hidden layers (without retraining) to study the behavior of the residuals in a deep realization of the same
Revised: We used the same weights in a network with 1000 hidden layers (without retraining) to study the behavior of the residuals in a deep realization of the same

Original: ResNet.
Revised: ResNet.

Original: There were a few non-zero ﬁxed points with some negligible standard deviation among 200 samples.
Revised: There were a few non-zero ﬁxed points with some negligible standard deviation among 200 samples.

Original: To check if the ﬁxed points were different and distinguished for each class, we plotted the average and the standard deviation of the residuals for the test set, separately for each class, on the ﬁnal hidden layer in ﬁgure 5 in the appendix.
Revised: To check if the ﬁxed points were different and distinguished for each class, we plotted the average and the standard deviation of the residuals for the test set, separately for each class, on the ﬁnal hidden layer in ﬁgure 5 in the appendix.

Original: The average for each class is different from any other class, however, a large number of dimensions are identical.
Revised: The average for each class is different from any other class, however, a large number of dimensions are identical.

Original: The standard deviation between the residuals in a single class are small, and negligible, and indicate that the classiﬁcation accuracy is not 100% .
Revised: The standard deviation between the residuals in a single class are small, and negligible, and indicate that the classiﬁcation accuracy is not 100% .

Original: For this prolonged simulation, we obtained the eigenvalue distribution for the average residuals for each class at layer 1000 .
Revised: For this prolonged simulation, we obtained the eigenvalue distribution for the average residuals for each class at layer 1000 .

Original: Residuals corresponding to classes 0 , 2 , 3 had a single small positive eigenvalue (around 0.02) among all other negative eigenvalues (indication of the existence of saddle point).
Revised: Residuals corresponding to classes 0 , 2 , 3 had a single small positive eigenvalue (around 0.02) among all other negative eigenvalues (indication of the existence of saddle point).

Original: This means that those classes are still in their transition period at layer 1000 , and due to the small value of the positive eigenvalue, the transition is slow.
Revised: This means that those classes are still in their transition period at layer 1000 , and due to the small value of the positive eigenvalue, the transition is slow.

Original: Due to the high dimensionality of the network, it is not viable to illustrate the transition dynamics of individual neurons separately for each class.
Revised: Due to the high dimensionality of the network, it is not viable to illustrate the transition dynamics of individual neurons separately for each class.

Original: However, to show different transition patterns of the residuals in each class, we chose one neuron for each class such that the classiﬁer had the highest sensitivity to the value of the cumulative transitions of that neuron.
Revised: However, to show different transition patterns of the residuals in each class, we chose one neuron for each class such that the classiﬁer had the highest sensitivity to the value of the cumulative transitions of that neuron.

Original: The index of this neuron was derived from the sensitivity of each class C with respect to x ( T ) , which is the classiﬁer K .
Revised: The index of this neuron was derived from the sensitivity of each class C with respect to x ( T ) , which is the classiﬁer K .

Original: For each output in the softmax layer, there exists a maximally sensitive weight for its corresponding classiﬁer K .
Revised: For each output in the softmax layer, there exists a maximally sensitive weight for its corresponding classiﬁer K .

Original: This method renders 10 different indices.
Revised: This method renders 10 different indices.

Original: In ﬁgure 3A, we plotted the average (over 1000 samples for each class) of the residuals for neurons that corresponded to those indices.
Revised: In ﬁgure 3A, we plotted the average (over 1000 samples for each class) of the residuals for neurons that corresponded to those indices.

Original: In almost all cases (apart from class 8 ), the maximum value of x (15) belonged to the neuron that had the largest coefﬁcient in the classiﬁer vector for that particular class.
Revised: In almost all cases (apart from class 8 ), the maximum value of x (15) belonged to the neuron that had the largest coefﬁcient in the classiﬁer vector for that particular class.

Original: This implies that separation between classes are encoded in the transient dynamics of those neurons (and other neurons that their cumulative trajectories are multiplied by big coefﬁcients in the classiﬁer).
Revised: This implies that separation between classes are encoded in the transient dynamics of those neurons (and other neurons that their cumulative trajectories are multiplied by big coefﬁcients in the classiﬁer).

Original: The transient dynamics of these neurons play an important role in the classiﬁcation result.
Revised: The transient dynamics of these neurons play an important role in the classiﬁcation result.

Original: To show that these transient dynamics are different and separable for each input class, we projected the trajectories of the residuals to a 2D plane, and observed a clear separation between the internal dynamics for each class (Appendix C).
Revised: To show that these transient dynamics are different and separable for each input class, we projected the trajectories of the residuals to a 2D plane, and observed a clear separation between the internal dynamics for each class (Appendix C).

Original: In a different experiment, we investigated the behavior of a similar ResNet, with 15 layers, but with variable weight matrix for each layer.
Revised: In a different experiment, we investigated the behavior of a similar ResNet, with 15 layers, but with variable weight matrix for each layer.

Original: The mean and standard deviations of the residual forsamples are illustrated in ﬁgure 2, bottom panel.
Revised: The mean and standard deviations of the residual forsamples are illustrated in ﬁgure 2, bottom panel.

Original: In this case, the residuals converge to their steady state solutions already on the fourth layer, as their standard deviations across samples converge to zero after the fourth layer.
Revised: In this case, the residuals converge to their steady state solutions already on the fourth layer, as their standard deviations across samples converge to zero after the fourth layer.

Original: A striking ﬁnding in this case is that the standard deviation of the residuals for samples from different classes are zero, meaning that only one stable ﬁxed point encodes all the similarities between different input classes.
Revised: A striking ﬁnding in this case is that the standard deviation of the residuals for samples from different classes are zero, meaning that only one stable ﬁxed point encodes all the similarities between different input classes.

Original: Considering this fact, we conclude that the sum of transient dynamics across layers for different input classes converges to different outputs that discriminate the inputs.
Revised: Considering this fact, we conclude that the sum of transient dynamics across layers for different input classes converges to different outputs that discriminate the inputs.

Original: Another interesting observation is that at the few layers close to the output layer, the weight matrix between layers converged to a ﬁxed matrix.
Revised: Another interesting observation is that at the few layers close to the output layer, the weight matrix between layers converged to a ﬁxed matrix.

Original: Also, compared to the previous example of a network with weight sharing, there are more residuals that converge to nonzero values.
Revised: Also, compared to the previous example of a network with weight sharing, there are more residuals that converge to nonzero values.

Original: This gives the network enough capacity to give divergent outputs for different classes, based on their initial conditions.
Revised: This gives the network enough capacity to give divergent outputs for different classes, based on their initial conditions.

Original: In this example, the classiﬁcation error on the test set was about 1 . 8% .
Revised: In this example, the classiﬁcation error on the test set was about 1 . 8% .

Original: This higher value of the error rate might be due the paucity of separate ﬁxed points to represent each class.
Revised: This higher value of the error rate might be due the paucity of separate ﬁxed points to represent each class.

Original: Considering the observation that a ResNet with multiple ﬁxed points for the residuals, corresponding to different classes, renders a smaller classiﬁcation error, hints to the point that having different similarity representations of the input encoded in the residuals results in a better generalization, compared to cases where only one single ﬁxed point for residuals stands for the entire input classes.
Revised: Considering the observation that a ResNet with multiple ﬁxed points for the residuals, corresponding to different classes, renders a smaller classiﬁcation error, hints to the point that having different similarity representations of the input encoded in the residuals results in a better generalization, compared to cases where only one single ﬁxed point for residuals stands for the entire input classes.

Original: For the MLP network applied on MNIST dataset, the residuals show an oscillatory dynamics, and no convergence at layer 1000 was observed (ﬁgure 2 bottom panel).
Revised: For the MLP network applied on MNIST dataset, the residuals show an oscillatory dynamics, and no convergence at layer 1000 was observed (ﬁgure 2 bottom panel).

Original: To compare the robustness of the ResNet and MLP network to noise in the input data, we injected a uniformly distributed noise ξ of different amplitude to the initial conditions.
Revised: To compare the robustness of the ResNet and MLP network to noise in the input data, we injected a uniformly distributed noise ξ of different amplitude to the initial conditions.

Original: We calculated the noise to signal ratio ( {+(cid:80) Tt = 
+} ξ ( T ) / x ( T ) ) for the ResNet, and ( ξ ( T ) / y ( T ) ) for the ResNet, for different classes with 10 independent realizations of the input noise.
Revised: We calculated the noise to signal ratio ( {+(cid:80) Tt = 
+} ξ ( t ) / x ( T ) ) for the ResNet, and ( ξ ( T ) / y ( T ) ) for the MLP network, for different classes with 10 independent realizations of the input noise.

Original: The average noise to signal ratio is depicted in ﬁgure 4A. For a maximum noise amplitude less than 0 .
Revised: The average noise to signal ratio is depicted in ﬁgure 4A. For a maximum noise amplitude less than 0 .

Original: 1 , the noise to signal ratio in the MLP network is smaller than that of the ResNet, which means that MLP is able to suppress the noise term better, and will render a bigger cosine similarity in the last hidden layer.
Revised: 1 , the noise to signal ratio in the MLP network is smaller than that of the ResNet, which means that MLP is able to suppress the noise term better, and will render a bigger cosine similarity in the last hidden layer.

Original: However, for a maximum noise amplitude in the range of 0 . 1 and 1 . , the MLP fails to cancel out the noise term, and the noise to signal ratio grows quickly as a function of noise amplitude.
Revised: However, for a maximum noise amplitude in the range of 0 . 1 and 1 . , the MLP fails to cancel out the noise term, and the noise to signal ratio grows quickly as a function of noise amplitude.

Original: Already at 0 . 1 maximum noise amplitude, the network misclassiﬁes the noisy input signals.
Revised: Already at 0 .maximum noise amplitude, the network misclassiﬁes the noisy input signals.

Original: The ResNet, however, shows a higher noise to signal ratio for small amplitudes of noise, and the misclassiﬁcation occurs at maximum amplitudes that are larger than 0 .
Revised: The ResNet, however, shows a higher noise to signal ratio for small amplitudes of noise, and the misclassiﬁcation occurs at maximum amplitudes that are larger than 0 .

Original: This implies more robustness of ResNets to input perturbations.
Revised: This implies more robustness of ResNets to input perturbations.

Original: As a suggestion to increase the robustness of ResNets even further, it is possible to add a regularization term to the cost function of the network which includes the integration of the right hand side of equation 5 for the last two hidden layers (this could be applied on the weights of the last hidden layer in a general network with variable weights, and with a continuous activation function.)
Revised: As a suggestion to increase the robustness of ResNets even further, it is possible to add a regularization term to the cost function of the network which includes the integration of the right hand side of equation 5 for the last two hidden layers (this could be applied on the weights of the last hidden layer in a general network with variable weights, and with a continuous activation function.)

Original:  A DAPTIVE DEPTH FOR R ES N ETS DURING TRAINING
Revised:  A DAPTIVE DEPTH FOR R ES N ETS DURING TRAINING

Original: Results of the previous sections shed some light on the mechanism of classiﬁcation in ResNets.
Revised: Results of the previous sections shed some light on the mechanism of classiﬁcation in ResNets.

Original: After understanding the role of transient dynamics in input classiﬁcation, we envisage a new method to design the depth of ResNets based on the layer-dependent behavior of the residuals.
Revised: After understanding the role of transient dynamics in input classiﬁcation, we envisage a new method to design the depth of ResNets based on the layer-dependent behavior of the residuals.

Original: In this method, during training, initially an arbitrary number of layers is chosen.
Revised: In this method, during training, initially an arbitrary number of layers is chosen.

Original: After training each epoch using the back-propagation algorithm, the difference between the residuals for the last successive layers of the
Revised: After training each epoch using the back-propagation algorithm, the difference between the residuals for the last successive layers of the

Original: ResNet block are calculated.
Revised: ResNet block are calculated.

Original: If this difference is less than a minimum threshold (we chose 0 .
Revised: If this difference is less than a minimum threshold (we chose 0 .

Original: 01 for each neuron on average), the last hidden layer in the block is to be removed, because the value of the residuals will not contribute much to the cumulative function.
Revised: 01 for each neuron on average), the last hidden layer in the block is to be removed, because the value of the residuals will not contribute much to the cumulative function.

Original: This process continues until the network is trained (minimum loss on the training set).
Revised: This process continues until the network is trained (minimum loss on the training set).

Original: Note that convergence of the residuals is not a necessary requirement for classiﬁcation, but a sufﬁcient condition; i.e. when the residuals converge, and when the training loss is minimum, there is no need for extra layers in the blocks (and also before
Revised: Note that convergence of the residuals is not a necessary requirement for classiﬁcation, but a sufﬁcient condition; i.e. when the residuals converge, and when the training loss is minimum, there is no need for extra layers in the blocks (and also before

Original: [Figure]
Revised: [Figure]

Original: the classiﬁcation layer).
Revised: the classiﬁcation layer).

Original: This algorithm can be implemented as a piece of code in parallel with other training algorithms for ResNets: while loss function is not minimum do for each epoch of the training data do residuals of the last hidden layer in the block → rresiduals of the second last hidden layer in the block → rif l 1 norm for r 1 − r 2 < threshold then remove the layer corresponding to rend end end
Revised: This algorithm can be implemented as a piece of code in parallel with other training algorithms for ResNets: while loss function is not minimum do for each epoch of the training data do residuals of the last hidden layer in the block → rresiduals of the second last hidden layer in the block → rif l 1 norm for r 1 − r 2 < threshold then remove the layer corresponding to rend end end

Original: In the examples shown in the previous sections, we demonstrated a converging behavior of the residuals to stable or metastable (saddle ﬁxed points) states.
Revised: In the examples shown in the previous sections, we demonstrated a converging behavior of the residuals to stable or metastable (saddle ﬁxed points) states.

Original: For a fully connected network with variable weights between layers, we applied our algorithm during training on MNIST dataset.
Revised: For a fully connected network with variable weights between layers, we applied our algorithm during training on MNIST dataset.

Original: The algorithm was able to shrink the network to 3 layers, and the error on the test set was 1 .
Revised: The algorithm was able to shrink the network to 3 layers, and the error on the test set was 1 .

Original: 7% (similar to the network performance with 15 layers).
Revised: 7% (similar to the network performance with 15 layers).

Original: Also, for a network with shared weights between layers, the network depth was reduced to 5 layers without any changes in classiﬁcation accuracy.
Revised: Also, for a network with shared weights between layers, the network depth was reduced to 5 layers without any changes in classiﬁcation accuracy.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this study, we showed that given an input, ResNets integrate samples of the residuals from each layer, and build an output representation for the input data in the ﬁnal hidden layer.
Revised: In this study, we showed that given an input, ResNets integrate samples of the residuals from each layer, and build an output representation for the input data in the ﬁnal hidden layer.

Original: This sum depends on the initial condition (input data) and its transition towards the steady state of the corresponding residual.
Revised: This sum depends on the initial condition (input data) and its transition towards the steady state of the corresponding residual.

Original: In some networks which show attracting and converging behavior, one or more stable ﬁxed point for the residuals exists.
Revised: In some networks which show attracting and converging behavior, one or more stable ﬁxed point for the residuals exists.

Original: In other cases, among many other possible dynamics, multiple ﬁxed points for different input classes might exist, some of which could be stable or metastable.
Revised: In other cases, among many other possible dynamics, multiple ﬁxed points for different input classes might exist, some of which could be stable or metastable.

Original: In both cases, different neural transient dynamics (with inputs of different classes as initial conditions) can result in different cumulative values of the residuals, and therefore, different classiﬁcation outcomes.
Revised: In both cases, different neural transient dynamics (with inputs of different classes as initial conditions) can result in different cumulative values of the residuals, and therefore, different classiﬁcation outcomes.

Original: Using a dynamical system interpretation of the networks, we compared internal dynamics of an MLP network with that of a ResNet on MNIST dataset, and we showed ResNets are more robust to signal perturbations.
Revised: Using a dynamical system interpretation of the networks, we compared internal dynamics of an MLP network with that of a ResNet on MNIST dataset, and we showed ResNets are more robust to signal perturbations.

Original: We also developed a new method for designing an adaptive depth for ResNets during training.
Revised: We also developed a new method for designing an adaptive depth for ResNets during training.

Original: The main idea is that after all the residuals have settled into their steady state value, or if there are negligible changes of the values of the residuals between successive layers, there is no need for any extra deeper layers.
Revised: The main idea is that after all the residuals have settled into their steady state value, or if there are negligible changes of the values of the residuals between successive layers, there is no need for any extra deeper layers.

Original: This is because any additional layer of the residual neurons will add almost the same values as the previous layer, without any extra information about the neural transitions.
Revised: This is because any additional layer of the residual neurons will add almost the same values as the previous layer, without any extra information about the neural transitions.

Original:  A PPENDIX 8. A: S ENSITIVITY ANALYSIS OF THE OUTPUT WITH RESPECT TO LAYER PERTURBATIONS
Revised:  A PPENDIX 8. A: S ENSITIVITY ANALYSIS OF THE OUTPUT WITH RESPECT TO LAYER PERTURBATIONS

Original: As mentioned in the Introduction, some lesion studies have shown that weight perturbations at the initial layers of the network can have more sever consequences on output classiﬁcation results than perturbations of the weights at deeper layers.
Revised: As mentioned in the Introduction, some lesion studies have shown that weight perturbations at the initial layers of the network can have more sever consequences on output classiﬁcation results than perturbations of the weights at deeper layers.

Original: Since we have considered a network with shared weights,we study the effect of perturbations of the residuals (which could be considered as the result of weight perturbations in previous studies).
Revised: Since we have considered a network with shared weights,we study the effect of perturbations of the residuals (which could be considered as the result of weight perturbations in previous studies).

Original: To understand how slight perturbations of the residuals affects the values of the output, we analyzed the sensitivity of the output C with respect to slight perturbations of the residuals y ( t ) .
Revised: To understand how slight perturbations of the residuals affects the values of the output, we analyzed the sensitivity of the output C with respect to slight perturbations of the residuals y ( t ) .

Original: Assuming slight perturbations on y ( s ) , we are interested in the evolution of this perturbation throughout the network and its effect on the output unit.
Revised: Assuming slight perturbations on y ( s ) , we are interested in the evolution of this perturbation throughout the network and its effect on the output unit.

Original: We used the chain rule of differentiation on the discrete time dynamics of the network to obtain a sensitivity matrix that propagates the perturbations from y ( t ) , layer to layer, until it reaches the output C (the analysis is in discrete case).
Revised: We used the chain rule of differentiation on the discrete time dynamics of the network to obtain a sensitivity matrix that propagates the perturbations from y ( t ) , layer to layer, until it reaches the output C (the analysis is in discrete case).

Original: The sensitivity equation reads
Revised: The sensitivity equation reads

Original: [Equation]
Revised: [Equation]

Original: where t = T is the last hidden layer.
Revised: where t = T is the last hidden layer.

Original: Based on the deﬁnition of x ( t ) , it is easy to verify that ∂ x ( t ) ∂
Revised: Based on the deﬁnition of x ( t ) , it is easy to verify that ∂ x ( t ) ∂

Original: x
Revised: x

Original: ( t − 1)
Revised: ( t − 1)

Original: =
Revised: =

Original: I + ∂ y ( t ) ∂
Revised: I + ∂ y ( t ) ∂

Original: x ( t − 1) .
Revised: x ( t − 1) .

Original: We represent ∂ y ( t ) ∂
Revised: We represent ∂ y ( t ) ∂

Original: x ( t − 1) by M ( t ) , which is ∂ f
Revised: x ( t − 1) by M ( t ) , which is ∂ f

Original: ( W x ( t − 1))
Revised: ( W x ( t − 1))

Original: ∂
Revised: ∂

Original: x ( t − 1) .
Revised: x ( t − 1) .

Original: Using vector representations, M = y ( t ) (cid:12) (1 − y ( t ))
Revised: Using vector representations, M = y ( t ) (cid:12) (1 − y ( t ))

Original: (cid:12) W , where (cid:12) represents element-wise multiplications.
Revised: (cid:12) W , where (cid:12) represents element-wise multiplications.

Original: Since the output class C is the result of the inner product between the classiﬁer vector K and the cumulative signal x ( T ) , ∂ C ∂ x ( T ) =
Revised: Since the output class C is the result of the inner product between the classiﬁer vector K and the cumulative signal x ( T ) , ∂ C ∂ x ( T ) =

Original: K .
Revised: K .

Original: It is also clear that the last term on the right hand side of equation 9 is equal to the identity matrix.
Revised: It is also clear that the last term on the right hand side of equation 9 is equal to the identity matrix.

Original: Therefore, in equation 9, the sensitivity matrix can be represented as
Revised: Therefore, in equation 9, the sensitivity matrix can be represented as

Original: [Equation]
Revised: [Equation]

Original: For different layers s 1 and s 2 , where s 2 > s 1 , we calculated M (cid:48)
Revised: For different layers s 1 and s 2 , where s 2 > s 1 , we calculated M (cid:48)

Original: ( s 1 ) and M (cid:48) ( s 2 ) .
Revised: ( s 1 ) and M (cid:48) ( s 2 ) .

Original: It turns out that for network simulations that we performed (see Experiment section), for s 2 = 10 and s 1 = 4 , the determinant of M (cid:48) ( s 1 ) is orders of magnitude larger than the determinant of M (cid:48)
Revised: It turns out that for network simulations that we performed (see Experiment section), for s 2 = 10 and s 1 = 4 , the determinant of M (cid:48) ( s 1 ) is orders of magnitude larger than the determinant of M (cid:48)

Original: ( s 2 ) .
Revised: ( s 2 ) .

Original: This implies perturbations at initial layers are greatly ampliﬁed at the ﬁnal hidden layer compared to perturbations at close to ﬁnal layers.
Revised: This implies perturbations at initial layers are greatly ampliﬁed at the ﬁnal hidden layer compared to perturbations at close to ﬁnal layers.

Original: Since the ﬁnal hidden layer is multiplied by the classiﬁer vector K , those ampliﬁed perturbations will have a more disruptive consequence on the output of the network.
Revised: Since the ﬁnal hidden layer is multiplied by the classiﬁer vector K , those ampliﬁed perturbations will have a more disruptive consequence on the output of the network.

Original: 8. B: L ONG - TIME BEHAVIOR OF THE R ESIDUALS ON MNIST DATASET
Revised: 8. B: L ONG - TIME BEHAVIOR OF THE R ESIDUALS ON MNIST DATASET

Original: Initially, we trained a ResNet with 15 layers with shared weights between layers.
Revised: Initially, we trained a ResNet with 15 layers with shared weights between layers.

Original: To observe the long-time behavior of the residuals in this network, we used the same weights in a network with 1000 hidden layers (without retraining).
Revised: To observe the long-time behavior of the residuals in this network, we used the same weights in a network with 1000 hidden layers (without retraining).

Original: There were a few non-zero ﬁxed points with some negligible standard deviation among 200 samples.
Revised: There were a few non-zero ﬁxed points with some negligible standard deviation among 200 samples.

Original: To check if the ﬁxed points were different and distinguished for each class, we plotted the average and the standard deviation of the residuals for the test set, separately for each class, on the ﬁnal hidden layer in ﬁgure 5.
Revised: To check if the ﬁxed points were different and distinguished for each class, we plotted the average and the standard deviation of the residuals for the test set, separately for each class, on the ﬁnal hidden layer in ﬁgure 5.

Original: The average for each class is different from any other class, however, a large number of dimensions are identical.
Revised: The average for each class is different from any other class, however, a large number of dimensions are identical.

Original: The fact that each class has a distinguished ﬁxed point indicates that the trajectories of the residuals for each class are separated.
Revised: The fact that each class has a distinguished ﬁxed point indicates that the trajectories of the residuals for each class are separated.

Original: According to ﬁgure 5, for the ResNet, the standard deviation between the residuals in a single class are small, and negligible, and indicate that the classiﬁcation accuracy is not 100% .
Revised: According to ﬁgure 5, for the ResNet, the standard deviation between the residuals in a single class are small, and negligible, and indicate that the classiﬁcation accuracy is not 100% .

Original: For the MLP network, the representations at layer 1000 are more diverse, and the standard deviations of this state for different input classes is high.
Revised: For the MLP network, the representations at layer 1000 are more diverse, and the standard deviations of this state for different input classes is high.

Original: This reﬂects the fact that the activities of the neurons in this network did not converge to a ﬁxed value (in fact, the neural dynamics are oscillatory).
Revised: This reﬂects the fact that the activities of the neurons in this network did not converge to a ﬁxed value (in fact, the neural dynamics are oscillatory).

Original: 8. C: I NTERNAL REPRESENTATIONS FOR R ES N ET AND MLP
Revised: 8. C: I NTERNAL REPRESENTATIONS FOR R ES N ET AND MLP

Original: To illustrate the role of internal transient dynamics in input classiﬁcation, we mapped the residual signals from a 1064 dimensional state space to a 2 dimensional distance space, using the Umap algorithm Mcinnes & Healy (2018).
Revised: To illustrate the role of internal transient dynamics in input classiﬁcation, we mapped the residual signals from a 1064 dimensional state space to a 2 dimensional distance space, using the Umap algorithm Mcinnes & Healy (2018).

Original: This algorithm provides a dimensionality reduction technique based on Riemannian geometry which preserves more of the global structure, and has a superior run time performance compared to t − SNE Maaten & Hinton (2008).
Revised: This algorithm provides a dimensionality reduction technique based on Riemannian geometry which preserves more of the global structure, and has a superior run time performance compared to t − SNE Maaten & Hinton (2008).

Original: As shown in ﬁgure 6A, the
Revised: As shown in ﬁgure 6A, the

Original: [Figure]
Revised: [Figure]

Original: transient dynamics that correspond to different input classes build up clusters in a 2D space that are well separated from each other.
Revised: transient dynamics that correspond to different input classes build up clusters in a 2D space that are well separated from each other.

Original: A similar representation holds for reservoir networks that are used for analyzing time-dependent inputs Jaeger & Haas (2004); Buonomano & Maass (2009).
Revised: A similar representation holds for reservoir networks that are used for analyzing time-dependent inputs Jaeger & Haas (2004); Buonomano & Maass (2009).

Original: For the ResNet with shared weight between layers, we calculated the average cosine similarity between the ﬁnal hidden layers of 200 samples per each class.
Revised: For the ResNet with shared weight between layers, we calculated the average cosine similarity between the ﬁnal hidden layers of 200 samples per each class.

Original: The cosine similarity is shown in a symmetric matrix in ﬁgure 6B, where in each row i the element in column i has a higher values.
Revised: The cosine similarity is shown in a symmetric matrix in ﬁgure 6B, where in each row i the element in column i has a higher values.

Original: This means more similarity between outputs of the last hidden layer for inputs of the same class.
Revised: This means more similarity between outputs of the last hidden layer for inputs of the same class.

Original: The same calculation was performed for the MLP network (ﬁgure 6C), and in this case, the cosine similarity for the last hidden representation of inputs from the same class is bigger, and the difference between classes is more pronounced than in the ResNet scenario.
Revised: The same calculation was performed for the MLP network (ﬁgure 6C), and in this case, the cosine similarity for the last hidden representation of inputs from the same class is bigger, and the difference between classes is more pronounced than in the ResNet scenario.

Original: A    8 Class
Revised: A    8 Class

Original: C l a ss 0 .0 .0 .0 .0 .0 .0 .0 .0 .
Revised: C l a ss 0 .0 .0 .0 .0 .0 .0 .0 .0 .

Original: 96 B    8 Class
Revised: 96 B    8 Class

Original: C l a ss 0 .0 .0 .0 .0 .0 .0 .0 .0 .
Revised: C l a ss 0 .0 .0 .0 .0 .0 .0 .0 .0 .

Original: C
Revised: C