  In this study, we analyze the input-output behavior of residual networks from
a dynamical system point of view by disentangling the residual dynamics from
the output activities before the classification stage. For a network with
simple skip connections between every successive layer, and for logistic
activation function, and shared weights between layers, we show analytically
that there is a cooperation and competition dynamics between residuals
corresponding to each input dimension. Interpreting these kind of networks as
nonlinear filters, the steady state value of the residuals in the case of
attractor networks are indicative of the common features between different
input dimensions that the network has observed during training, and has encoded
in those components. In cases where residuals do not converge to an attractor
state, their internal dynamics are separable for each input class, and the
network can reliably approximate the output. We bring analytical and empirical
evidence that residual networks classify inputs based on the integration of the
transient dynamics of the residuals, and will show how the network responds to
input perturbations. We compare the network dynamics for a ResNet and a
Multi-Layer Perceptron and show that the internal dynamics, and the noise
evolution are fundamentally different in these networks, and ResNets are more
robust to noisy inputs. Based on these findings, we also develop a new method
to adjust the depth for residual networks during training. As it turns out,
after pruning the depth of a ResNet using this algorithm,the network is still
capable of classifying inputs with a high accuracy.
