I TERATIVE CONVERGENT COMPUTATION IS NOT A USEFUL INDUCTIVE BIAS FOR R ES N ETS
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent work has suggested that feedforward residual neural networks (ResNets) approximate iterative recurrent computations. Iterative computations are useful in many domains, so they might provide good solutions for neural networks to learn. Here we quantify the degree to which ResNets learn iterative solutions and introduce a regularization approach that encourages learning of iterative solutions. Iterative methods are characterized by two properties: iteration and convergence.
To quantify these properties, we deﬁne three indices of iterative convergence. Consistent with previous work, we show that, even though ResNets can express iterative solutions, they do not learn them when trained conventionally on computer vision tasks. We then introduce regularizations to encourage iterative convergent computation and test whether this provides a useful inductive bias. To make the networks more iterative, we manipulate the degree of weight sharing across layers using soft gradient coupling.
This new method provides a form of recurrence regularization and can interpolate smoothly between an ordinary ResNet and a “recurrent” ResNet (i.e., one that uses identical weights across layers and thus could be physically implemented with a recurrent network computing the successive stages iteratively across time).
To make the networks more convergent we impose a Lipschitz constraint on the residual functions using spectral normalization. The three indices of iterative convergence reveal that the gradient coupling and the Lipschitz constraint succeed at making the networks iterative and convergent, respectively.
However, neither recurrence regularization nor spectral normalization improve classiﬁcation accuracy on standard visual recognition tasks (MNIST, CIFAR-10, CIFAR-100) or on challenging recognition tasks with partial occlusions (Digitclutter).
Iterative convergent computation, in these tasks, does not provide a useful inductive bias for ResNets.
I NTRODUCTION
An iterative method solves a difﬁcult estimation or optimization problem by starting from an initial guess and repeatedly applying a transformation that is known to improve the estimate, leading to a sequence of estimates that converges to the solution.
Iterative methods provide a powerful approach to ﬁnding exact or approximate solutions where direct methods fail (e.g., for difﬁcult inverse problems or solutions to systems of equations that are nonlinear and/or large).
Recurrent neural networks (RNNs) iteratively apply the same transformation to their internal representation, suggesting that they may learn algorithms similar to the iterative methods used in mathematics and engineering.
The idea of iterative reﬁnement of a representation has also driven recent progress in the context of feedforward networks.
New architectures, based on the idea of iterative reﬁnement, have allowed for the training of very deep feedforward models with hundreds of layers.
Prominent architectures for achieving high depth are residual (ResNets; He et al., 2016a) and highway networks (Srivastava et al., 2015), which use skip connections to drive the network to learn the residual: a pattern of adjustments to the input, thus encouraging the model to learn successive reﬁnements of a representation of the input that is shared across layers.
These architectures combine two ideas. The ﬁrst is to use skip connections to alleviate the problem of vanishing or exploding gradients (Hochreiter, 1991).
The second is to make these skip connections ﬁxed identity connections, such that the layers learn successive reﬁnement of a shared repre- sentational format. The second idea relates residual and highway networks to RNNs and iterative methods.
Learning a single transformation that can be iteratively applied is attractive because it enables trading speed for accuracy by iterating longer (Spoerer et al., 2019). In addition, a preference for an iterative solution may provide a useful inductive bias for certain computational tasks.
However, it is unclear whether ResNets indeed learn solutions akin to iterative methods and, if they do, whether this is a useful inductive bias.
The two deﬁning features of iterative methods are (1) iteration and (2) convergence.
Here we analyze to what extent these features emerge in ResNets.
In order to investigate whether these features provide a useful inductive bias, we introduce two simple modiﬁcations of classical ResNets and study their impact on a number of datasets.
First, we study CIFAR-10, CIFAR-100 (Krizhevsky, 2009), and MNIST (LeCun et al., 2010) as examples of classical vision tasks, assessing the networks’ performance and sample efﬁciency.
Since iterative and convergent inductive biases may be more useful for tasks that require some degree of recurrence, we also assess the networks’ performance and sample efﬁciency on several variations of Digitclutter , a task which requires the recognition of multiple digits that occlude each other (Spoerer et al., 2017).
To study the effect of iteration, we manipulate the degree of weight sharing in ResNets, smoothly interpolating between ordinary and recurrent ResNets.
We ﬁnd that a higher degree of weight sharing tends to make the network more iterative, but does not result in improved performance or sample efﬁciency.
This suggests that in ordinary ResNets, recurrent connections do not provide a useful inductive bias and the networks can harness the additional computational ﬂexibility provided by non-recurrent residual blocks. Recurrence implies iteration, but not convergence, and so is not sufﬁcient for a network to implement an iterative method as deﬁned above. ResNets, whether they are recurrent (i.e. sharing weights across layers) or not, are therefore neither required nor encouraged to converge during training. We demonstrate empirically that ResNets in general do not exhibit convergent behavior and that recurrent ResNets are more convergent than non-recurrent networks.
To study the effect of convergence, we upper bound the residual blocks’ Lipschitz constant.
This modiﬁcation adversely impacts performance, suggesting that the non-convergent behavior in ordinary ResNets is not merely due to lack of incentive, but underpins the networks’ high performance. Across convergent ResNets, a higher degree of weight sharing does not negatively affect performance.
This suggests that convergent ResNets, in contrasts to non-convergent ones, do not beneﬁt from the increased computational ﬂexibility of non-recurrent residual blocks. Taken together, our results suggest that an inductive bias favoring an iterative convergent solution does not outweigh the computational ﬂexibility of non-recurrent residual blocks for the considered tasks.
R ELATED WORK
Prior theoretical work has focused on explaining the success of ResNets (He et al., 2016a) and the more general class of highway networks (Srivastava et al., 2015) by studying the learning dynamics in ResNets (Hochreiter, 1991; Orhan & Pitkow, 2018;
Balduzzi et al., 2017) and their interpretation as an ensemble of shallow networks (Veit et al., 2016; Huang et al., 2018), as a discretized dynamical system (E, 2017; Haber & Ruthotto, 2018; E et al., 2019), and as performing iterative reﬁnement.
The iterative reﬁnement hypothesis.
Our work builds on Jastrzebski et al.
(2018) who argue that the sequential application of the residual blocks in a ResNet iteratively reﬁnes a representational estimate.
Their work builds on observations that dropping out residual blocks, shufﬂing their order, or evaluating the last block several times retains reasonable performance (Veit et al., 2016; Greff et al., 2017) and can be used for training (Huang et al., 2016).
Another set of methods uses such perturbations to train deep neural networks, using stochastic depth (Huang et al., 2016;
Hu et al., 2019; Press et al., 2020).
Other methods learn to evaluate a limited number of layers that depends on the input (Graves, 2017; Figurnov et al., 2017) or learn increasingly ﬁne-grained object categories across layers (Zamir et al., 2017).
Instead of using perturbations to encourage stability of the trained network, Ciccone et al.
(2018) propose a method inspired by dynamical systems theory to guarantee such stability in their model.
Iterative reﬁnement and inverse problems.
The iterative reﬁnement hypothesis is particularly important in the context of inverse problems, which are often solved using iterative methods.
This is particularly relevant for ResNets trained for perceptual tasks since perception is often conceptualized as an inverse problem (Poggio et al., 1985; Pizlo, 2001).
Rao & Ballard (1999) modeled visual cortex using recurrent neural networks that iteratively infer the latent causes of a hierarchical Bayesian model.
Though these networks have been applied to complex datasets (Wen et al., 2018), most models either learn the inverse model (Rezende et al., 2014; Kingma & Welling, 2013) or deﬁne an analytically invertible forward model (Rezende & Mohamed, 2015; Gomez et al., 2017).
A notable exception are invertible ResNets (Behrmann et al., 2019), whose inverse can be computed through a ﬁxed point iteration.
Another set of works starts out with a classical iterative algorithm and unfolds this algorithm to create a deep network, approaching a similar problem from the opposite direction (Gregor & LeCun, 2010; Hershey et al., 2014; Wisdom et al., 2016; 2017).
Moreover, Nguyen et al.
(2019) reﬁne CNNs
to yield an inference algorithm on a speciﬁc generative model (recently implemented by Huang et al., 2020).
Recurrent and residual neural networks.
The idea of iterative reﬁnement has motivated an increasing number of recurrent convolutional neural networks being applied to computer vision without an explicit implementation of iterative inference (Kubilius et al., 2019; Spoerer et al., 2019).
ResNets may be seen as RNNs that have been unfolded for a ﬁxed number of iterations.
Sharing weights between the different blocks allows us to train a recurrent residual neural network (Liao & Poggio, 2016).
In this framework, recurrent residual neural networks may be seen as a special case of residual neural networks where the weights are equal between all blocks.
Savarese & Maire (2019) and Oh et al. (2019) relaxed this constraint by deﬁning the weights of the different blocks as a linear combination of a smaller set of underlying weights. The work by Savarese & Maire (2019) is particularly interesting in this context, as their method yielded a more efﬁcient parameterization of Wide ResNets (Zagoruyko & Komodakis, 2017), which generally have more channels, but far fewer layers than the architectures we consider here. Inductive bias of recurrent operations on visual tasks.
The inverse problem perspective on perception suggests that ordinary recognition tasks in computer vision may already beneﬁt from an iteratively convergent inductive bias.
For other tasks, we have additional reasons to believe that recurrent processing may be beneﬁcial.
For example, object recognition in the primate ventral stream can be well captured by feedforward models (Hung et al., 2005; Majaj et al., 2015) but beneﬁts form recurrent processing under challenging conditions.
This includes tasks where the presented objects are partially occluded or degraded (Wyatte et al., 2014) or tasks that involve perceptual grouping according to local statistical regularities (Roelfsema et al., 1999) or object-level information (Vecera &
Farah, 1997).
These observations have inspired tasks such as Digitclutter and Digitdebris (Spoerer et al., 2017) as well as Pathﬁnder and cABC
(Kim et al., 2020).
For these datasets, adding recurrent networks have been shown to outperform corresponding feedforward control models (Spoerer et al., 2017; Linsley et al., 2020b)).
Implicit recurrent computations.
Beyond potentially useful inductive bias, recurrent neural networks can provide additional computational advantages.
If the recurrent operation converges to a ﬁxed point, this ﬁxed point can be determined more efﬁciently by classical iterative algorithms such as ODE solvers (Chen et al., 2018).
Moreover, in this case, recurrent backpropagation (Pineda, 1987; Almeida, 1990) can compute the parameters’ gradients much more efﬁciently than backpropagation through time (Werbos, 1988) because it does not require storing the intermediate activation.
Its memory cost is therefore constant in depth.
This method has inspired several promising models in computer vision.
Deep equilibrium models (Bai et al., 2019) harness a quasi-Newton method to ﬁnd the ﬁxed point of their recurrent operation.
They have recently been applied to object recognition and image segmentation tasks (Bai et al., 2020).
These works empirically demonstrated the existence of a ﬁxed point under their parameterization.
Other models enforce such a ﬁxed point by the Lipschitz constraints we here employ as well.
Ciccone et al.
(2018) use an upper bound based on the Frobenius norm, whereas Linsley et al.
(2020a)
approximate the Lipschitz constant using a vector-Jacobian product.
Our method (see below) is based on Yoshida & Miyato (2017) and has recently been employed (for different purposes) by Miyato et al.
(2018) and Behrmann et al.
D O R ES N ETS IMPLEMENT ITERATIVE COMPUTATIONS ?
Iterative methods are characterized by two key properties: iteration (i.e., recurrent computation) and convergence.
In this section, we examine whether residual neural networks have these proper- xxxxz^z^●●●●●●●●● ● ● ● ● ● ●●●●●●●●●●●●●●●●●● −   − 1  b x ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●●● 0.
x ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.8 0.
z ^1.3 1.4 1.5 1.
z^c x
●● ●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●● 0.4 0.
z ^1.01.21.41.61.
z^d
[Figure]
[Figure]
ties.
We start by showing that ResNets can express iterative algorithms , but then demonstrate that
ResNets do not automatically learn iterative algorithms .
We show that this observation extends to large-scale neural networks trained on real data.
Finally, we introduce a paradigm that allows us to compare neural networks to iterative methods.
3. R ES N ETS CAN REPRESENT ITERATIVE COMPUTATIONS
A popular application for iterative methods is given by inverse problems of the form x = f ( z ) .
It is often not possible to directly compute the inverse, f − 1 , of the forward model, because of analytical or combinatorial intractability.
Instead, approximations (Gershman & Goodman, 2014) or iterative error-corrective algorithms (Donoho, 2006) are used.
Consider the linear forward model x = f ( z ) :
= ( α 1 z 1 , α 2 z 2 ) .
Though f has an analytical inverse, it serves as an illustrative example.
Based on an input x , we can infer the latent variable z using the iterative update
[Equation]
where ˆ z ( T ) , for some T , is the estimate of z and (cid:15) i > 0 should be sufﬁciently small 1 .
This update can be implemented in a small ResNet.
Figure 1a contains a schematic illustration of one block of this network.
The representation spanned by an encoder in the beginning of the network contains an input representation x and a representation of the current estimate z .
In the hidden layer of the residual block, we determine the positive and negative prediction errors and use them to update z :
[Equation]
This recurrent residual building block would implement an iteratively convergent computation in a ResNet.
The linear model is, of course, a trivial case, but serves as an illustration of the appeal of this approach.
A wide neural network is a ﬂexible function approximator and learning to represent the prediction error instead of the prediction is easier in many cases (Zamir et al., 2017).
3. R ES N ETS DO NOT AUTOMATICALLY LEARN ITERATIVE COMPUTATIONS
The fact that ResNets can express iterative computations does not imply that they necessarily learn iterative solutions when trained via backpropagation.
Here we consider the simple example from above to better understand the behavior that may emerge.
We highlight three behaviors that distinguish iterative from non-iterative computations and will examine the behavior of large-scale neural networks in the following sections.
As an example, we [Figure] train a conventional ResNet, and a ResNet that uses the same weights in each block (equivalent to a recurrent network) to invert the function f ( z ) :=
( 32 z 1 , 34 z 2 ) .
We contrast their behavior with the iterative error-corrective algorithm outlined above.
Due to the lack of constraints, a non-residual feedforward neural network changes its representation in every layer.
As a consequence, the linear decoder at the end of the network is not aligned to the intermediate representation and early readout (as depicted by the orange dots in Fig. 1b) leads to a meaningless estimate.
In contrast, the skip connections encourage a ResNet to use the same representational format across blocks.
This is to say that its intermediate representations are better aligned with the ﬁnal decoder.
Early readout is therefore possible and the representation across blocks will approach the ﬁnal estimate.
As a consequence, the across-block dynamics of the nonresidual network are meaningless, whereas the recurrent and residual network’s early readouts are close to the ﬁnal estimate and approach it in a smooth manner, just like the error-corrective algorithm (Figure 1b).
Since iterative methods iteratively reﬁne their initial estimate, their behavior is more similar to the ResNets’ monotonic convergence.
Aside from their smooth convergence, a fundamental property of iterative methods is their recurrence (i.e., the repeated use of the same computation).
This means that dropping out an earlier block has the same effect as dropping out the last one.
We can relax the requirement for exact repetition (weight sharing) and require merely similar computations.
Figure 1c illustrates that the trained
ResNet is indeed relatively robust to block dropout.
Yet the learned models fall short of an iterative method, which is apparent from a third mode of investigation.
Iterative convergence would imply that applying the last block’s transformation iteratively should keep the readout in the vicinity of the actual estimate.
This is clearly not the case in our toy example (Figure 1d).
Rather than representing a convergent estimate, this result is more compatible with understanding ResNets as approaching their ﬁnal estimate at a constant speed and, in the case of late readout, moving past this estimate and overshooting at the same speed.
Notably, both the non-recurrent and recurrent networks exhibit this behavior.
3. I TERATIVE C ONVERGENCE I NDICES
This behavior is not surprising.
After all, the network is trained to work for a ﬁxed number of steps and not constrained to stay within the vicinity of its ﬁnal estimate if more steps are added.
However, it reveals that even recurrent ResNets do not automatically learn iterative convergent computations.
To assess the extent to which they do learn iterative convergent computations we deﬁne three continuous indices, which measure convergence, recurrence, and divergence (deﬁned below).
We evaluate the indices for six instances of ResNet-101, trained on CIFAR-10 (Krizhevsky, 2009).
This ResNet consists of three stages of 16 residual blocks with 16, 32, and 64 channels, respectively.
The ResNet achieved 5.2% classiﬁcation error on the validation set.
To characterize the extent to which the ResNets have learned an iterative convergent computation, we introduce three indices measuring different aspects of such computations.
Convergence Index.
Viewing ResNets as performing iterative reﬁnement suggests that each stage gradually approaches its ﬁnal estimate before passing this estimate to the next stage using a downsampling layer.
By passing the estimate at each of the residual blocks to the next stage, we can monitor how the stages approach their ﬁnal estimate across blocks (see Figure 2a, left panel).
In accordance with previous results (Jastrzebski et al., 2018), we ﬁnd that all stages smoothly approach their ﬁnal estimate, conﬁrming the earlier intuition of a shared representational format.
To measure the rate of convergence, we compute the area under the curve (AUC) of the classiﬁcation error, which we call the Convergence Index.
We invert and normalize this value such that a Convergence
Index of 0 corresponds chance level read-out at each residual block, whereas a Convergence Index of 1 corresponds to an instant convergence to the ﬁnal classiﬁcation error at the ﬁrst residual block.
Figure 2b, left panel, depicts this value for each stage, and averaged across stages.
Recurrence Index.
To measure the degree of recurrence, we evaluated the effect of dropping out individual blocks on the error rate of the network (see Figure 2a, middle panel).
In a non-recurrent
ResNet, dropping out earlier blocks may have a stronger effect on the error rate than dropping out the last block.
In contrast, in a recurrent ResNet, the effect on error rate is the same for dropping out either earlier blocks or the last block.
We therefore computed the difference in error rate observed after these two manipulations.
We summarized the behavior by the AUC, which we refer to as
Recurrence Index (RI).
We invert and normalize this value such that the RI is 0 if dropping out any block leads to an error rate at chance level and the RI is 1 in the case of a recurrent algorithm.
Even though we study non-recurrent ResNets, dropping out any block other than the very ﬁrst leads to a negligible drop in performance, replicating previous results (Jastrzebski et al., 2018; Veit et al., 2016).
As a consequence, the RI, across all stages, is close to 1.0 (see Figure 2b, middle panel).
Divergence Index.
ResNets may either converge to their ﬁnal estimate or simply approach it in a sequence of steps.
An iterative algorithm should not be negatively affected by additional applications of the same function.
To examine this property, we apply the last block of each stage for an additional up to sixteen steps (see Figure 2a, right panel) and determine the AUC (Divergence Index,
DI, see Figure 2b, right panel).
We ﬁnd that no stage is particularly robust to such additional evaluations, though the ﬁrst stage has the lowest DI, indicating that it is the most robust.
This suggests that ResNets approach and move away from their ﬁnal estimate in a sequence of steps, with their computations bearing little similarity to an iterative convergent algorithm.
A high DI does not indicate that the ResNet has failed in some way.
After all, it was not trained to be robust to such perturbations.
However, it indicates that the ResNet may not implement an iterative convergent computation.
M ANIPULATING CONVERGENCE AND ITERATION IN RESIDUAL NETWORKS
We provided indices measuring convergence, recurrence, and divergence to assess the degree to which a ResNet implements an iterative method.
Even though they are able to, ResNets do not necessarily learn to implement a purely iterative method.
In particular, they show divergent behavior.
Nevertheless, as we have shown above, their behavior does show some similarity to iterative methods and their success has been attributed to these similarities (Greff et al., 2017; Jastrzebski et al., 2018).
This suggests that even though the parameterization and optimization does not promote the emergence of an iterative method in a ResNet, a ResNet with iteratively convergent behavior may still have a better inductive bias.
To test this hypothesis, we therefore here control the inductive bias, namely recurrence and convergence, of ResNets.
4. S OFT GRADIENT COUPLING CAN INTERPOLATE SMOOTHLY BETWEEN ORDINARY AND RECURRENT OPTIMIZATION
We propose a method to blend between recurrent and non-recurrent networks without changing the architecture or the loss landscape.
The method is motivated by the observation that we can train a recurrent neural network by sharing the different blocks’ gradients (Jastrzebski et al., 2018; Liao & Poggio, 2016).
In ordinary ResNets, the residual block t with the weights W t is changed by following the gradient ∆ t = ∂ W t L , whereas RNNs impose as gradient
[Equation]
where the weights across residual blocks within a stage must start from the same initialization.
The former means that we do not employ any inductive bias towards recurrence, whereas the latter imposes a possibly overly restrictive function space on the architecture.
To address both limitations, we propose soft gradient coupling , which uses as its update rule
[Equation]
For λ < 1 , this retains the entire space of computations enabling both non-recurrent as well as recurrent computations.
However, for λ > 0 , the optimization is biased to ﬁnd more recurrent optima.
In contrast to penalty regularizations or strict weight sharing models (Oh et al., 2019; Savarese &
Maire, 2019), this does not change the network or loss landscape, but simply the accessibility of different local minima of the loss landscape.
For networks with coupled gradients (i. e., 0 < λ ≤ 1 ), we initialize their weights recurrently (i.e., all residual blocks within one stage share the same initialization).
We unshare batch norm statistics as suggested by Jastrzebski et al.
(2018), but leave their parameters (softly) coupled.
4. S PECTRAL NORMALIZATION CAN GUARANTEE CONVERGENCE IN RESIDUAL NETWORKS
Iterative methods preserve a stable output when applied repeatedly. In contrast, the output of the ResNets diverged when the last block was applied repeatedly beyond the number of steps it was trained for.
In order to control the degree of convergence in a ResNet, we constrain the Lipschitz constant L of the residual function f . L is deﬁned as the minimal value such that for any input x , y , (cid:107) f ( x ) − f ( y ) (cid:107) ≤ L (cid:107) x − y (cid:107) . The smaller L , the more stable f .
The Lipschitz constant is hard to determine accurately as it is a global property of f .
We therefore determine an upper bound ˆ
L ( f ) ≥ L based on the linear operations within f (see section E.1). Using this upper bound, we replace f by its spectral normalization
[Equation]
for a certain value µ . If ˆ L ( f ) ≤ µ , the corrective factor c will not change the function. If ˆ L ( f ) > µ , it will set the corresponding upper bound of ˜ f at ˆ L ( ˜ f ) = µ , constraining the residual function’s
Lipschitz constant. For a given input x , a recurrent residual stage is deﬁned by the iterative application z 0 : = x , z t := R ( z t − 1 ) , where R deﬁnes the recurrently applied residual block.
We wish to guarantee that z t eventually converges to a ﬁxed point z ∞ and hope that empirically, z T , the representation after the speciﬁed number of iterations, will be close to this ﬁxed point. According to the Banach ﬁxed point theorem, one way to guarantee such convergence is to require that the residual block’s Lipschitz constant L R be smaller than one. We can achieve this by replacing the residual connections between adjacent blocks by residual connections between the input x to the stage and each block, i. e. ˜ R ( z ) : = x + ˜ f
( z ) . ˜ R has the same
Lipschitz constant as ˜ f . Setting µ < 1 , we therefore guarantee that ˜ R converges to a ﬁxed point deﬁned by x . We call this network the properly convergent ResNet (PCR). To get a network more similar to an ordinary ResNet, we consider R ( z ) :
= z + ˜ f ( z ) .
Though we convergence is not guaranteed for the network deﬁned by this residual block, we show its empirical convergence.
We thus call this network the improperly convergent ResNet (ICR).
[Figure]
c
[Figure]
●●●
[Figure]
[-(non-rec.) 
-]
[-(rec.) (rec.) (non-rec.) 
 (rec.)0.5 0.-]●●●
[Figure]
E XPERIMENTS
To assess our hypotheses, we considered non-recurrently initialized (ordinary) ResNets as well as recurrently initialized ResNets with coupling parameters 0, 0.5, 0.9, and 1. {+In addition, we considered properly and improperly convergent ResNets across the same coupling parameters, setting µ = 
+}
.(we only trained these networks on CIFAR-10). We trained several instances of all these ResNets with 8, 16, 32, and 64 channels in the ﬁrst stage on classical visual recognition tasks (CIFAR-10,
MNIST, and CIFAR-100) as well as Digitclutter , a challenging task with partially occluded objects, which has previously been observed to beneﬁt from recurrent connections (Spoerer et al
5. S OFT GRADIENT COUPLING IMPROVES I TERATIVE R ECURRENCE I NDICES
As Figure 3a shows, soft gradient coupling indeed improves iterative convergence, increasing the Convergence Index and decreasing the Divergence Index.
The Recurrence Index is centered closely around 1.
Convergence and Divergence Index, on the other hand, tend to increase and decrease, respectively, both with higher coupling parameters and with a higher number of channels.
Notably, this trend appears to not hold up for a fully recurrent ResNet, corresponding to a coupling parameter of 1.
These results show that soft gradient coupling is an effective way of manipulating a ResNet’s behavior.
Increasing weight similarity across blocks leads to more iterative convergence in a ResNet.
5. S PECTRAL NORMALIZATION MAKES R ES N ETS CONVERGENT
Both the properly and improperly convergent ResNets have a high Convergence Index as well as a Divergence Index at almost zero (see Fig. 4a for ICRs and section E.2 for PCRs). The Recurrence
Index is again centered around one.
For improperly convergent ResNets with 16 or 32 channels in the ﬁrst stage, higher coupling parameters generally have a lower Convergence Index and a higher
Divergence Index.
Nevertheless, these indices indicate that the spectrally normalized ResNets exhibit much more convergent behavior than the ordinary networks.
This was only guaranteed for the recurrent, properly convergent ResNets and is therefore an important observation.
5. S TRONGER ITERATIVE CONVERGENCE DOES NOT PROVIDE A USEFUL INDUCTIVE BIAS
We ﬁrst assessed the effect of gradient coupling on the performance of non-convergent ResNets.
As Fig. 3b shows, a higher coupling parameter consistently leads to a higher error rate, both for
CIFAR-10 and Digitclutter. Additional supporting experiments on CIFAR-100, MNIST, the Digit- [Figure] ●
[Figure] ● [Figure] clutter task, and on sample efﬁciency can be found in section
F. However, for intermediate coupling parameters of 0.5 and 0.9, this increase in error rate is smaller for networks with higher capacity (i [Figure]
e., more channels). This effect can also be seen from the relationship between iterative convergence indices and performance. In particular, Figure 3c demonstrates that performance is higher for ResNets with a higher Convergence Index and a lower Divergence Index.
This effect, however, is driven by the fact that ResNets with a higher number of channels also show higher measures of iterative convergence (see Figure 3a).
When controlling for the number of parameters (see lines in Figure 3c), we ﬁnd no clear relationship between Convergence and Divergence Index and performance.
We then assessed the effect of convergence regularization on performance by training several convergent ResNets on CIFAR-10 (see Fig. 4b for ICRs).
Convergence regularization led to a higher error rate across all coupling parameters and architectures.
A notable exception is the fully coupled
ResNet with 16 channels, which performs equally with and without convergence regularization.
This suggests that convergence is not a useful inductive bias in ResNets.
Taken together, these experiments suggest that iterative convergence may not provide a useful inductive bias for ResNets.
D ISCUSSION
We introduced soft gradient coupling, a new method of recurrence regularization, and demonstrated that this enabled us to manipulate iterative convergence properties in ResNets.
To measure these properties, we introduced three indices of iterative convergence, quantifying the effect of perturbations previously introduced in the literature (Veit et al., 2016; Jastrzebski et al., 2018).
Iterative methods are considered powerful approaches in particular for solving difﬁcult inverse problems.
However, here we did not ﬁnd iterative convergence to be a useful inductive bias for ResNets.
Moreover, we found that higher degrees of weight sharing did not improve a ResNet’s parameter efﬁciency.
One reason for this may be that soft gradient coupling or the spectral normalization are the wrong methods for this purpose or require a different optimization strategy.
Our ﬁndings also suggest, however, that deep feedforward computations ahould perhaps not be characterized as iterative reﬁnement on a latent representation, but simply as a sequence of operations smoothly approaching their ﬁnal estimate.
Our conclusions are based on experiments on four visual classiﬁcation tasks.
Visual tasks have been proposed to be inverse problems and therefore lend themselves to iterative inference algorithms.
Recognition tasks like Digitclutter that involve partial occlusions of objects have in particular been shown to beneﬁt from recurrent computations (Wyatte et al., 2014; Spoerer et al., 2017).
However, an iterative method like an error-corrective algorithm that would require a forward model of the data may be more complex and therefore harder to learn than a purely discriminative model.
Hence, for the four tested tasks, ResNets may learn direct inference rather than error-corrective inference via a forward model.
Although it did not improve performance here, soft gradient coupling provides a method for smoothly interpolating between feedforward and recurrent neural networks.
More generally, soft gradient coupling provides a simple way to encourage sets of weights to remain similar to each other.
This technique may ﬁnd further use in relaxing weight-sharing constraints and studying the beneﬁt of various forms of weight sharing, including recurrence and convolution, in deep neural networks.
A N ETWORK TRAINING A. S OFTWARE
We trained the networks using PyTorch (Paszke et al., 2019), PytorchLightning, numpy (Oliphant, 2006), and pandas in Python (Van Rossum & Drake, 2009).
The remaining analysis was conducted in R (R Core Team, 2019), using ggplot2 (Wickham, 2016), dplyr (Wickham et al., 2019), tidyr (Wickham & Henry, 2019), patchwork (Pedersen, 2019) and DescTools (Signorell, 2020).
The implementation of ResNet-104 was signiﬁcantly supported by an existing implementation (Idelbayev, 2018).
A. D ATASETS
We trained and evaluated the networks on four datasets: CIFAR-10, CIFAR-100 (Krizhevsky, 2009), MNIST (LeCun et al., 2010) and Digitclutter (Spoerer et al., 2017).
All images were normalized before being provided to the network.
For CIFAR-10, we used a training/validation sample split of 45000/5000, augmented the data by random cropping and random horizontal ﬂips during training.
For MNIST, we used a split of 50000/10000, and augmented the data by random cropping.
Finally, for Digitclutter, we used between two and ﬁve overlapping digits (referred to as Digitclutter-3) with a split of 100000/10000 without data augmentation. [Figure]
A. A RCHITECTURE DETAILS
As our base architecture we use a ResNet with the preactivation unit recommended by He et al.
(2016b).
This means that a residual block consists of
[Equation]
where BN stands for batch normalization (Ioffe & Szegedy, 2015), ReLU for rectiﬁed linear unit, and Conv for a convolutional layer.
For CIFAR-10, CIFAR-100, and Digitclutter, this ResNet contained three stages with 16 blocks each. This is a common depth for a ResNet.
For example He et al.
(2016a) use a ResNet-110.
Two stages were connected by a downsampling block, which halved width and height of the representation and doubled the number of channels.
The downsampling block consisted of a residual block where the ﬁrst convolutional layer had stride 2 and doubled the number of channels. The shortcut connection (otherwise simply the identity function) decreased the image resolution by taking into account only every second column and row. To increase the number of channels, the shortcut connection added a number of layers initialized as zero. This downsampling block was never removed as part of the perturbations deﬁning the indices of iterative convergence.
This means that for a standard ResNet with 16 channels in the ﬁrst stage, the second stage consisted of (16x16)-representations with 32 channels, and the third stage consisted of (8x8)-representations with 64 channels.
In contrast, we only used one stage with 16 blocks for MNIST.
A. T RAINING DETAILS
We initialized our networks using Kaiming initialization (He et al., 2015) for the convolutional and linear weights.
In the case of a recurrent initialization, these were equal across residual blocks of a stage.
In the case of a non-recurrent initialization, they were drawn independently for each block.
We initialized the batch normalization’s scale γ with 1 and its bias with 0 .
We trained the original networks for 400 epochs using gradient descent with momentum 0.9.
For CIFAR-10, we used an initial learning rate of 0.1, which was divided by 10 after 200 and 300 epochs (see ﬁgure 5 for the evolution of the learning rate of the ResNets with 16 channels in the ﬁrst stage). For CIFAR-100 and Digitclutter, we trained the networks for 200 epochs and used an initial learning rate of 0.1, which was divided by 10 after 100 and 150 epochs.
For MNIST, we used an initial learning rate of 0.025, which was divided by 2.5 after 200 and 300 epochs.
For Digitclutter, we used an initial learning rate of 0.05, which was divided by 5 after 200 and 300 epochs.
We then identiﬁed the epoch at which the network had obtained the best validation classiﬁcation error and used the corresponding model for all analyses. B A LTERNATIVE INDICES OF ITERATIVE CONVERGENCE
We have deﬁned our indices of iterative convergence as based on the accuracy with respect to the true image labels.
This makes sense because computer vision models are usually evaluated on accuracy and we may not care about perturbations which do not affect accuracy.
Nevertheless, only studying the perturbations’ effect on accuracy may lead us to miss certain phenomena.
For example, an intermediate representation may have a similar accuracy as the unperturbed model, but make largely inconsistent predictions. This would be important to know. To address this limitation, we studied the accuracy of intermediate representations with respect to the predictions made by the unperturbed model instead of the ground truth. Fig.
6a depicts the indices of iterative convergence for the nonconvergent ResNets trained on CIFAR-10.
As we can see, this alternative deﬁnition does not change the conclusions we can draw from the experiment.
C P ARAMETER EFFICIENCY C. E FFECTIVE PARAMETER COUNT
Softly coupled ResNets have the same number of raw parameters as ordinary ResNets with the same architecture.
Their soft weight sharing, however, increases the similarity between weights of different blocks.
This weight similarity is not reﬂected in the raw parameter count.
We therefore introduce an effective parameter count (EPC).
For this measure, three quantities are of interest: the parameters that are not coupled with any other parameters, W uncoupled , the mean parameters for a given stage s , W ( s ) , and the normalized deviations from this mean of a parameters within a particular layer, D ( s ) l :=
( W ( s ) l − W ( s ) ) /
ˆ σ ( W ( s ) ) .
As a measure of the effective number of parameters which is sensitive to different degrees of weight sharing, we now propose
[Equation]
{+(4) where the L 0 -norm (cid:107) · (cid:107) 
+}
corresponds to the parameter count, and S and L to the number of stages and blocks, respectively.
This allows us to interpolate between the raw parameter counts of nonrecurrent and recurrent ResNet.
For softly coupled ResNets, the EPC measure lies between the raw parameter count of non-recurrent and recurrent ResNets as a function of the weight similarity across weights (see Fig.
7b).
This deﬁnition also allows us to examine the effect of gradient coupling on the average deviation of the parameters from the mean.
As expected higher coupling parameter lead to a lower average deviation and λ = 1 leads to no deviation at all (see Fig. 7c).
C. S TRONGER GRADIENT COUPLING DOES NOT INCREASE PARAMETER EFFICIENCY
We hypothesized that ResNets may employ implicitly recurrent computations.
This would mean that there is a more recurrent network which is more parameter-efﬁcient.
If this is the case, soft gradient coupling might be able to ﬁnd such a solution.
For a given number of parameters, however, recurrent and softly gradient-coupled networks do not outperform ordinary ResNets (see Figure 3c, left).
Softly coupled networks even tend to do worse.
This is because pure parameter counts do not take into account the soft weight sharing across blocks. We therefore developed a measure of the effective parameter count which takes the similarity of weights across layers into account (see Section C.1). Figure 7c, right, shows that for a given effective parameter count, ordinary, softly gradient-coupled, and recurrent ResNets all perform equally well.
Nonetheless, this suggests that we cannot increase parameter efﬁciency by making the ResNets more recurrent. D T RAINING VARIATIONS On CIFAR-10, we explored several variations of the training algorithm, which were consistent with the ﬁndings presented above. l l l l l l l l l l l l l l ll l ll l l lll l lll llll llll
llll l l l l l l l l l l l l l l ll ll ll
[Table]
[Figure]
[Table]
[Figure] evaluated [Figure] # additional evaluations
[Figure] [Table] [Figure]
[Figure] [Figure]
(rec.)
[Figure]
t
[Figure]
I
[Figure]
l l l l l l l ll l l l l ll l l l l ll l l l l l l ll ll ll l l l l l l l l ll ll l l ll l l l l l l l l l l
[Figure]
lllllllllllllll
[Figure]
l
[Figure]
l l
[Figure]
llllllllllllllll
[Figure]
Cooijmans et al. (2017) proposed to initialize the scale of the batch normalization γ as 0.1 in recurrent neural networks.
When applying this method to softly gradient-coupled networks, we ﬁnd that the method strongly improves performance for λ = 0 . 9 , but not for any other coupling parameters (see ﬁgure 8a).
We may generalize soft gradient coupling to the coupling rule
[Equation]
where κ is some kernel depending on the coupling parameter λ . Ordinary soft gradient coupling can be recovered with a uniform kernel
[Equation]
Alternatively, we may want to couple adjacent blocks more strongly than blocks that are far away from each other.
This can, for example, be achieved with a triangular kernel
[Equation]
where ( · ) + = max( · , 0) .
Again, λ = 0 corresponds to an ordinary ResNet and λ = 1 corresponds to a fully recurrent ResNet, whereas intermediate values for λ smoothly interpolate between the two.
llll llll llll llll l lll llll llll llll llll l l ll
[Table]
[Table]
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
lll lll lll lll
[Table]
ll lll lll lll
[Table]
ll
[Table]
[Figure]
[Table]
[Figure]
[Table]
l l l
[Table]
[Figure]
llll llll l lll
[Table]
lll
[Table]
lll l l ll
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
Coupling parameter (Initialization)
E rr o r r a t e on C
I F A
R −# channels
l l l l
Batchnorm initialization l gamma= gamma=0.
c
l l l l l l l ll l l l l ll l l l l l ll l ll l l l l l l l l l l ll ll ll l ll l l l l l l l l l l ll l l l l l l ll l l l l l ll ll l ll l ll l l l l l l l l l l lll ll l l l l l l l l lll l l l l l l l l l l l l l l l l l ll l ll l l l ll l l l l l l l l l ll ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l ll l l l l ll l l l l l ll l ll l l l l l l l l l l ll ll ll l ll l l l l l l l l l l ll l l l l l l ll l l l l l ll ll l l l l ll l l l l l l l l l l lll ll l l l l l l l l l ll l l l l l l l l l l l l l l ll l ll l ll l l l ll l l ll l l l l l ll ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l # raw parameters # effective parameters 101010105.0% 7.0% 10.0%
E rr o r r a t e on C
I F A
R −
Coupling parameter (Initialization) l l l l l l 0 (non−rec.)
(rec.)
0.25 (rec.) 0.5 (rec.)
0.9 (rec.)
1 (rec.)
Figure 8b shows the effect of triangular as compared to uniform gradient coupling.
For a given coupling parameter, this method appears to outperform the uniformly gradient-coupled network, but neither method outperforms an ordinary ResNet.
D. S TART COUPLING AT LATER LAYERS
Finally, we uncoupled the ﬁrst ﬁve residual blocks and only constrained the remaining blocks.
For the fully recurrent case, this corresponds to the implementation by Jastrzebski et al.
We also studied the effect of this late coupling together with a batchnorm initialization of γ = 0 .
In both cases, the late coupling means that the coupled networks are better than coupled networks which start coupling at block 0, but still worse than an ordinary ResNet.
D. P ARAMETER EFFICIENCY
Figure 9 shows that across all these extended networks, the relationship between effective parameter count and performance is also largely unaffected by coupling parameter.
[-Coupling parameter (Initialization) l l l l l l 0 (non−rec.) 
-]
D. I TERATIVE CONVERGENCE INDICES
Figure 10 demonstrates that for the extended training variations, higher coupling parameters generally tend to increase the Convergence Index and decrease the Divergence Index, as well.
The extended training variations also predominantly yield networks with a Recurrence Index around 1.
E D ETAILS ON THE SPECTRAL NORMALIZATION E. U PPER BOUND ON THE L IPSCHITZ CONSTANT
To determine an upper bound on the Lipschitz constant of f , we use an alternative characterization based on the Jacobian J f ( x ) .
The spectral norm (cid:107) A (cid:107) 2 of a matrix A is deﬁned as its maximal singular value.
The Lipschitz constant is then given by
[Equation]
According to the chain rule, l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l
[Figure]
S t a r t c oup li
ng a t:
K e r ne l t y pe : un
i f
o r m
S t a r t c oup li
ng a t:
K e r ne l t y pe : t r i angu l a r
S t a r t c oup li
ng a t:
K e r ne l t y pe : un
i f
o r m
[Figure]
[Figure]
# channels l l l l
[Equation]
Here z 1 and z 2 are given by the representation at the appropriate intermediate stages of the residual function, i. e. z 1 is the representation after BN 1 and z 2 is the representation after BN 2 .
The
Jacobians of the batch normalizations and convolutions do not depend on the input as these are linear operations.
Since J ReLU and J BN 1 are both diagonal matrices, they commute and therefore,
[Equation]
We have therefore split the J f into a product of two constant functions and two Jacobian of the rectiﬁed linear unit.
The spectral norm (cid:107) · (cid:107) 2 is known to be sub-multiplicative.
This means that for matrices A and B , (cid:107) BA (cid:107) 2 ≤ (cid:107) B (cid:107) 2 (cid:107) A (cid:107) 2 .
Moreover, J ReLU , depending on whether its input is positive or negative is given by a diagonal matrix with ones or zeros on the diagonal.
Its singular values are therefore at most 1.
Putting this together, we can upper bound the Lipschitz constant as
[Equation]
Theoretically, we could determine the maximal singular value of the convolution using a singular value decomposition.
However, the singular value decomposition of such a large matrix is computationally expensive.
Instead, Yoshida & Miyato (2017) lay out how the maximal singular value can be approximated using a power iteration (von Mises & Pollaczek-Geiringer, 1929).
The only difference to their method consists in the fact that our ﬁrst convolution additionally involves multiplying the input by the diagonal matrices given by the two batch normalizations.
l ll lll ll l l l l lll l ll lll ll l l l l lll
[Figure]
l l l l l l
[Figure]
[Figure]
[Figure]
[Figure]
Fig.
11 demonstrates that the properly convergent ResNets indeed converge.
This is guaranteed for the recurrent ResNet, but empirically we see that it is also the case for non-recurrent properly convergent ResNets.
Fig.
12 demonstrates that these ResNets tend to perform a bit worse than the improperly convergent ResNets we studied in the main article.
F R ESULTS ON D IGITCLUTTER , MNIST, CIFAR-100 AND SAMPLE EFFICIENCY
Fig.
13 depicts the performance of the gradient-coupled networks on Digitclutter, CIFAR-100,
MNIST, and CIFAR-10 trained with fewer samples.
In none of these case, higher gradient coupling parameters systematically improve performance.
lll l l l l l l l l l l l l l ll l l l l ll l l l l l l l ll l l l l l l l l l l l l l l l l ll l l l l l l l l l l ll l l l l l l l l l l l l l
[Table]
Coupling parameter
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
r a t
[Figure]
d