Original: The Natural Tendency of Feed Forward Neural Networks to Favor Invariant Units
Revised: The Natural Tendency of Feed Forward Neural Networks to Favor Invariant Units

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: A central goal in the study of the primate visual cortex and hierarchical models for object recognition is understanding how and why single units trade off invariance versus sensitivity to image transformations.
Revised: A central goal in the study of the primate visual cortex and hierarchical models for object recognition is understanding how and why single units trade off invariance versus sensitivity to image transformations.

Original: For example, in both deep networks and visual cortex there is substantial variation from layer-to-layer and unit-to-unit in the degree of translation invariance.
Revised: For example, in both deep networks and visual cortex there is substantial variation from layer-to-layer and unitto-unit in the degree of translation invariance.

Original: Here, we provide theoretical insight into this variation and its consequences for encoding in a deep network.
Revised: Here, we provide theoretical insight into this variation and its consequences for encoding in a deep network.

Original: Our critical insight comes from the fact that rectiﬁcation simultaneously decreases response variance and correlation across responses to transformed stimuli, naturally inducing a positive relationship between invariance and dynamic range in the subsequent layer.
Revised: Our critical insight comes from the fact that rectiﬁcation simultaneously decreases response variance and correlation across responses to transformed stimuli, naturally inducing a positive relationship between invariance and dynamic range.

Original: Invariant  units then tend to drive the network more than those sensitive to small image transformations.
Revised: Invariant input units then tend to drive the network more than those sensitive to small image transformations.

Original: We discuss consequences of this relationship for AI: deep nets naturally weight invariant units over sensitive units, and this can be strengthened with training, perhaps contributing to generalization performance.
Revised: We discuss consequences of this relationship for AI: deep nets naturally weight invariant units over sensitive units, and this can be strengthened with training, perhaps contributing to generalization performance.

Original: Our results predict a signature relationship between invariance and dynamic range that can now be tested in future neurophysiological studies.
Revised: Our results predict a signature relationship between invariance and dynamic range that can now be tested in future neurophysiological studies.

Original:  Introduction
Revised:  Introduction

Original: Invariances to image transformations, such as translation and scaling, have been reported in single units in visual cortex, but just as often sensitivity to these transformations has been found (El-Shamayleh and Pasupathy, 2016, Sharpee et al.
Revised: Invariances to image transformations, such as translation and scaling, have been reported in single units in visual cortex, but just as often sensitivity to these transformations has been found (El-Shamayleh and Pasupathy, 2016, Sharpee et al.

Original: 2013, Rust and
Revised: 2013, Rust and

Original: DiCarlo, 2012).
Revised: DiCarlo, 2012).

Original: Similarly, in deep networks there is variation in translation invariance both within and across layers (Pospisil et al., 2018, Shen et al., 2016, Shang et al., 2016, Goodfellow et al., 2009).
Revised: Similarly, in deep networks there is variation in translation invariance both within and across layers (Pospisil et al., 2018, Shen et al., 2016, Shang et al., 2016, Goodfellow et al., 2009).

Original: Notionally, information about the position of the features composing objects may be important to category selectivity.
Revised: Notionally, information about the position of the features composing objects may be important to category selectivity.

Original: For example, the detection of eyes, nose and lips are not sufﬁcient for face recognition, the relative positions of these parts must also be encoded.
Revised: For example, the detection of eyes, nose and lips are not sufﬁcient for face recognition, the relative positions of these parts must also be encoded.

Original: Thus it is reasonable to expect some balance between invariance and sensitivity to position.
Revised: Thus it is reasonable to expect some balance between invariance and sensitivity to position.

Original: We empirically observe that in a popular deep network, in both its trained and untrained state, invariant units tend to have higher dynamic range than sensitive units (Figure 1B and C).
Revised: We empirically observe that in a popular deep network, in both its trained and untrained state, invariant units tend to have higher dynamic range than sensitive units (Figure 1B and C).

Original: This raises the possibility that the effective gain on invariant units into the subsequent layer is stronger than that of sensitive units.
Revised: This raises the possibility that the effective gain on invariant units into the subsequent layer is stronger than that of sensitive units.

Original: Here we provide theoretical insight to show that rectiﬁcation in a deep network  naturally biases networks to this difference between invariant and sensitive units.
Revised: Here we provide theoretical insight into how rectiﬁcation in a deep network could naturally biase networks to this difference between invariant and sensitive units.

Original: We do this by examining how co-variance of a multivariate normal distribution is inﬂuenced by rectiﬁcation, and we then test these insights in a deep neural network.
Revised: We do this by examining how co-variance of a multivariate normal distribution is inﬂuenced by rectiﬁcation, and we then test these insights in a deep neural network.

Original:  Statistical model
Revised:  Statistical model

Original: The response of a unit in a feed forward neural network is: r = (cid:126)w · g ( S ) where S is the response of all n subunits in the previous layer, g the non-linearity of rectiﬁcation g ( x ) = max (0 , x ) , (cid:126)w is the n × 1 vector of weights, and r is the response of the unit.
Revised: The response of a unit in a feed forward neural network is: r = (cid:126)w · g ( S ) where S is the response of all n input units in the previous layer, g the non-linearity of rectiﬁcation g ( x ) = max (0 , x ) , (cid:126)w is the n × 1 vector of weights, and r is the response of the unit.

Original: Randomly sampling from a distribution of input images, the response S takes on a distribution with some expectation and covariance across these images: E
Revised: Randomly sampling from a distribution of input images, the response S takes on a distribution with some expectation and covariance across these images: E

Original: [ S ] = µ (an n × 1 vector), and Cov [ S ] = Σ (an n × n matrix).
Revised: [ S ] = µ (an n × 1 vector), and Cov [ S ] = Σ (an n × n matrix).

Original: The application of the non-linearity transforms these moments: E [ g ( S )]
Revised: The application of the non-linearity transforms these moments: E [ g ( S )]

Original: = ˜ µ , Cov [ g ( S )] =
Revised: = ˜ µ , Cov [ g ( S )] =

Original: ˜Σ .
Revised: ˜Σ .

Original: Let S 1 be the responses to randomly sampled input images and S 2 the responses to a transformation of those same images.
Revised: Let S 1 be the responses to randomly sampled input images and S 2 the responses to a transformation of those same images.

Original: So the moments of the full distribution are:
Revised: So the moments of the full distribution are:

Original: [Equation]
Revised: [Equation]

Original: where ˜Σ 1 , 1 is the covariance of rectiﬁed subunits responding to the original images, ˜Σ 2 , 2 is the covariance of rectiﬁed subunits to the transformed images and ˜Σ 1 , 2 is the covariance between rectiﬁed subunits responding to the reference and transformed images
Revised: where ˜Σ 1 , 1 is the covariance of rectiﬁed input units responding to the original images, ˜Σ 2 , 2 is the covariance of rectiﬁed input units to the transformed images and ˜Σ 1 , 2 is the covariance between rectiﬁed input units responding to the reference and transformed images. We note we only deﬁne the 1st two moments above and no assumption about the distribution of the rectiﬁed responses is made.

Original: The joint distribution of a with weights (cid:126)w on the n rectiﬁed  is:
Revised: The covariance of an output unit with weights (cid:126)w on the n rectiﬁed input units is:

Original: [Equation]
Revised: [Equation]

Original: so the correlation between the response of the  unit to the reference and transformed images is:
Revised: so the correlation between the response of the output unit to the reference and transformed images is:

Original: [Equation]
Revised: [Equation]

Original: Below we investigate how the ˜Σ i,j depend on Σ i,j and µ , which provides insight into a relationship between ˜ ρ (cid:48) and ˜ σ 2 (cid:48) .
Revised: Below we investigate how the ˜Σ i,j depend on Σ i,j and µ , which provides insight into a relationship between ˜ ρ (cid:48) and ˜ σ 2 (cid:48) .

Original: We begin by examining a model of a single rectiﬁed subunit responding to the reference and transformed images.
Revised: We begin by examining a model of a single rectiﬁed input unit responding to the reference and transformed images.

Original: 2.1 Single rectiﬁed input
Revised: 2.1 Single rectiﬁed input

Original: We model the responses, S 1 and S 2 , of a single subunit to the reference and transformed inputs, respectively, as a bivariate normal distribution:
Revised: We model the responses, S 1 and S 2 , of a single input unit to the reference and transformed inputs, respectively, as a bivariate normal distribution:

Original: [Equation]
Revised: [Equation]

Original: When these responses are acted on by rectiﬁcation, both the variances of the responses and the correlation between the sets of responses is decreased.
Revised: When these responses are acted on by rectiﬁcation, both the variances of the responses and the correlation between the sets of responses is decreased.

Original: This observation is analogous to that of de la Rocha et al.
Revised: This observation is analogous to that of de la Rocha et al.

Original: (2007) where they investigated the inﬂuence of neuronal ﬁring threshold rectiﬁcation on the pairwise correlations between neurons as a function of ﬁring rate.
Revised: (2007) where they investigated the inﬂuence of neuronal ﬁring threshold rectiﬁcation on the pairwise correlations between neurons as a function of ﬁring rate.

Original: We extend this observation in the next section to consider how this effect inﬂuences invariance in downstream units.
Revised: We extend this observation in the next section to consider how this effect inﬂuences invariance in downstream units.

Original: It is instructive to consider a schematic (Figure 2A) of the distribution of responses.
Revised: It is instructive to consider a schematic (Figure 2A) of the distribution of responses.

Original: The fall in correlation occurs because the variance from the linear relationship V ar ( E [ ˜ S 2 | ˜ S 1 ]) decreases as a result of truncation, whereas the residual E [ V ar ( ˜ S 1 | ˜ S 1 )] is not reduced as much. 
Revised: The fall in correlation occurs because the variance from the linear relationship V ar ( E [ ˜ S 2 | ˜ S 1 ]) decreases as a result of truncation, whereas the residual E [ V ar ( ˜ S 1 | ˜ S 1 )] is not reduced as much. (see Intuition behind variance correlation relation).

Original: Notionally, the average length of a vertical line in the remaining elipsoid ( E [ V ar ( ˜ S 1 | ˜ S 1 )] ) is not reduced as much as the diagonal line of best ﬁt ( V ar [ E [ ˜ S 1 | ˜ S 1 ]] ).

Original: For further demonstration, see de la Rocha et al.
Revised: For further demonstration, see de la Rocha et al.

Original: In the following section, we will write the correlation and variance after rectiﬁcation explicitly as a function of the relevant parameters: ˜ σ ( µ/σ ) 2 and ˜ ρ ( µ/σ, ρ ) .
Revised: In the following section, we will write the correlation and variance after rectiﬁcation explicitly as a function of the relevant parameters: ˜ σ ( µ/σ ) 2 and ˜ ρ ( µ/σ, ρ ) .

Original: 2.2 Unit integrating across multiple rectiﬁed subunits
Revised: 2.2 Unit integrating across multiple rectiﬁed input units

Original: Here we extend from the single subunit case to the multi-subunit case by examining the invariance ˜ ρ (cid:48) resulting from taking weighted combinations of rectiﬁed subunits.
Revised: Here we extend from the single input unit case to the multi-input unit case by examining the invariance ˜ ρ (cid:48) resulting from taking weighted combinations of rectiﬁed input units.

Original: Our key insight is that invariance increases to the degree that directions of maximal variance in the response distribution of rectiﬁed subunits are integrated.
Revised: Our key insight is that invariance increases to the degree that directions of maximal variance in the response distribution of rectiﬁed input units are integrated.

Original: A helpful geometric perspective comes from decomposing the numerator and denominator of ˜ ρ (cid:48) into their eigenbasis. We assume Σ 1 , 1 = Σ 2 , 2 = Σ 1 , 2 (cid:15) ( (cid:15) a small constant) a more reasonable approximationfor the case of smaller transformations of the input images. Furthermore we assume that each subunit has the same covariance structure as another (analogous to the covariance of a stationary stochastic process) so that the primary driver in differences post-rectiﬁcation is the original µ i /σ . Thus we have:
Revised: For a ﬁrst order approximation of this relationship we approximate input covariance before rectiﬁcation as an identity matrix scaled by the average variance:

Original: [Equation]
Revised: [Equation]

Original: Σ 1 , 1 (cid:126)w
Revised: This approximation improves as off diagonal covariance shrinks and ¯ σ 2 V ar ( σ 2 i ) increases.

Original: [Equation] =( (cid:126)w
Revised: Thus our approximate model is: Cov [ S 1 ] =

Original: [Equation]
Revised: Cov [ S 2 ] =

Original: (cid:126)u
Revised: Σ 1 , 1 = Σ 2 , 2 = ¯ σ 2 I where ¯ σ 2 is averaged across the diagonals of the original S 1 and S 2 E

Original: [Equation]
Revised: [ S 1 ] =

Original: [Equation]
Revised: E [ S 2 ] = µ = [ µ 1 ...µ n ] T where means are approximated by averaging across the original S 1 and S 2 .

Original: (cid:113) λ (1 , 2))
Revised: Cov [ S 1 , S 2 ] = Σ 1 , 2 = ¯

Original:  2 + ...
Revised: ρσ 2 I where ρσ 2 > 0 justiﬁed by the assumption of a small transformation thus correlation is positive.

Original: + ( (cid:126)w T (cid:126)u (1 , 2) n (cid:113) λ (1 , 2) n ) [Equation]
Revised: For convenience sort the µ i in from high to low, then it naturally follows that the eigenvectors of ˜Σ 1 , 2 and ˜Σ 1 , 1 are the same: I the identity (since the covariance matrices are diagonal) and the eigenvalues are simply the entries of the diagonal (in order from high to low since ˜ ρ , and ˜ σ 2 are decreasing in µ ; see Figure 2B)

Original: ( (cid:126)w
Revised: so we have:

Original: [Equation]
Revised: [Equation]

Original: (cid:126)u (1 , 1)(cid:113) λ (1 , 1)) 2+ ... + ( (cid:126)w T (cid:126)u (1 , 1) n (cid:113) λ (1 , 1) n ).
Revised: thus we have the geometric picture described in Figure 3A exactly. The denominator as a function of the direction of a unit length (cid:126)w (length of (cid:126)w does not change ρ (cid:48) ) is an axis aligned ellipsoid with length along the i th axis of ˜ σ 2 ( µ i /σ ) .

Original: We also assume that the eigenvectors (cid:126)µ (1 , 2) i ≈ (cid:126)µ (1 , 1) i . Thus the geometric picture (Figure 3A) is  of the  numerator as an eigenvector-aligned ellipsoid (blue) nested inside of the larger ellipsoid formed by the denominator (red).
Revised: Notice that the numerator is the variation of the output unit thus more invariant units contribute more variance than less invariant units assuming there is not a negative correlation between w 2 i and ˜ σ 2 i ˜ ρ i . The numerator is another axis aligned ellipsoid (blue) with length along the i th axis of ˜ σ 2 ( µ i /σ )

Original: The correlation being determined by the relative distance from the origin of these ellipses in the direction of (cid:126)w (pink).
Revised: ˜

Original: The ellipsoid is nested because (˜ ρ  ≤ 1) . The widths of these ellipsoids along the eigenvectors are determined by the eigenvalues.
Revised: ρ ( µ i /σ, ρ ) this numerator ellipsoid is contained within the denominator since ˜ ρ (cid:48) ≤ 1 . Recognizing ˜ ρ (cid:48) as a weighted arithmetic mean with with weights c i =

Original: In general, small eigenvalues are paired with linear combinations of low-variance sensitive subunits, whereas large eigenvalues are paired with high-variance invariant subunits.
Revised: [Equation] ) (note (cid:80) c i = 1 ) we see that if there is not a negative correlation of w i 2 with ˜ σ 2 i than output units will tend to have a higher invariance then the average of input units thus pushing output units invariance upwards

Original: Performing simulations of a few simple subunit covariance structures shows that the ˜ ρ (cid:48) to ˜ σ 2 (cid:48) relationship is maintained, though its form changes (Figure 3B).
Revised: Performing simulations of a few simple input unit covariance structures shows that the ˜ ρ (cid:48) to ˜ σ 2 (cid:48) relationship is maintained, though its form changes (Figure 3B).

Original: Integrating over a population of subunits, the form of the relationship changes from the single subunit case (Figure 3B black dashed and dotted line).
Revised: Integrating over a population of input units the form of the relationship changes from the single input unit case (Figure 3B black dashed and dotted line).

Original: Averaging over subunits with different µ i /σ shifts the relationship because of the curvature of the non-linear transforms ˜ σ ( µ/σ ) 2 and ˜ ρ ( µ/σ, ρ ) .
Revised: Averaging over input units with different µ i /σ shifts the relationship because of the curvature of the non-linear transform of the ratio ˜ ρ ( µ/σ, ρ ) .

Original: Finally, in the case where subunits have non-zero correlation (i.e. shared tuning , off-diagonals of Σ i,j non-zero; Figure 3B red and cyan) overall variance increases because correlated subunits are being added.
Revised: Finally, in the case where input units have non-zero correlation (i.e. shared tuning, off-diagonals of Σ i,j non-zero; Figure 3B red and cyan) overall variance increases because correlated input units are being added.

Original:  Variation in the neighborhood being integrated over in terms of ρ and µ/σ would weaken the strength of the ˜ ρ (cid:48) to ˜ σ 2 (cid:48) relationship. For example, sampling randomly from the different traces in Figure 3B, the relationship would appear less systematic. This leads us to empirically test whether in a well known deep neural network the relationship emerges.
Revised:  Experimental results

Original: [Table]
Revised: [Table]

Original: Here we analyze the covariance structure of the inputs of a popular deep neural network (AlexNet) for translations of input images.
Revised: Here we analyze the covariance structure of the inputs of a popular deep neural network (AlexNet) for translations of input images.

Original: We ﬁrst tested the network in its untrained state by presenting a collection of 500 image patches drawn from the 2012 ImageNet validation set.
Revised: We ﬁrst tested the network in its untrained state by presenting a collection of 500 image patches drawn from the 2012 ImageNet validation set.

Original: References images were cropped enough to allow the original and translated images to ﬁt within the maximal receptive ﬁeld of the units being tested.
Revised: References images were cropped enough to allow the original and translated images to ﬁt within the maximal receptive ﬁeld of the units being tested.

Original: We included a small and a large translation, and at each convolutional layer we measured the correlation and variance.
Revised: We included a small and a large translation, and at each convolutional layer we measured the correlation and variance.

Original: We ﬁnd a positive relationship at all layers with signiﬁcant Spearman’s ranked correlation for both transformations (Table 1 Untrained).
Revised: We ﬁnd a positive relationship at all layers with signiﬁcant Spearman’s ranked correlation for both transformations (Table 1 Untrained).

Original: The strength of the relationship tended to be stronger for the smaller translation (Figure 1C, orange).
Revised: The strength of the relationship tended to be stronger for the smaller translation (Figure 1C, orange).

Original: Thus in a popular deep network with no training, units which tended to have greater invariance also had higher dynamic range.
Revised: Thus in a popular deep network with no training, units which tended to have greater invariance also had higher dynamic range.

Original: We repeated the same analysis in AlexNet after it was fully trained for object recognition.
Revised: We repeated the same analysis in AlexNet after it was fully trained for object recognition.

Original: Again we observed a signiﬁcant positive relationship (Table 1 Trained; Figure 1B).
Revised: Again we observed a signiﬁcant positive relationship (Table 1 Trained; Figure 1B).

Original: The relationship was somewhat weaker than in the trained network.
Revised: The relationship was somewhat weaker than in the trained network.

Original: Thus training weakens but does not remove the bias of the network to associate higher dynamic range with higher translation invariance.
Revised: Thus training weakens but does not remove the bias of the network to associate higher dynamic range with higher translation invariance.

Original: Finally, we asked whether the network may compensate for this imbalance by placing weights of higher magnitude on low dynamic range units, thus effectively removing this bias.
Revised: Finally, we asked whether the network may compensate for this imbalance by placing weights of higher magnitude on low dynamic range units (a negative correlation between ˜ σ 2 i and w 2 i ), thus effectively removing this bias.

Original: We measured whether the percent of weight magnitude on a given subunit across afferent units was greater for subunits with higher variance.
Revised: We measured whether the percent of weight magnitude on a given input unit across output units was greater for input units with higher variance.

Original: We found Conv3 ( r s = 0 . 34 ) and Conv4 ( r s = 0 .
Revised: We found Conv3 ( r s = 0 . 34 ) and Conv ( r s = 0 .

Original: 19 ) tended to have higher weights on higher variance subunits while There was no correlation in Conv2 and and Conv5.
Revised: 19 ) tended to have higher weights on higher variance input units while there was no correlation in Conv2 and and Conv5.

Original: This indicates the network does not compensate for the imbalance in dynamic range between invariant and sensitive units but actually sometimes emphasizes it.
Revised: This indicates the network does not compensate for the imbalance in dynamic range between invariant and sensitive units but actually sometimes emphasizes it.

Original:  Discussion
Revised:  Discussion

Original: We have documented an empirical relationship between the dynamic range of unrectiﬁed units in a deep network and their invariance.
Revised: We have documented an empirical relationship between the dynamic range of unrectiﬁed units in a deep network and their invariance.

Original: We provided a simple  statistical model to explain this effect in which rectiﬁcation caused the population representation to primarily vary in dimensions that were invariant to small image perturbations, whereas small perturbations were represented in directions of lower variance.
Revised: We provided a simple 1st order statistical model to explain this effect in which rectiﬁcation caused the population representation to primarily vary in dimensions that were invariant to small image perturbations, whereas small perturbations were represented in directions of lower variance.

Original: Further work can investigate whether this imbalance improves generalization because of the emphasis placed on invariant over sensitive units.
Revised: Further work can investigate whether this imbalance improves generalization because of the emphasis placed on invariant over sensitive units.

Revised: We note this relationship is weaker in the trained then untrained network further work can udnerstand this difference.

Revised: Our approximations assumed low covariance between input units and homoegenous input variance while this may be expected in a random network it may not be true in a trained network.

Revised: More crucially further theoretical work should consider the inﬂuence of co-variance between input units and invariance of output units as a function of weights.

Original: To extend insights from simpliﬁed, artiﬁcial networks to neurobiology, it will ﬁrst of all be important to test whether cortical neurons showing more invariance also tend to have a higher dynamic range.
Revised: To extend insights from simpliﬁed, artiﬁcial networks to neurobiology, it will ﬁrst of all be important to test whether cortical neurons showing more invariance also tend to have a higher dynamic range.

Original: If they do, this will establish a fundamental theoretical connection between computations of deep networks and the brain.  
Revised: If they do, this will establish a fundamental theoretical connection between computations of deep networks and the brain.  Response to reviewer comments

Revised: We thank our reviewers for their careful and insightul comments.

Revised: Above we have taken their comments into account in editing our ﬁnal draft.

Revised: Below we address their three main concerns.

Revised: 5.1 Intuition behind variance correlation relation

Revised: It is instructive to consider a schematic (Figure 2A) of the distribution of responses.

Revised: The probability mass of the response is broken into 4 quadrants, the 1st green is unaffected by rectiﬁcations, the 2nd (purple) is projected onto the vertical axis (thick purple), the 3rd (red) is projected onto the origin (red dot), and the 4th (green) projected onto the horizontal axis (thick green).

Revised: The diagonal line is the line of best ﬁt expressing the linear relationship and the vertical line is a conditional distribution whose variance is the conditional residual which averaged gives the residual variance of the linear relationship.

Revised: ˜ σ 2 decreases as µ/σ decreases because the spread of the distribution is truncated to the degree that the distribution falls beneath the threshold.

Revised: For correlation it is useful to consider: ˜ ρ

Revised: 2 =

Revised: Var(Best Linear Predictor)

Revised: [Equation]

Revised: V ar

Revised: [ E [ ˜ S 2 | ˜ S 1 ]]

Revised: V ar [

Revised: [Equation]

Revised: [Equation]

Revised: S

Revised: [Equation]

Revised: [Equation]

Revised: E [ V ar ( ˜ S 2 | ˜ S 1 )] where V ar [ E [ ˜ S 2 | ˜ S 1 ]]

Revised: =

Revised: V ar

Revised: [˜ ρ ˜ S 1 ] = ρ 2 V ar ( ˜ S 1 ) decreases more rapidly then the average residual E [ V ar ( ˜ S 2 | ˜ S 1 )] .

Revised: Notionally we can think of V ar [ E [ ˜ S 2 | ˜ S 1 ]] as the vertical height of the diagonal line that has not been truncated (solid not truncated, dashed truncated) in Figure 2A and E [ V ar ( ˜ S 2 | ˜ S 1 )] as the average length of vertical lines not truncated.

Revised: Notice that the ratio of truncated to untruncated is lower for the diagonal then the vertical average.

Revised: In the ﬁgure at µ = 0 the diagonal line is cut in half and so is the length of a vertical line drawn here.

Revised: But at all other positions ( µ > 0 ) where a vertical line would be drawn the vertical solid lines length is truncated less than half thus on average the vertical line is less truncated than the diagonals vertical length.

Revised: 5.2 Rectiﬁcation does not explain invariance

Revised: We would like to emphasize we are not arguing that rectiﬁcation explains the generalization properties of networks only that its inﬂuence on covariance may be one of many factors inﬂuencing invariance.

Revised: 5.3 Justiﬁcation of simplifying assumptions

Revised: We would like to emphasize that in this paper we pursue intuition by trying to understand a simple approximation to rectiﬁcations inﬂuence on invariance which results in a simple analytic form.

Revised: Our ﬁrst approximation is to remove off-diagonal covariances.

Revised: Since the inﬂuences of off-diagonals are additive they can be seen as modulating the effects induced by the diagonals:

Revised: [Equation]

Revised: Thus here we analytically study the ﬁrst order effect of rectiﬁcation in output neurons on the basis of the variance but not covariance of their inputs.

Revised: Finally we approximate the diagonal of the input variance with the average variance an approximation which minimizes squared error

Revised: [Equation]

Revised: variation in σ 2 will hurt the strength of this approximation but not change the main effect unless this variation is negatively correlated with µ i thus canceling out the relationship between correlation and variance.

Revised: We would not expect this negative correlation in an untrained network and further work can check whether this, potentially interesting, relationship exists in trained networks.

Revised: We note that normalization enforces this approximation and thus these approximations may be particularly suited to networks using normalization.

Revised: Acknowledgements

Revised: This work was funded by a National Science Foundation (NSF) Graduate Research Fellowship (D.A.P.), the National Science Foundation CRCNS Grant IIS-1309725 (W.B.), and NIH Grant R01 EY-027023 (W.B.)