Convolutional neural networks with extra-classical receptive ﬁelds
Abstract
In the visual system, neurons respond to a patch of the input known as their classical receptive ﬁeld (RF), and can be modulated by stimuli in the surround.
These interactions are often mediated by lateral connections, giving rise to extra-classical RFs.
We use supervised learning via backpropagation to learn feedforward connections, combined with an unsupervised learning rule to learn lateral connections between units within a convolutional neural network.
These connections allow each unit to integrate information from its surround, generating extra-classical receptive ﬁelds for the units in our new proposed model (CNNEx).
We demonstrate that these connections make the network more robust and achieve better performance on noisy versions of the MNIST and CIFAR-10 datasets.
Although the image statistics of MNIST and CIFAR-10 differ greatly, the same unsupervised learning rule generalized to both datasets.
Our framework can potentially be applied to networks trained on other tasks, with the learned lateral connections aiding the computations implemented by feedforward connections when the input is unreliable.
Introduction
While feedforward convolutional neural networks have resulted in many practical successes [1] they are highly susceptible to adversarial attacks [2].
In contrast, the brain makes use of extensive recurrent connections, including lateral and feedback connections, which may provide some level of immunity to these attacks (for results on human adversarial examples, see [3]).
Additionally, the brain is able to build rich internal representations of information with little to no labeled data, which is a form of unsupervised learning, in contrast to the supervised learning required by most models.
We present a model incorporating lateral connections (learned using a modiﬁed Hebbian rule) into convolutional neural networks, with feedforward connections trained in a supervised manner.
When applying different noise perturbations to the MNIST [4] and CIFAR-10 [5] datasets, lateral connections in our model improve the overall performance and robustness of these networks.
Our results suggest that integration of lateral connections into convolutional neural networks is an important area of future research. [Figure] Relation to previous work
A number of normative and dynamical models relating contextual modulation of neuronal responses and lateral connectivity have been proposed in the literature.
Normative models based on sparse coding [9, 10, 11, 12, 13, 14, 15] predict anti-Hebbian lateral connections between excitatory neurons, in contrast with the experimentally observed like-to-like excitatory connectivity (but see [16] which extends the sparse coding model to learn like-to-like horizontal connections by including a pairwise coupling term in the prior) Our model bears close resemblance to the MGSM (mixture of Gaussian scale mixtures) model of natural images proposed by Coen-Cagli, Dayan and Schwartz [17] which infers contextual interactions between the RF and surround that would lead to optimal coding of images. Recent work has demonstrated the ability to learn either ﬂexible normalization [18] or divisive normalization [19] in deep neural networks. In addition to these approaches, other normalization schemes from the machine learning ﬁeld (e.g. batch normalization [20], layer normalization [21], local response normalization [22]) have been used to accelerate training of neural networks.
In contrast with these approaches, we propose a computational role for each pyramidal neuron (or unit in a deep neural network) in terms of how it integrates lateral input information optimally from a
Bayesian perspective.
Our proposal enables us to incorporate optimal lateral connections (learned in an unsupervised manner) straightforwardly into feedforward neural networks.
Neuroscience-inspired extra-classical receptive ﬁeld model. Let fc m, ( l ) j,x denote the classical receptive ﬁeld (RF) response of a unit representing feature j in layer l at image location m , given image x .
The full response of a unit representing feature j at image location m in layer l , given image x can then be written as:
[Equation]
where the second term on the right side represents the contribution from the extra-classical RF, α represents a hyperparameter that tunes the strength of the lateral connections, and W mn, ( l ) jk are the synaptic weights from surrounding units n on to unit m within layer l .
These weights are learned in an unsupervised manner using the rule:
[Equation] [Figure]
where (cid:104) .
(cid:105) x represents an average over the set of training images.
Note that this formula differs from a
Hebbian learning rule, in that only the covariance between the feedforward responses of units leads to changes in the lateral connections.
A derivation of the above equations is shown in the Appendix.
Comparison to experimental data. Using natural images from the Berkeley Segmentation Dataset [23] and a dictionary of classical RF features parameterized from mouse V1 electrophysiological responses [8], we computed lateral connection weights using Eq. 2.
Figure 1 shows that predicted lateral connections qualitatively match the orientation and distance dependence of connectivity observed in mouse cortex.
Datasets. We used the standard train/test splits, holding out 10% of the training data for validation.
We also added two types of noise to the original images: additive white Gaussian noise (AWGN) and salt-and-pepper noise (SPN).
The mean of the AWGN was set to zero and the standard deviation varied in increasing levels from 0.1-0.5.
For the SPN, the fraction of noisy pixels varied in increasing levels from 0.1-0.5.
Example stimuli (original and noisy images) are shown in Figure 2.
Network architecture and training. The model architectures used are described in Table 1. The
CNNEx and CNN-PM models have the same number of parameters, with the additional convolutional layers in CNN-PM being standard feedforward layers trained in a supervised manner (in contrast to the lateral connections given by Eq. 1 for CNNEx, which are trained in an unsupervised manner).
MNIST models were trained for 10 epochs with a minibatch size of 64 using stochastic gradient descent with a learning rate of 0.01 and a momentum value of 0.5. CIFAR models were trained for 50 epochs and a momentum value of 0.9, keeping all other hyperparameters the same.
We trained 10 different instantiations using different random seeds to ensure the robustness of our results.
All experiments were performed using Pytorch (v. 0.3.1) on a NVIDIA GTX 1080 Ti GPU.
[Table] [Table]
Lateral connections. We ﬁrst trained feedforward weights in the network using supervised learning.
After freezing the feedforward weights, we introduced lateral connections given by Eqn. 2 between units in the ﬁrst two convolutional layers.
Importantly, we note that the lateral connection weights are only learned once at the end of supervised training, by keeping the feedforward weights of the network ﬁxed and using the activations of units over a set of training images. We do not update the feedforward weights of the network by backpropagating through the computed lateral connections. Future work will explore new methods for semi-supervised learning, which combine supervised learning of feedforward weights with unsupervised learning of the lateral connections.
Network regularization. We chose a weight decay value of 0.005 and a dropout fraction of 0.5.
Weight decay acted on all non-bias parameters of the model, while dropout was applied after each convolutional layer in the model, as well as after the ﬁrst fully connected layer.
We also tested the combination of these regularization techniques with the lateral connections.
Validation and testing. Lateral connections had a spatial extent of 7x7 (3x3) pixels in the ﬁrst (second) convolutional layers, with connections from the same spatial location set to zero.
Hyperparameters α for each of the two layers were chosen based on a grid search over the parameter range { 0 . 1 , 0 . 01 , 0 . 001 , 0 . 0001 } using the validation dataset, followed by a ﬁner search starting from this coarse value.
We did not use lateral connections for the two fully-connected layers.
Results
Model performance.
Table 2 summarizes model test accuracy with and without regularization (weight decay + dropout) for the network architectures described in Table 1.
For MNIST, the learned lateral connections provide improvement over the baseline and parameter-matched models only at higher levels of Gaussian noise (0.5) and salt-and-pepper noise (0.3 and above).
For CIFAR-10, we found that the learned lateral connections provide improved robustness to noise for almost all noise levels, with the cost of slightly decreased accuracy on the original images. We did not try ﬁne-tuning the models after incorporating the learned lateral connections, which may help recover some of this loss in accuracy. Furthermore, we found that lateral connections combined with regularization often resulted in even better performance on both MNIST and CIFAR-10.
Effect of lateral connections. Lateral connections had two complimentary roles: redundancy reduction leading to sparsiﬁcation of feature activations on images without noise, and noise reduction on noisy images leading to feature activations closer to those on the original images.
The use of contextual information to modulate unit responses may underlie the ability of the CNNEx model to achieve higher accuracies under noisy conditions. We show both effects on features in the ﬁrst convolutional layer of our MNIST model (Figure 3). [Figure]
Discussion
In our model, lateral connections capture structure in the statistics of the world via unsupervised learning. This structure allows for inference that can make use of the integration of information across space and features. By combining these lateral connections with features learned in a supervised manner using backpropagation, the network does not learn any arbitrary structure present in the world, but only the structure of features which is needed to solve a particular task.
As a result, our method allows us to predict the structure of the world which is relevant to a given task. The vast majority of deep neural networks are feedforward in nature, although recurrent connections have been added to convolutional neural networks [24, 25]. Recurrent connections have also been used to implement different visual attention mechanisms [26, 27]. However, these networks are still largely trained in a supervised manner.
An exception are ladder networks, which have been proposed as a means to combine supervised and unsupervised learning in deep neural networks [28]. However, different from our approach, ladder networks use noise injection to introduce an unsupervised cost function based on reconstruction of the internal activity of the network. Our model instead relies on a modiﬁed Hebbian learning rule which learns the optimal lateral connections between features within each layer based solely on the activations of units coding for these features. Neurons are inherently noisy, and their responses can vary even to the same stimulus.
These neurons are embedded in cortical circuits that must perform computations in the absence of information, such as under visual occlusion. Optimal lateral connections can provide additional robustness to these networks by allowing integration of information from multiple sources (i.e. different features and spatial locations). This type of computation is also potentially useful for applications in which artiﬁcial neurons are not simulated with high ﬁdelity, e.g. in neuromorphic computing. We chose a relatively simple network architecture as a proof-of-concept for our model.
As such, we did not achieve state-of-the art performance on either image dataset.
This accuracy could be further improved by either ﬁne-tuning models after learning the optimal lateral connections or using deeper network architectures with more parameters. Future experiments will also have to test the scalability of learning optimal lateral connections on more complex network architectures and larger image datasets (e.g. ImageNet), and whether these connections provide any beneﬁt against noise or other types of perturbations such as adversarial images. Acknowledgements We wish to thank the Allen Institute founder, Paul G. Allen, for his vision, encouragement, and support.
We assume a simple neural code for each excitatory neuron: the steady-state ﬁring rate of the neuron maps monotonically to the probability of the feature that the neuron codes for being present in the image (similar to codes assumed in previous studies [29, 30, 31]). We have
[Equation]
where f nk,x represents the ﬁring rate of a neuron coding for feature F k at location n in image x p
( F nk | im x ) represents the probability of presence of the corresponding feature and g is a monotonic function. For simplicity, we assume a linear mapping between the probability of feature presence and ﬁring rate ( g ( y ) = y ) in the rest of the paper, as the qualitative conclusions are not dependent on this choice.
We subdivide the image into multiple patches corresponding to the size of the classical RF.
We deﬁne the classical RF response of the neuron (with g ( y ) = y ) as [Equation] where im nx denotes the image patch at location n .
We require that the sum of probabilities of all features in a patch is one, for every image, thereby implying a normalization of classical RF responses in a spatial region equal to the size of a patch so that(cid:88) [Equation]
We show that a network of neurons can directly implement Bayes rule to optimally integrate information from the surround (see Supplementary Information (SI) for the derivation).
Intuitively, the activity of a neuron representing a feature is inﬂuenced by the probability that another feature is present in a surrounding patch and by the statistics of co-occurrence of these features.
In such a network, the activity of a neuron representing feature j in patch m , given image x , can be shown to be (see SI)
[Equation]
In Eq.
(6), N mx represents a normalization coefﬁcient (see SI). The term W mnjk represents a weight from the neuron coding for feature k in patch n to the neuron coding for feature j in patch m and can be estimated as: [Equation]
where x spans the set of images used and (cid:104) . (cid:105) x represents the average over all images in the set. Thus, lateral connections between neurons in our network are proportional to the relative probability of feature co-occurrences above chance in the set of images used. While the formalism can be applied to any scene statistics, we focus here on the analysis of natural scenes.
Eq.
(6) encapsulates a local computation by a network of excitatory neurons - optimal context integration - through functional lateral connections given by Eq. The calculation of the steady-state responses in Eq. (6) requires ﬁrst summing the contributions from the lateral connections corresponding to a set of neurons in a surrounding patch n , and then multiplying the contributions from each such patch with the classical RF response.
However, if the contributions from the surround are sufﬁciently small, the computation becomes simpler and Eq.
(6) for the ﬁring rate of the neuron simpliﬁes to [Equation] Extension to multi-layer networks
When applied to multi-layer networks (e.g. deep neural networks), we treat each feature map as containing units which respond to a given feature at a speciﬁc location within the image.
For the ﬁrst layer of the network (which sees the image as input), the learned lateral connections are captured by the derivations above.
For deeper layers, we use the same formalism and set of assumptions, learning lateral connections between the hidden units based on their activations over a set of training images.
During inference, we pass the real-valued activations modulated by the learned lateral connections onto the next layer (we do not perform any probabilistic sampling).