Bio-inspired learnable divisive normalization for ANNs
Abstract
In this work we introduce DivNormEI, a novel bio-inspired convolutional network that performs divisive normalization , a canonical cortical computation, along with lateral inhibition and excitation that is tailored for integration into modern Artiﬁcial
Neural Networks (ANNs).
DivNormEI, an extension of prior computational models of divisive normalization in the primate primary visual cortex, is implemented as a modular layer that can be integrated in a straightforward manner into most commonly used modern ANNs.
DivNormEI normalizes incoming activations via learned non-linear within-feature shunting inhibition along with across-feature linear lateral inhibition and excitation.
In this work, we show how the integration of
DivNormEI within a task-driven self-supervised encoder-decoder architecture encourages the emergence of the well-known contrast-invariant tuning property found to be exhibited by simple cells in the primate primary visual cortex.
Additionally, the integration of DivNormEI into an ANN (VGG-9 network) trained to perform image classiﬁcation on ImageNet-100 improves both sample efﬁciency and topaccuracy on a held-out validation set.
We believe our ﬁndings from the bio-inspired
DivNormEI model that simultaneously explains properties found in primate V neurons and outperforms the competing baseline architecture on large-scale object recognition will promote further investigation of this crucial cortical computation in the context of modern machine learning tasks and ANNs.
Introduction
Past computational models of vision have served the important coupled goals of understanding biological vision and progressing towards creating machines with powerful visual capability.
Hubel and Wiesel’s seminal work characterizing receptive ﬁelds in the cat striate cortex (Hubel & Wiesel, 1968)
inspired Fukushima’s Neocognitron (Fukushima, 1980)
(a hierarchical extension of this building block) and later the LeNet (LeCun et al., 1998) model that combined convolutional neural networks with gradient-based learning, and is the predecessor to most of today’s modern ANNs that achieve tremendous success in the ﬁeld of computer vision.
Following this line of brain-inspired architectures for computer vision, in this work we introduce
DivNormEI, a novel computational model of divisive normalization and lateral interactions that is an extension of prior computational neuroscience models of normalization and horizontal connections (Blakeslee & McCourt, 1999; Robinson et al.; Schwartz & Simoncelli, 2001; Grossberg & Raizada, 2000; Li, 1998).
Divisive normalization has been extensively studied in the ﬁelds of neuroscience and perception; these studies have highlighted the importance of this canonical computation for several traits of biological vision such as nonlinear response properties, efﬁcient coding, invariance with respect to speciﬁc stimulus dimensions and redundancy reduction. Here, we combine divisive 3rd Workshop on Shared Visual Representations in Human and Machine Intelligence (SVRHM 2021) the Neural Information Processing
, Virtual. normalization with linear lateral interactions to design a single circuit that we call DivNormEI.
Different from prior models that were ﬁt to explain behavioral/physiological data or used a small stimulus set, here We explore training DivNormEI’s parameters in a task-driven fashion by optimizing performance on large-scale supervised object recognition and by optimizing self-supervised objective functions with gradient-based learning.
We demonstrate the emergence of the ubiquitous contrast invariant tuning property in a self-supervised ANN equipped with DivNormEI.
Additionally, we report our observation of improved large-scale object recognition accuracy on the ImageNetdataset by virtue of the DivNormEI block.
Comparing the tuning properties of our model’s simplecell equivalent neurons pre- and post-normalization, we observe the crucial role played by lateral connections and normalization to emergence of contrast invariance.
We observed that a VGGnetwork pretrained on the large-scale ImageNet dataset did not show this property of contrast invariant tuning, further highlighting the speciﬁc role of normalization and lateral connections in developing this particular invariance.
We hypothesize that the hierarchical incorporation of our DivNormEI (out of the scope of this paper) will promote development of invariance to more stimulus dimensions that shall be advantageous for high-level vision tasks such as image segmentation and object detection.
Methods
In this work, we extend prior computational models of divisive normalization and lateral interactions, and implement them within the framework of modern ANNs.
We develop a learnable version of this nonlinear normalization computation along with linear excitatory and inhibitory lateral connections in a single module we call DivNormEI explained as follows.
2.1 DivNormEI layer: Learnable divisive normalization with lateral connections
We begin by deﬁning a typical instantiation of divisive normalization that has been widely studied by prior art wherein, divisive normalization computes the ratio of an individual neuron’s response to the summed activity of other neurons in its neighborhood.
The following equation summarizes this well-studied formulation of a neuron i ’s normalization corresponding to an input stimulus drive x :
[Equation]
where j represents neighboring neurons of i and w , σ are learnable free parameters.
We now deﬁne our proposed learnable divisive normalization layer with lateral connections called
DivNormEI.
Let Y be an intermediate feature map – in response to stimulus drive X – that is being normalized by DivNormEI s.t.
Y ∈
IR h , w , c .
c is the number of features present in Y (e.g. if Y is the conv1 feature map in a ResNet-50 network, c = 64 ) while h , and w are the spatial dimensions of Y .
In our model, each neuron in feature map k at spatial location i, j receive three kinds of activity modulation from neighboring neurons with i, j at their center: (i) divisive (or shunting) modulation that performs learned upscaling / downscaling (similar to gain control), (ii) linear excitatory modulation that positively inﬂuences activity with a learned additive operation and (iii) linear inhibitory modulation that negatively inﬂuences activity with a subtractive operation.
Once the intermediate feature map Y arrives into the DivNormEI block, the ﬁrst modulatory inﬂuence described above, i.e., divisive normalization of incoming features is performed as follows:
[Equation]
Per the above equation, divisive normalization of neurons in feature map k is performed using the weighted summed activity of neighboring neurons (indexed by m, n ) with a learnable 2-D depthwiseconvolutional weight matrix
w divk ∈
IR h
div , w div that is unique to each feature map.
h div , w div represent the spatial extent of divisive normalization.
Neurons in feature map k are normalized only by their spatial neighbors in the same feature map k .
This particular design choice ensures that activations in neighboring spatial locations have reduced redundancy after divisive normalization.
The divisively normalized output ˜ Y i,j,k from Eqn. 2 is then modulated by two opposing lateral interactions: additive excitation and subtractive inhibition.
Two learnable weight matrices w exc ∈
IR h e , w e , c , c and w inh ∈
IR h
i , w i , c , c compute the weighted summed activity of neighboring neurons from all feature channels for excitation and inhibition as follows ( h e , w e and h i , w i denote the spatial extent of excitatory and inhibitory lateral connections):
[Equation]
Eqn. 3 and 4 are implemented as 2D convolutions with kernels w exc and w inh respectively.
Linear excitation and inhibition E and I are integrated with ˜ Y as follows to produce the normalized output:
[Equation]
In the above Eqn. 6, BN corresponds to batch normalization (Ioffe & Szegedy, 2015) and γ represents the ReLU nonlinearity.
In our experiments, we initialize w div to be a set of c 2-D Gaussian kernels to build strong local divisive inhibition that prevent redundancies with gradually decreasing inhibition from far-off neurons.
However, it is to be noted that these parameters are also trained with backpropagation along with w exc and w inh that are initialized randomly.
Unless speciﬁed otherwise, all lateral connection weights w ∗ are maintained non-negative at each step of training.
In subsequent sections, we discuss our experiments training ANNs with DivNormEI using self-supervised and supervised objective functions.
In all our experiments, we set h div =
w div = 5 , h e = w e = {+
 and h+} i =
w i = 7 based on hyperparameter optimization w.r.t ImageNet-100 validation accuracy.
Experiments 3.1 Experiment 1: In-
silico electrophysiology with task-driven ANN
Berkeley Segmentation Dataset 500.
In this experiment, we explored the self-supervised task of image super-resolution on natural images from the Berkeley Segmentation Dataset (Arbelaez et al., 2010), referred from here onward as BSDS500.
For training super-resolution models in this experiment, we sample random crops of size 48x48 pixels from the training images of BSDS (original image size is 321x481 pixels) and use them as the high-resolution ground truth images.
Corresponding low-resolution input images are obtained by down-sampling the ground truth images by a factor of 4 to size 12x12 pixels.
Encoder-decoder architecture for super-resolution.
We implemented an encoder-decoder architecture for super-resolution.
The encoder contains a ﬁxed convolution layer initialized with a Gabor ﬁlter bank and a DivNormEI layer.
The Gabor ﬁlter bank contains square ﬁlters of size 15px and 21px.
At each ﬁlter size, we design ﬁlters selective to 4 orientations ( θ = 0 , π/ 4 , π/ 2 , 3 π/ 4 ), 2 spatial frequencies (2 cycles per degree, 3 cycles per degree) and 4 phase values ( φ = 0 , π/ 2 , π, 3 π/ 2 ).
The encoder’s output thus contains 64 feature maps.
The decoder consists of 3 layers that upsample the encoder’s output; ﬁrst two layers are instantiated with transposed convolution (Zeiler & Fergus, 2014) with 64 ﬁlters each followed by batch normalization and ReLU nonlinearity.
The last decoder layer is a 1x1 convolution with tanh nonlinearity that produces the ﬁnal output in image space.
in primary visual cortex of cats and primates maintain contrast-invariant orientation tuning, i.e., the orientation selective response of neurons remains roughly steady despite varying input stimulus contrast (Troyer et al., 1998; Nowak & Barone, 2009).
In this paper, we studied whether lateral connections and data-driven self-supervised learning can contribute to the emergence of this property.
To study this hypothesis, we generated 100 sinusoidal grating stimuli that correspond to 25 grating orientations obtained at uniform intervals between 0 and π at 4 contrast levels (shown in Fig.
1.E).
For each of the 64 feature maps in our encoder, we computed the neural tuning curves in response to the above 100 stimuli.
Using stimuli at each contrast level, we computed the average of these tuning curves after ordering them such that each neuron’s response to its preferred orientation stimulus was at the center of its tuning curve.
The average tuning curves before normalization are shown as a function of stimulus contrast in
Fig.1.A, wherein the orientation selectivity decreases with decreasing stimulus contrast .
This is similar to the average tuning curves of an ImageNet pretrained VGG-16 that we show in Fig.
1.B, i.e., both the self-supervised pre-normalization encoder neurons and ImageNet-pretrained VGGneurons behave similarly and show a lack of contrast invariance.
On the other hand, the average tuning curve (of the same neurons in Fig.1.A) after normalization using DivNormEI as shown in Fig.1.C post-normalization is signiﬁcantly more invariant to stimulus contrast (orientation selectivity and tuning curve variance is consistent across contrast levels).
This post-normalization behavior shown in Fig.
1.C is similar to the reference behavior of cat primary visual cortical neurons we show in Fig.
1.B obtained from (Busse et al., 2009).
3.2 Experiment 2: DivNormEI improves object recognition accuracy on ImageNet-
In this experiment, we evaluated the utility of our proposed DivNormEI model for the computer vision task of object recognition on the ImageNet-100 dataset (a subset of the ImageNet dataset with 100 randomly sampled classes which are also present in the validation set, standardized by Tian et al.
For fast prototyping and GPU memory constraints, we created a custom shallower variant of the VGG architecture with 9 layers of processing that we used in this experiment.
We compared the following two architectures on ImageNet-100 classiﬁcation: (a) Baseline-VGG9 – a 9-layer
ANN with 3x3 convolution layers, max pooling and fully connected layers for classiﬁcation, and (b)
DivNorm-VGG9 – a 9-layer ANN with the same architecture as Baseline-VGG9, with the addition of an end-to-end trainable DivNormEI block at the output of the ﬁrst convolution layer (output of
DivNormEI block sent to subsequent 8 layers for classiﬁcation).
We trained two initializations of each of these models wherein the ﬁrst pair of Baseline-VGG9 and DivNorm-VGG9 were initialized with the same weights, same as the second pair of Baseline-VGG9 and DivNorm-VGG9 models.
We observed that DivNorm-VGG9 models outperformed Baseline-VGG9 models on the ImageNet100 validation set.
DivNorm-VGG9 had better sample efﬁciency than Baseline-VGG9, and was more accurate in classiﬁcation by 1.8% (Top-1 validation accuracy of Baseline-VGG9: 73.3%, DivNormVGG9: 75.12% ) .
This observation suggests that DivNormEI is also relevant to improving the discriminative power of modern ANNs and can be integrated into further computer vision solutions like image segmentation and detection that rely on object semantics and discriminability.
Related work
Computational models of divisive normalization and lateral connections have been studied previously by the neuroscience and vision science communities.
Of these models, we ﬁnd Schwartz & Simoncelli (2001)’s model to be most relevant to our divisive normalization computation, where the authors developed a simple yet neurally plausible circuit for divisive normalization optimized to maximize independence of ﬁlter responses.
Our model is also related to Robinson et al.
’s model that performs orientation- and spatial-frequency dependent normalization of ﬁlter responses with model parameters ﬁt to best explain a suite of brightness illusions.
Our work is also related to prior computational models of lateral interactions in the primary visual cortex such as Li (1998); Grossberg & Raizada (2000).
Our proposed model is an extension of these sophisticated yet small-scale models (in terms of size and stimulus exposure) to integration within large-scale gradient-trained ANNs.
We ﬁnd our work to be related to Ballé et al.
(2015) where a learnable Generalized Divisive Normalization (albeit without lateral connections that are present in our model) for the task of image density modeling and compression.
Our proposed work is also related to Burg et al.
(2021) where the authors develop an end-to-end trainable model of divisive normalization similar to ours.
Key differences between our work and the above work are: (i) our model performs linear lateral inhibition and excitation on top of learnable divisive normalization, (ii) parameters of our model are estimated with computer vision tasks such as super-resolution and image recognition, whereas Burg et al.
(2021) train their model to predict V1 neuronal responses recorded from macaque primary visual cortex.
Discussion
We developed DivNormEI, a novel computational model of divisive normalization and lateral interactions – both canonical computations that are ubiquitously found in biological visual neurons associated with diverse functions such as contrast normalization, redundancy reduction, and nonlinear neuronal response properties.
We conducted two experiments to address (1) the emergent biological similarity from data-driven training of our model and (2) its utility in modern ANNs trained on computer vision problems.
We computed the orientation tuning curves of neurons post normalization by DivNormEI and observed their response to be invariant to input stimulus contrast.
This property of contrast invariant tuning is similar to that of primary visual cortical neurons.
We also tested the speciﬁc role of divisive normalization in developing contrast invariant tuning; i.e., an
ImageNet-pretrained VGG-16 model exposed to a million natural images still does not possess this property.
We also compared two pairs of identical convolutional architectures with the difference that one network among each pair contained a DivNormEI layer after its ﬁrst convolution layer on large-scale object recognition from images in the ImageNet-100 dataset.
We observed that the architectures with DivNormEI blocks possessed higher sample efﬁciency and classiﬁcation accuracy compared to their identical baseline architectures without DivNormEI.
We ﬁnd this superior performance of DivNorm-architectures suggestive of the role of DivNormEI (and similar brain-inspired computations) in improving performance on computer vision tasks like image segmentation, where object discriminability is key.
The studies in this paper were limited to speciﬁc forms of lateral interaction and by application to smaller-sized deep networks due to time-limited computational constraints.
However, we believe that our promising initial ﬁndings encourage further investigation of the role and implementation of divisive normalization and other relevant lateral and recurrent cortical computations in modern ANN architectures.  Acknowledgments
This work was supported by the Sony Research Award program from Sony Research, the Kavli
Symposium Inspired Proposals award from the Kavli Institute of Brain and Mind and support from the Department of Cognitive Science at UC San Diego.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] Code, data required to reproduce results may be released upon reasonable request upon publication.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]