Visual response inhibition for increased robustness of convolutional networks to distribution shifts
Abstract
Convolutional neural networks have been shown to suffer from distribution shifts in the test data, for instance caused by the so called common corruptions and perturbations.
Test images can contain noise, digital transformations, and blur that were not present in the training data, negatively impacting the performance of trained models.
Humans experience much stronger robustness to noise and visual distortions than deep networks.
In this work, we explore the effectiveness of a neuronal response inhibition mechanism, called push-pull, observed in the early part of the visual system, to increase the robustness of deep convolutional networks.
We deploy a Push-Pull inhibition layer as a replacement of the initial convolutional layers (input layer and in the first block of residual and dense architectures) of standard convolutional networks for image classification.
We show that the PushPull inhibition component increases the robustness of standard networks for image classification to distribution shifts on the CIFAR10-C and CIFAR10-P test sets.
Introduction
Despite exceptional results and over-human performance achieved on benchmark visual processing tasks, f.i. image classification (1; 2; 3; 4) convolutional networks have shown decrease of performance when the test images are affected by unexpected changes (5).
The nature of these changes can be artificial, resulting in subtle and imperceptible adversarial attacks (6; 7; 8; 9) or natural, determined by common corruptions and perturbations in computer vision, such as noise, contrast changes, elastic transformations, blur and so on (10).
The lack of robustness has been attributed to the tendency of the network to learn superficial statistics that exist in the training data (11; 12) that favour ‘easy’ solutions to the optimization problem, namely a simplicity bias (13).
This is also regarded as shortcut learning (14)
One common approach to increase the generalization and robustness to distribution shifts of convolutional networks is data augmentation, in which models are trained by extending the training data with (patch-wise) noise (15) or blur(16), masking random patches (17), or with more advanced schemes such as AutoAugment (18; 19), AugMix (20) or DeepAugment (21), among others.
Data augmentation is a powerful tool, but can cause underfitting (5) or a misleading sense of robustness.
For example, using high-frequency noise for data augmentation (e.g. Gaussian noise) makes the network focusing on low frequency features that are discriminant instead of becoming explicitly robust to high-frequency corruptions (22).
In (22), it was suggested that architectural changes to networks, and robustness by design are promisng directions to undertake, and eventually be combined with data augmentation.
The human visual system, instead, has remarkably strong generalization capabilities across a large range of changes of the input distribution, e.g. illumination changes, blur and motion-related distortion, noise, and weather conditions (5), among others.
Part of the robustness of the visual system has been attributed to neural response inhibition mechanisms (23), e.g. the push-pull inhibition that is exhibited by simple neurons and acts as a spatial frequency tuning function to increase the stability of cortical activity (24).
A simple neuron that exhibits push-pull inhibition is believed to be connected with an interneuron whose activity suppresses (inhibits) the activity of the neuron concerned.
An inhibitory interneuron responds to the same preferred stimulus, but of opposite contrast , of the neuron to which it is connected.
For simplification sake, we can think of a push-pull neuron as having two receptive fields, referred to as the excitatory (push) and inhibitory (pull).
Both receptive fields are similar in structure but are in antiphase of each other, with the inhibitory receptive field tending to be larger (25).
A stimulus that elicits the activation of the interneuron may suppress, to some extent, the response of the excitatory one.
A computational model of the push-pull inhibition was introduced in image processing filters with application to contour and line (26; 27; 28) detection that demonstrated its outstanding performance in terms of robustness against various types of noise.
In this work, we deploy a push-pull layer (29) as a substitute of the classical convolutional layer in CNNs and investigate its impact on improving the robustness of image classification models to distribution shifts caused by common corruptions and perturbations using the CIFAR10-C and CIFAR10-P benchmarks (10).
We build on (29), and here we deploy the push-pull layer to replace all convolutions up to the first block of residual and dense networks.
Furthermore, we combine these modifications with data augmentation strategies to train networks, demonstrating their complementarity to further boost robustness to distribution shifts caused by visual corruptions and perturbations.  Push-Pull inhibition layer
[Figure]
The pull kernel is derived from the push one by negation and upsampling (by a factor h ).
Their responses are rectified and combined linearly.
The response of a push-pull layer (Figure 1) is the combination of the activation map of two convolutions processed with a non-linear function (ReLU, in our case), one with an excitatory (push) kernel k ( · ) and one with an inhibitory (pull) kernel − ˆ k ( · ) (29).
The response R of a push-pull layer is computed as:
[Equation]
where Θ( · ) is the ReLU function, α is the inhibition strength, k is the push kernel and ˆ k h ↑ is its upsampled (by a factor h ) and negated version, i.e. the pull kernel.
The pull kernel has a larger receptive field, to allow for suppression of the inhibitory response on the pattern of interest and in the immediate surrounding, as found in neurophysiological studies (30).
In the experiments, the value α and h were set to 1 and 2 , respectively, following indications in (29) and a grid search.
Figure 2 shows an example of the application of one push-pull operation.
An implementation of the push-pull layer is publicly available 1 .
2.1 Push-pull in residual and dense blocks
We extend the residual and dense composite layers (1; 3) by substituting the convolutional layers with push-pull layers.
This allows us to investigate the affect of the inhibition component up to the first block of ResNet and DenseNet architectures with respect to the robustness against distribution shifts.
The design decision to deploy the push-pull inhibition up to the fist block is inspired by neurophysiological studies that observed this kind of inhibition exihibited in area V1 of the visual cortex (24).
In the Appendix, we report a scheme of the modified residual and dense layers.
Experiments and results
Training and evaluation.
We trained several ResNet and DenseNet models and their versions with the push-pull layer, using the CIFAR10 training set to which we applied a light data augmentation, [Figure] [Table] consisting only of center-cropping and random horizontal flipping (31).
For each architecture, we trained the original model with convolutional layers and two other models, one (-pp) with push-pull in the first layer only and the other (-b1) with the push-pull layer replacing all convolutions in the first block of the network as well.
Training details are reported in the Appendix.
We applied the evaluation protocol proposed in (10), testing the models also on the corruption
CIFAR10-C dataset and the perturbation CIFAR10-P dataset.
Besides the classification error, we compute the corruption error C , i.e. the classification error achieved on the CIFAR10-C set, and the flip-probability FP , that is the probability that the outcome of the classification changes between consecutive frames, on the sequences in the CIFAR10-P set.
We compute the mean and relative corruption errors mCE and rCE and the normalized flip probability, i.e. the flip rate FR .
Details about these metrics are in (10).
Robustness to corruptions and perturbations.
In Table 1, we report the results of ResNet and
DenseNet networks on the CIFAR10 and CIFAR10-C/P test sets.
The lower the numbers, the better the performance.
We tested architectures with different number of layers (and growing factors, for
DenseNet).
The measurements mCE , rCE and mFR are normalized with respect to the results of the baseline AlexNet.
The push-pull layer contributes to the improvement of the robustness of an existing model to distribution shifts caused by corruptions or perturbations of the input images, with negligible reduction
[Table]
of the classification accuracy on the original data.
Its effect, especially when it is deployed in the first block of existing convolutional networks, is noticeable, with a reduction of the mean corruption error mCE by more than 10% with respect to the counterpart network without inhibition, in some cases.
The improvement in robustness trades off with a generally small reduction of the classification accuracy on original (clean) test data.
The presence of the push-pull layer has a substantial effect in reducing the relative corruption error rCE , i.e. the average gap between the classification error on clean data (on which the model is trained) and corrupted data.
It indicates that models with push-pull inhibition are generally better in generalizing to data with heavy corruptions.
In some cases, the relative error is decreased by more than 30% .
The flip rate ( mFR ), computed on the
CIFAR10-P sequences, also improved for all models with push-pull layers.
We report detailed results per corruption type in the Appendix.
Networks with the push-pull layers generally show an improvement of robustness to input corruptions with high-frequency components.
For instance, ResNet20-pp and ResNet20-b1 reduced the CE by 10% to 25% on noise (Gaussian, shot, impulse), up to 18% on glass blur, and between 15% and 20% on pixelate and jpeg compression corruptions with respect to that of the original ResNet20.
Marginal improvements or lower robustness are obtained on low-frequency corruptions, such as defocus, motion and zoom blur ( +2 / 6% of CE), snow, frost and fog (between − 4% and +4% of
CE).
We include the detailed results per corruption and perturbation type in the Appendix.
Push-pull inhibition and data augmentation.
We also trained WideResNet models (2) with and without the push-pull layer, using different data augmentations, namely cutout (17), AutoAugment (18), and a combination of cutout and AutoAugment.
We show that the push-pull layer can be combined with the use of data augmentation techniques to further improve the model robustness to different corruptions and perturbations of test images.
In
Table 2, we report the robustness results ( mCE and FR ) of WideResNet models (with and without push-pull inhibition) trained with data augmentation techniques.
The cutout augmentation does not increase the robustness of the models to input corruptions, while AutoAugment contributes to a noticeable reduction of the corruption error.
This is in line with the findings in (22).
In all cases, push-pull inhibition layers contribute to further increase the robustness of networks trained with data augmentation.
The best mCE and FR values are obtained by models that deploy the push-pull layer in the first layer and in the first residual block, and are trained using AutoAugment plus cutout augmentations.
In the Appendix, we report detailed results per type of corruption and perturbation.
Conclusions
We demonstrated that the inclusion of a response inhibition mechanism, inspired by the push-pull neurons of the visual system of the brain, into existing convolutional networks improves their robustness to corruptions and perturbations of the input that are not present in the training data.
The improvement is mainly attributable to the filters in the push-pull layer that are less sensitive to noise and perturbations, especially at high frequency.
We carried out experiments on the CIFAR-C/P datasets and the results that we achieved (more than 10% of reduction of the error on corrupted and perturbed data) demonstrate the effectiveness of the push-pull inhibition layer.
A Push-pull residual and dense layers
In Figure 3, we show the structure of the composite residual and dense layers augmented with the push-pull inhibition.
The architecture of the layers is the same of the original residual and dense layers, with the difference that all convolutions are replaced by push-pull operators.
[Figure]
B Training details
For (wide)ResNet- and DenseNet-based models we optimized the parameters with stochastic gradient descent (SGD), Nesterov momentum equals to 0 .
9 and weight decay equals to 10 − 4 .
We trained
ResNet architectures for 200 epochs, with batch size equals to 128 and initial learning rate of 0 . 1 , which we decreased by a factor of 10 at epochs 80, 120 and 160.
We trained DenseNet models for 350 epochs with batch size of 64 and initial learning rate of 0 . 1 , which we decreased by a factor of 10 at epochs 150, 225 and 300.
These hyperparameters are the same used to train the original ResNet and DenseNet models.
The use of the push-pull layer in the first block of existing architectures does not require to change the hyperparameters used to train the original networks.
C Results per corruption type
In Table 3, we report the detailed results achieved on the corruption sets in the CIFAR10-C data set by ResNet networks with and without push-pull inhibition.
The push-pull filters are very effective on high-frequency corruptions.
In Table 4, we report detailed results per corruption type achieved by WideResNet models for which the convolutions in the first layer and in the first residual block were replaced by push-pull filters, and that are trained using different types of data augmentation, namely the cutout (17) and AutoAugment (18) techniques.
The use of the push-pull filters can be combined with data augmentation to further improve the robustness of existing models.
[Table] [Table]
D Results per perturbation type
We report in Table 5 the detailed performance results, in terms of flip probability, per perturbation type achieved on the CIFAR10-P data set by ResNet models, of different depth, that deploy the push-pull layers.
The flip probability measures the likelihood that a model changes its predictions on consecutive frames of a video on which an incremental corruption is applied.
The push-pull inhibition layers contribute to a substantial improvement of stability against high-frequency components
[Table]
[Table] In Table 6 we report detailed flip probability results per perturbation type achieved by WideResNet models that deploy push-pull filters in the first layer and in the first residual block to replace the original convolutions, and that are trained using different types of data augmentation, namely the cutout (17) and AutoAugment (18) techniques.
The push-pull inhibition and the different data augmentation techniques provide complementary contributions to the improvement of the model robustness and stability to perturbations.
This demonstrates that the push-pull layer can be deployed in existing architectures and can be combined with other approaches to further improve the classification performance
[Table]
[Table]