  The Convolutional Neural Networks (CNNs) have emerged as a very powerful data
dependent hierarchical feature extraction method. It is widely used in several
computer vision problems. The CNNs learn the important visual features from
training samples automatically. It is observed that the network overfits the
training samples very easily. Several regularization methods have been proposed
to avoid the overfitting. In spite of this, the network is sensitive to the
color distribution within the images which is ignored by the existing
approaches. In this paper, we discover the color robustness problem of CNN by
proposing a Color Channel Perturbation (CCP) attack to fool the CNNs. In CCP
attack new images are generated with new channels created by combining the
original channels with the stochastic weights. Experiments were carried out
over widely used CIFAR10, Caltech256 and TinyImageNet datasets in the image
classification framework. The VGG, ResNet and DenseNet models are used to test
the impact of the proposed attack. It is observed that the performance of the
CNNs degrades drastically under the proposed CCP attack. Result show the effect
of the proposed simple CCP attack over the robustness of the CNN trained model.
The results are also compared with existing CNN fooling approaches to evaluate
the accuracy drop. We also propose a primary defense mechanism to this problem
by augmenting the training dataset with the proposed CCP attack. The
state-of-the-art performance using the proposed solution in terms of the CNN
robustness under CCP attack is observed in the experiments. The code is made
publicly available at
\url{https://github.com/jayendrakantipudi/Color-Channel-Perturbation-Attack}.
