  In this paper, we propose Selective Output Smoothing Regularization, a novel
regularization method for training the Convolutional Neural Networks (CNNs).
Inspired by the diverse effects on training from different samples, Selective
Output Smoothing Regularization improves the performance by encouraging the
model to produce equal logits on incorrect classes when dealing with samples
that the model classifies correctly and over-confidently. This plug-and-play
regularization method can be conveniently incorporated into almost any
CNN-based project without extra hassle. Extensive experiments have shown that
Selective Output Smoothing Regularization consistently achieves significant
improvement in image classification benchmarks, such as CIFAR-100, Tiny
ImageNet, ImageNet, and CUB-200-2011. Particularly, our method obtains
77.30$\%$ accuracy on ImageNet with ResNet-50, which gains 1.1$\%$ than
baseline (76.2$\%$). We also empirically demonstrate the ability of our method
to make further improvements when combining with other widely used
regularization techniques. On Pascal detection, using the SOSR-trained ImageNet
classifier as the pretrained model leads to better detection performances.
Moreover, we demonstrate the effectiveness of our method in small sample size
problem and imbalanced dataset problem.
