  Convolutional neural network training can suffer from diverse issues like
exploding or vanishing gradients, scaling-based weight space symmetry and
covariant-shift. In order to address these issues, researchers develop weight
regularization methods and activation normalization methods. In this work we
propose a weight soft-regularization method based on the Oblique manifold. The
proposed method uses a loss function which pushes each weight vector to have a
norm close to one, i.e. the weight matrix is smoothly steered toward the
so-called Oblique manifold. We evaluate our method on the very popular
CIFAR-10, CIFAR-100 and ImageNet 2012 datasets using two state-of-the-art
architectures, namely the ResNet and wide-ResNet. Our method introduces
negligible computational overhead and the results show that it is competitive
to the state-of-the-art and in some cases superior to it. Additionally, the
results are less sensitive to hyperparameter settings such as batch size and
regularization factor.
