  Convolutional neural networks are very popular nowadays. Training neural
networks is not an easy task. Each convolution corresponds to a structured
transformation matrix. In order to help avoid the exploding/vanishing gradient
problem, it is desirable that the singular values of each transformation matrix
are not large/small in the training process. We propose three new
regularization terms for a convolutional kernel tensor to constrain the
singular values of each transformation matrix. We show how to carry out the
gradient type methods, which provides new insight about the training of
convolutional neural networks.
