  While normalizations aim to fix the exploding and vanishing gradient problem
in deep neural networks, they have drawbacks in speed or accuracy because of
their dependency on the data set statistics. This work is a comprehensive study
of a novel method based on unitary synaptic weights derived from Lie Group to
construct intrinsically stable neural systems. Here we show that unitary
convolutional neural networks deliver up to 32% faster inference speeds while
maintaining competitive prediction accuracy. Unlike prior arts restricted to
square synaptic weights, we expand the unitary networks to weights of any size
and dimension.
