  We study low-rank parameterizations of weight matrices with embedded spectral
properties in the Deep Learning context. The low-rank property leads to
parameter efficiency and permits taking computational shortcuts when computing
mappings. Spectral properties are often subject to constraints in optimization
problems, leading to better models and stability of optimization. We start by
looking at the compact SVD parameterization of weight matrices and identifying
redundancy sources in the parameterization. We further apply the Tensor Train
(TT) decomposition to the compact SVD components, and propose a non-redundant
differentiable parameterization of fixed TT-rank tensor manifolds, termed the
Spectral Tensor Train Parameterization (STTP). We demonstrate the effects of
neural network compression in the image classification setting and both
compression and improved training stability in the generative adversarial
training setting.
