  Classifiers that are linear in their parameters, and trained by optimizing a
convex loss function, have predictable behavior with respect to changes in the
training data, initial conditions, and optimization. Such desirable properties
are absent in deep neural networks (DNNs), typically trained by non-linear
fine-tuning of a pre-trained model. Previous attempts to linearize DNNs have
led to interesting theoretical insights, but have not impacted the practice due
to the substantial performance gap compared to standard non-linear
optimization. We present the first method for linearizing a pre-trained model
that achieves comparable performance to non-linear fine-tuning on most of
real-world image classification tasks tested, thus enjoying the
interpretability of linear models without incurring punishing losses in
performance. LQF consists of simple modifications to the architecture, loss
function and optimization typically used for classification: Leaky-ReLU instead
of ReLU, mean squared loss instead of cross-entropy, and pre-conditioning using
Kronecker factorization. None of these changes in isolation is sufficient to
approach the performance of non-linear fine-tuning. When used in combination,
they allow us to reach comparable performance, and even superior in the
low-data regime, while enjoying the simplicity, robustness and interpretability
of linear-quadratic optimization.
