  Although the lower layers of a deep neural network learn features which are
transferable across datasets, these layers are not transferable within the same
dataset. That is, in general, freezing the trained feature extractor (the lower
layers) and retraining the classifier (the upper layers) on the same dataset
leads to worse performance. In this paper, for the first time, we show that the
frozen classifier is transferable within the same dataset. We develop a novel
top-down training method which can be viewed as an algorithm for searching for
high-quality classifiers. We tested this method on automatic speech recognition
(ASR) tasks and language modelling tasks. The proposed method consistently
improves recurrent neural network ASR models on Wall Street Journal,
self-attention ASR models on Switchboard, and AWD-LSTM language models on
WikiText-2.
