  We present a simple and efficient auxiliary loss function for automatic
speech recognition (ASR) based on the connectionist temporal classification
(CTC) objective. The proposed objective, an intermediate CTC loss, is attached
to an intermediate layer in the CTC encoder network. This intermediate CTC loss
well regularizes CTC training and improves the performance requiring only small
modification of the code and small and no overhead during training and
inference, respectively. In addition, we propose to combine this intermediate
CTC loss with stochastic depth training, and apply this combination to a
recently proposed Conformer network. We evaluate the proposed method on various
corpora, reaching word error rate (WER) 9.9% on the WSJ corpus and character
error rate (CER) 5.2% on the AISHELL-1 corpus respectively, based on CTC greedy
search without a language model. Especially, the AISHELL-1 task is comparable
to other state-of-the-art ASR systems based on auto-regressive decoder with
beam search.
