  This paper is the first study to apply deep mutual learning (DML) to
end-to-end ASR models. In DML, multiple models are trained simultaneously and
collaboratively by mimicking each other throughout the training process, which
helps to attain the global optimum and prevent models from making
over-confident predictions. While previous studies applied DML to simple
multi-class classification problems, there are no studies that have used it on
more complex sequence-to-sequence mapping problems. For this reason, this paper
presents a method to apply DML to state-of-the-art Transformer-based end-to-end
ASR models. In particular, we propose to combine DML with recent representative
training techniques. i.e., label smoothing, scheduled sampling, and
SpecAugment, each of which are essential for powerful end-to-end ASR models. We
expect that these training techniques work well with DML because DML has
complementary characteristics. We experimented with two setups for Japanese ASR
tasks: large-scale modeling and compact modeling. We demonstrate that DML
improves the ASR performance of both modeling setups compared with conventional
learning methods including knowledge distillation. We also show that combining
DML with the existing training techniques effectively improves ASR performance.
