  End-to-end automatic speech recognition (ASR), unlike conventional ASR, does
not have modules to learn the semantic representation from speech encoder.
Moreover, the higher frame-rate of speech representation prevents the model to
learn the semantic representation properly. Therefore, the models that are
constructed by the lower frame-rate of speech encoder lead to better
performance. For Transformer-based ASR, the lower frame-rate is not only
important for learning better semantic representation but also for reducing the
computational complexity due to the self-attention mechanism which has O(n^2)
order of complexity in both training and inference. In this paper, we propose a
Transformer-based ASR model with the time reduction layer, in which we
incorporate time reduction layer inside transformer encoder layers in addition
to traditional sub-sampling methods to input features that further reduce the
frame-rate. This can help in reducing the computational cost of the
self-attention process for training and inference with performance improvement.
Moreover, we introduce a fine-tuning approach for pre-trained ASR models using
self-knowledge distillation (S-KD) which further improves the performance of
our ASR model. Experiments on LibriSpeech datasets show that our proposed
methods outperform all other Transformer-based ASR systems. Furthermore, with
language model (LM) fusion, we achieve new state-of-the-art word error rate
(WER) results for Transformer-based ASR models with just 30 million parameters
trained without any external data.
