The introduction of deep neural networks (DNNs) leads to a significant improvement of the automatic speech recognition (ASR) performance. However, the whole ASR system remains sophisticated due to the dependent on the hidden Markov model (HMM). Recently, a new endto-end ASR framework, which utilizes recurrent neural networks (RNNs) to directly model context-independent targets with connectionist temporal classification (CTC) objective function, is proposed and achieves comparable results with the hybrid HMM/DNN system. In this paper, we investigate per-dimensional learning rate methods, ADAGRAD and ADADELTA included, to improve the recognition of the end-to-end system, based on the fact that the blank symbol used in CTC technique dominates the output and these methods give frequent features small learning rates. Experiment results show that more than 4% relative reduction of word error rate (WER) as well as 5% absolute improvement of label accuracy on the training set are achieved when using ADADELTA, and fewer epochs of training are needed. key words: connectionist temporal classification, adaptive perdimensional learning rate method, end-to-end ASR
Wang et al. REF greatly improved the recognition rate of end-to-end systems by studying the per-dimensional learning rate method of Adagrad and Adadelta.
Improved End-to-End Speech Recognition Using Adaptive Per-Dimensional Learning Rate Methods
IEICE Transactions
IEICE Transactions
Computer Science