This paper presents multistream CNN, a novel neural network architecture for robust acoustic modeling in speech recognition tasks. The proposed architecture accommodates diverse temporal resolutions in multiple streams to achieve robustness in acoustic modeling. For the diversity of temporal resolution in embedding processing, we consider dilation on TDNN-F, a variant of 1D-CNN .  Each stream stacks narrower TDNN-F layers whose kernel has a unique, stream-specific dilation rate when processing input speech frames in parallel. Hence it can better represent acoustic events without the increase of model complexity . We validate the effectiveness of the proposed multistream CNN architecture by showing consistent improvement across various data sets. Trained with data augmentation methods, multistream CNN improves the WER of the test-other set in the LibriSpeech corpus by 12\% (relative). On custom data from ASAPP's production  system for a contact center, it records a relative WER improvement of 11\% for the customer channel audios (10\% on average for the agent and customer channel recordings) to prove the superiority of the proposed model architecture in the wild. In terms of real-time factor (RTF) , multistream CNN outperforms the normal TDNN-F by 15\%, which also suggests its practicality on production systems or applications .
This paper proposes multistream CNN, a novel neural network architecture for robust acoustic modeling in speech recognition tasks. The proposed architecture processes input speech with diverse temporal resolutions by applying different dilation rates to convolutional neural networks across multiple streams to achieve the robustness . The dilation rates are selected from the multiples of a sub-sampling rate of 3 frames. Each stream stacks TDNN-F layers (a variant of 1D CNN), and output embedding vectors from the streams are concatenated then projected to the final layer . We validate the effectiveness of the proposed multistream CNN architecture by showing consistent improvements against Kaldi's best TDNN-F model across various data sets. Multistream CNN improves the WER of the test-other set in the LibriSpeech corpus by 12\% (relative). On custom data from ASAPP's production ASR system for a contact center, it records a relative WER improvement of 11\% for customer channel audio to prove its robustness to data in the wild. In terms of real-time factor , multistream CNN outperforms the baseline TDNN-F by 15\%, which also suggests its practicality on production systems . When combined with self-attentive SRU LM rescoring, multistream CNN contributes for ASAPP to achieve the best WER of 1.75\% on test-clean in LibriSpeech .
R
presents
proposes
R
accommodates
processes input speech with
R
in
by applying different dilation rates to convolutional neural networks across
R
robustness in acoustic modeling. For the diversity of temporal resolution in embedding processing, we consider dilation on TDNN-F, a variant of 1D-CNN
the robustness
A
The dilation rates are selected from the multiples of a sub-sampling rate of 3 frames.
D
narrower
R
whose kernel has a unique, stream-specific dilation rate when processing input speech frames in parallel. Hence it can better represent acoustic events without the increase of model complexity
(a variant of 1D CNN), and output embedding vectors from the streams are concatenated then projected to the final layer
R
improvement
improvements against Kaldi's best TDNN-F model
R
Trained with data augmentation methods, multistream
Multistream
A
ASR
R
the customer channel audios (10\% on average for the agent and customer channel recordings) to prove the superiority of the proposed model architecture
customer channel audio to prove its robustness to data
D
(RTF)
R
normal
baseline
R
or applications
. When combined with self-attentive SRU LM rescoring, multistream CNN contributes for ASAPP to achieve the best WER of 1.75\% on test-clean in LibriSpeech