Self-supervised acoustic pre-training has achieved impressive results on low-resource speech recognition tasks . It indicates that the pretrain-and-finetune paradigm is a promising direction. In this work, we propose an end-to-end model for the low-resource speech recognition, which fuses a pre-trained audio encoder (wav2vec2.0) and a pre-trained text decoder (BERT) . The two modules are connected by a linear attention mechanism without parameters. A fully connected layer is introduced for  hidden mapping between speech and language modalities. Besides, we design an effective fine-tuning strategy to preserve and utilize the text context modeling ability of the pre-trained decoder. Armed with this strategy, our model exhibits distinct faster convergence and better performance . Our model achieves approaching recognition performance in CALLHOME corpus ( 15h) as the SOTA pipeline modeling .
End-to-end models have achieved impressive results on the task of automatic speech recognition (ASR). For low-resource ASR tasks, however, labeled data can hardly satisfy the demand of end-to-end models. Self-supervised acoustic pre-training has already shown its amazing ASR performance, while the transcription is still inadequate for language modeling in end-to-end models . In this work, we fuse a pre-trained acoustic encoder (wav2vec2.0) and a pre-trained linguistic encoder (BERT) into an end-to-end ASR model. The fused model only needs to learn the transfer from speech to language during fine-tuning on limited labeled data. The length of the two modalities is matched by a monotonic attention mechanism without additional parameters. Besides, a fully connected layer is introduced for the hidden mapping between modalities. We further propose a scheduled fine-tuning strategy to preserve and utilize the text context modeling ability of the pre-trained linguistic encoder. Experiments show our effective utilizing of pre-trained modules . Our model achieves better recognition performance on CALLHOME corpus ( 15 hours) than other end-to-end models .
R
Self-supervised acoustic pre-training has
End-to-end models have
R
low-resource speech recognition tasks
the task of automatic speech recognition (ASR). For low-resource ASR tasks, however, labeled data can hardly satisfy the demand of end-to-end models. Self-supervised acoustic pre-training has already shown its amazing ASR performance, while the transcription is still inadequate for language modeling in end-to-end models
D
It indicates that the pretrain-and-finetune paradigm is a promising direction.
R
propose an end-to-end model for the low-resource speech recognition, which fuses
fuse
R
audio
acoustic
R
text decoder
linguistic encoder
R
. The two modules are connected by a linear
into an end-to-end ASR model. The fused model only needs to learn the transfer from speech to language during fine-tuning on limited labeled data. The length of the two modalities is matched by a monotonic
R
parameters. A
additional parameters. Besides, a
A
the
R
speech and language modalities. Besides, we design an effective
modalities. We further propose a scheduled
R
decoder. Armed with this strategy, our model exhibits distinct faster convergence and better performance
linguistic encoder. Experiments show our effective utilizing of pre-trained modules
R
approaching recognition performance in
better recognition performance on
R
15h) as the SOTA pipeline modeling
15 hours) than other end-to-end models