Wav- BERT : Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognition (2021)

Abstract: Unifying acoustic and linguistic representation learning has become increasingly crucial to transfer the knowledge learned on the abundance of high-resource language data for low-resource speech recognition. Existing approaches simply cascade pre-trained acoustic and language models to learn the transfer from speech to text. However, how to solve the representation discrepancy of speech and text is unexplored, which hinders the utilization of acoustic and linguistic information. Moreover, previous works simply replace the embedding layer of the pre-trained language model with the acoustic features, which may cause the catastrophic forgetting problem. In this work, we introduce Wav-BERT, a cooperative acoustic and linguistic representation learning method to fuse and utilize the contextual information of speech and text. Specifically, we unify a pre-trained acoustic model (wav2vec 2.0) and a language model (BERT) into an end-to-end trainable framework. A Representation Aggregation Module is designed to aggregate acoustic and linguistic representation, and an Embedding Attention Module is introduced to incorporate acoustic information into BERT, which can effectively facilitate the cooperation of two pre-trained models and thus boost the representation learning. Extensive experiments show that our Wav-BERT significantly outperforms the existing approaches and achieves state-of-the-art performance on low-resource speech recognition.

## introduction recently, automatic speech recognition (asr) has achieved remarkable success, which can be attributed to two complementary aspects: 1) designing more effective and larger deep neural networks for asr, and 2) training on a large amount of data @xcite @xcite . however, in practice, unlike the commonly used languages (e.g. english and chinese) with sufficient training data, many other languages (e.g. swahili, tamil) have only low-resource data due to the scarcity of audios and the huge labor resources consumed in transcription. in this way, the aforementioned data-driven mechanism is impractical for low-resource languages and thus suffers from unsatisfactory performance. to resolve this learning difficulty in the lowresource domain, many efforts have been devoted to leveraging unlabeled data. one mainstream research paradigm is unsupervised pre-training, or representation learning, which has achieved great success in natural language processing @xcite and received increasing attention in speech recognition @xcite . as a representation in this line, wav2vec @xcite and wav2vec 2.0 @xcite apply unsupervised contrastive pre-training and show promising results. to utilize linguistic information, some works @xcite also aim to build language models to rescore the n -best hypotheses generated by acoustic models. the most recent approach @xcite even cascaded the pre-trained wav2vec 2.0 and bert into a single model for low-resource asr. however, there leave two critical challenges on how to integrate the acoustic model and language model to utilize the contextual information of speech and text. 1) representation discrepancy: the acoustic model focuses more on local dependencies of the speech sequence, while the language model aims at capturing long-term semantic information of texts. it is desired to explore an effective model to fuse and leverage the two kinds of representation. 2) embedding inconsistency: the language model applies a token embedding layer during pre-training but previous methods @xcite simply replace the embedding layer with the features generated by the acoustic model, which may result in the catastrophic forgetting problem @xcite . to tackle the above challenges, in this work, we make the first attempt to successfully integrate the well-trained acoustic model and language model for low-resource speech recognition. towards this end, we introduce a new framework that incorporates the two kinds of pre-trained models for cooperative acoustic and linguistic representation learning by exploiting complementary contextual information of both speech and text. first, to solve representation discrepancy, unlike the previous works @xcite ) that simply connect the acoustic model and the language model by treating them as an encoder and a decoder, we consider them as two encoders that provide two different representations. specifically, we propose a representation aggregation module, a plug-in component to better exploit and fuse the acoustic and linguistic information. we design and evaluate several representation aggregation mechanisms, including gated acoustic-guided attention, gated linguistic-guided attention, and gated cross-modal attention. the experimental results show the proposed gated cross-modal attention is the most effective method for representation aggregation. second, to fill the gap of embedding inconsistency, we introduce an embedding attention module to incorporate the acoustic features into bert by a gated attention process, which not only preserves the capability of bert but also takes advantage of acoustic information. moreover, as bert requires audio transcripts as input to create word embedding, it may be easy to overfit when using ground truth transcripts. on the other hand, it is also hard to converge when using transcripts predicted by the acoustic model. to facilitate the cooperation of the two encoders, we propose a sampling strategy with decay to randomly select the ground truth and generated transcripts for smooth training. we adopt pre-trained wav2vec 2.0 @xcite and bert @xcite as the encoders to provide acoustic and linguistic representations respectively for their flexible pre-training then fine-tuning paradigm as well as excellent local contextual modeling ability. accordingly, we denominate our method as wav-bert. we evaluate our method on several datasets with diverse languages from the public iarpa babel dataset @xcite and aishell-1 corpus @xcite . the experimental results demonstrate that our wav-bert significantly outperforms the existing approaches on low-resource asr. furthermore, our exhaustive ablation studies demonstrate the effectiveness of the proposed mechanisms for cooperative acoustic and linguistic representations learning. we hope this work will be useful for the community on the way to explore different pre-trained models for low-resource asr. ## preliminaries here we briefly introduce the architectures of acoustic and linguistic encoders in our framework. wav2vec 2.0. we adopt wav2vec 2.0 @xcite as our acoustic encoder because of its effectiveness and efficiency. it has two stages: (i) contrastive pre-training to learn representations of speech and (ii) fine-tuning to adapt the learned representations on labeled data with connectionist temporal classification(ctc) loss @xcite for downstream speech recognition tasks. in this work, we aim to utilize the public pre-trained model and mainly focus on the fine-tuning stage. the architecture of wav2vec 2.0 contains a feature encoder, a context network with a transformer and a quantization module. during fine-tuning, the quantization module is removed and a randomly initialized linear projection layer is attached on top of the context network. bert. bert @xcite is employed as our linguistic encoder since it is one of the most popular text pre-training approaches and has shown remarkable performance in many downstream natural language processing tasks. it also consists of two steps: (i) self-supervised pre-training to learn deep bidirectional linguistic representations from a large text corpus and (ii) fine-tuning to adapt to downstream tasks using labeled data. bert consists of an embedding table, a multi-layer bidirectional transformer encoder, and an additional output layer for fine-tuning. ## experiments in this section, we first illustrate the implementation details of our wav-bert. then we introduce two low-resource speech recognition datasets containing several languages as well as the comparison results among our approach and baseline methods. furthermore, we conduct ablation studies to validate the effectiveness of each main component of our wav-bert and present some case studies for perceptual comparison. implementation details. for our proposed representation aggregation module and embedding attention module, the heads and embedding dimensions of all multi-head attention are set to 8 and 768 respectively. meanwhile, the inner-layer dimension of the position-wise feed-forward is set to 2048. regarding optimization details, we train our model as well as baselines based on wav2vec 2.0 base for 200k steps with one geforce rtx 3090 gpu, setting max tokens and update frequency to 640000 and 4 correspondingly. as for experiments using xlsr-53 @xcite , three geforce rtx 3090 gpus are used with max tokens as 480000 and update frequency as 4. we use the three-stage learning rate policy with the initial learning rate as 5e-5, and set each stage ratio to 0.05, 0.45 and 0.5. besides, we set the weight µ 1 , µ 2 , µ 3 and µ 4 for each loss to 0.5 for training. other optimizer settings are the same as wav2vec 2.0 @xcite . in terms of the "sampling with decay" policy, languages in iarpa ba-bel start from 100k steps to 200k steps, while in aishell-1 it starts from 40k steps to 100k steps, all with p decreasing from 90% to 10%. datasets. iarpa babel @xcite is an open-source multilingual corpus of conversational telephone speech. for low resource evaluation, we randomly select 3 kinds of languages with few data: swahili (sw), tamil (ta) and vietnamese (vi). we adopt the same setup as @xcite and use the dev folder of the babel dataset as our test set since "eval" data are not released. we re-sample audios of all languages to 16khz. aishell-1 @xcite is an open-source and high-quality mandarin speech corpus, and is widely used in the speech community, which contains 178 hours of mandarin speech data. although the data is in chinese, a common used language, the quantity is small. thus, it can also verify our wav-bert for low-resource data. moreover, there are many latest state-of-the-art methods on this dataset to be compared. for a fair comparison, we use the official wav2vec 2.0 (base/large) model, xlsr-53, and mbert models as the initial encoders. all model checkpoint download links are described in the appendix. ## conclusion in this work, based on the powerful wav2vec 2.0 and bert models, we introduce cooperative acoustic and linguistic representation learning for lowresource speech recognition. to solve the representation discrepancy and embedding inconsistency challenges, we design a representation aggregation module and an embedding attention module to facilitate the cooperation of the two pre-trained models and thus boost the representation learning. extensive experimental results demonstrate that our proposed wav-bert can significantly improve low-resource asr performances in different languages. in future work, we will investigate more effective modules to infuse more types of knowledge, and apply our framework to more pre-trained models to promote the development of low-resource speech tasks.