Masked Audio Text Encoders are Effective Multi-Modal Rescorers (2023)

Abstract: Masked Language Models (MLMs) have proven to be effective for second-pass rescoring in Automatic Speech Recognition (ASR) systems. In this work, we propose Masked Audio Text Encoder (MATE), a multi-modal masked language model rescorer which incorporates acoustic representations into the input space of MLM. We adopt contrastive learning for effectively aligning the modalities by learning shared representations. We show that using a multi-modal rescorer is beneficial for domain generalization of the ASR system when target domain data is unavailable. MATE reduces word error rate (WER) by 4%-16% on in-domain, and 3%-7% on out-of-domain datasets, over the text-only baseline. Additionally, with very limited amount of training data (0.8 hours) MATE achieves a WER reduction of 8%-23% over the first-pass baseline.

## introduction performance of automatic speech recognition (asr) systems has been traditionally improved during inference time via either editing/refinement @xcite @xcite or second-pass rescoring @xcite @xcite using language models . in recent studies, transformerbased pre-trained large language models (llms) have shown promising results when used as secondpass rescorers. previous works @xcite @xcite have shown that deep bidirectional transformers @xcite perform better than their unidirectional counterparts such as gpt-2 @xcite . while llms are trained on giant text corpora, they may not be representative of the specific domain of interest, in this case, speech transcriptions. this may result in limited generalization ability without domain-specific fine-tuning. further, asr applications warrant robustness to noise and other distortions, which text-only llms are incapable of handling on their own at rescoring time. a potential solution to mitigate these limitations is to incorporate the speech input into llm rescorers. recent studies have demonstrated the effectiveness of leveraging audio information during second-pass rescoring @xcite @xcite to improve performance. however, a tight integration of rescorer, attending to a shared speech encoder used in the first-pass, relies on asr architecture, training mechanism and internal features, limiting the flexibility of being applied to other asr systems. inspired by recent multi-modal llm works @xcite @xcite , we propose mate, a multi-modal mlm rescorer, which is compatible with encapsulated asr systems: our method by design can work with any first-pass asr models (hybrid / ctc / transducer). the rescorer is agnostic to asr architecture, training mechanism and internal features, leading to better generalization capability. to the best of our knowledge, this is the first work to integrate a pre-trained self-supervised learning (ssl) speech representation model @xcite @xcite into the second-pass rescoring. one key challenge of incorporating acoustic information into llms is to transform the speech into a form that can be accepted by the language model. we overcome this by using a cross-modal adaptation module consisting of convolutional neural network (cnn) @xcite and adapter network @xcite . we experiment with different auxiliary alignment losses for audio-text alignment, to effectively learn shared representations across the two modalities, and adopt contrastive learning which significantly improves the model performance. empirically, we show that mate transfers well to new domains in zero-shot and few-shot settings, outperforming text-only baselines. (2)l ctr to align the audio and text latent representations. ## approach mate consists of a pre-trained masked language model bert, an self-supervised learning (ssl) based speech encoder wavlm @xcite and a modality matching module (cnn and adapter network), as illustrated in figure 1 . ## results and analysis we summarize the observations and analysis of the results from our experiments foot_3 as follows: mate excels at both in-domain and out-ofdomain generalization: parameter-efficient fine-tuning results in limited gains: rows 9-11 study the performance of a multi-modal rescorer under different parameter efficient fine-tuning settings. we observe that performance degrades as we move from full finetuning to adapter-tuning and freezing the full bert encoder layers, indicating that fine-tuning bert encoder is the most beneficial in terms of performance improvement. as expected, in comparison to model with full fine-tuning (row 6), rows 9-11 exhibit lower performance. this suggests that frozen or parameter-efficient training methods may lack the model capacity to fully leverage the acoustic information present in the multi-modal data. ## conclusions we propose a novel multi-modal rescorer, mate, which achieves significant wer, cwer reduction on in-domain and ood datasets. in zero-shot and few-shot settings, mate performs well on unseen domains and adapts rapidly with limited data. the domain generalization capability of mate makes it an effective choice as a second-pass rescorer for scaling asr systems to new domains.