Whispering LL a MA : A Cross-Modal Generative Error Correction Framework for Speech Recognition (2023)

Abstract: We introduce a new cross-modal fusion technique designed for generative error correction in automatic speech recognition (ASR). Our methodology leverages both acoustic information and external linguistic representations to generate accurate speech transcription contexts. This marks a step towards a fresh paradigm in generative error correction within the realm of n-best hypotheses. Unlike the existing ranking-based rescoring methods, our approach adeptly uses distinct initialization techniques and parameter-efficient algorithms to boost ASR performance derived from pre-trained speech and text models. Through evaluation across diverse ASR datasets, we assess our fusion technique, demonstrating a 37.66% improvement in word error rate (WER) relative performance compared to the n-best Oracle. To encourage future research, we have made our code and pre-trained models open source at [https://github.com/Srijith-rkr/Whispering-LLaMA](https://github.com/Srijith-rkr/Whispering-LLaMA)

## introduction end-to-end (e2e) trained speech models have demonstrated state-of-the-art performance on automatic speech recognition (asr) tasks. several methods @xcite @xcite @xcite have widely adopted a two-pass rescoring paradigm to leverage upon language models to further enhance the capabilities of these models. in the two-pass paradigm, the first pass asr system "generates" n-best hypotheses using an e2e acoustic model, while the second pass "re-ranks" these hypotheses by incorporating a language model (lm). this two-pass reranking approach has several notable advantages over single-pass end-to-end (e2e) asr systems @xcite . firstly, the subsequent large language model often captures a more comprehensive understanding @xcite of language structures beyond the knowledge of transcribed audio present in the asr model's pre-training data, thereby improving performance on unseen words. furthermore, adapting the twopass paradigm to accommodate domain shifts @xcite @xcite is much easier as only the language model needs to be fine-tuned on the new dataset. this alleviates the need for a spoken transcription corpus, which can be particularly beneficial for under-resourced or endangered spoken languages. the recent emergence of conversational abilities in large language models, such as chatgpt (ope-nai, 2023a) and @xcite , has further sparked interest in leveraging the representational power of large pre-trained models for more complex tasks involving diverse data modalities @xcite . moreover, this new research direction also introduces a set of unique challenges related to considering information from other input modalities, such as acoustic and visual conditions @xcite , in which could enrich using context beyond text-only input. recognizing speech signals is a task that necessitates both acoustic information @xcite (e.g., speaking environments) and linguistic information @xcite ) (e.g., context and domains). efficiently amalgamating or integrating representation learning from acoustic modeling into language modelling to bolster its performance represents a notably intricate research domain that warrants further exploration. in this paper, we present a tokenlevel fusion framework, merging two foundation (large-scale pre-trained) models into a recognition error correction paradigm, with the objective of enhancing the performance of asr systems. transformer-based language models @xcite approach the two-pass paradigm by utilizing the summation of negative log-likelihoods of individual tokens from the language model to re-score the n-best output. recent works on deliberation method @xcite and audio-attention based rescoring @xcite @xcite in improving asr-lm rescoring with the incorporation of acoustic features. recent works on decoder prompting (yang et al., 2023a) and encoder-decoder based error correction @xcite have demonstrated benefits in using an external language model for reducing the transcription error rate. meanwhile, how to inject or fuse representations from a large acoustic model into another language model remains under investigation. ## method we discuss the model architecture and the intuition behind the proposed feature combination in section 3.1. the cross-modal fusion mechanism and weight initialization are explained in section 3.2 and section 3.3, respectively. ## conclusion we propose a novel framework to leverage the external knowledge from llm to improve the transcription accuracy of asr systems. our framework presents a parameter-efficient way to integrate large foundational speech and language models to achieve competitive werr improvements. we further conduct extensive ablation experiments to validate our intuitions and open source our code and pretrained-weights to the research community.