Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Models (2024)

Abstract: Recent advances in large language models (LLMs) have promoted generative error correction (GER) for automatic speech recognition (ASR), which aims to predict the ground-truth transcription from the decoded N-best hypotheses. Thanks to the strong language generation ability of LLMs and rich information in the N-best list, GER shows great effectiveness in enhancing ASR results. However, it still suffers from two limitations: 1) LLMs are unaware of the source speech during GER, which may lead to results that are grammatically correct but violate the source speech content, 2) N-best hypotheses usually only vary in a few tokens, making it redundant to send all of them for GER, which could confuse LLM about which tokens to focus on and thus lead to increased miscorrection. In this paper, we propose ClozeGER, a new paradigm for ASR generative error correction. First, we introduce a multimodal LLM (i.e., SpeechGPT) to receive source speech as extra input to improve the fidelity of correction output. Then, we reformat GER as a cloze test with logits calibration to remove the input information redundancy and simplify GER with clear instructions. Experiments show that ClozeGER achieves a new breakthrough over vanilla GER on 9 popular ASR datasets.

## introduction recent advances in large language models (llms) have attracted a surge of research interest thanks to their remarkable language generation and reasoning ability @xcite @xcite , which achieve a wide range of success on natural language processing (nlp) tasks @xcite @xcite . powered by llms, latest work @xcite proposes a generative error correction @xcite (ger) benchmark 1 for automatic speech @xcite . left: violate source speech, llm removes the word "think" in first two hypotheses as it rarely appears at the beginning of a sentence and followed by a subject according to grammar, but this actually happens in the source speech. right: information redundancy in n-best hypotheses input, there is only one difference between n-best candidates, making it redundant to send all of them for ger, which confuses llm about which tokens to focus on for correction. recognition (asr), and they release a hyporadise dataset foot_0 that contains over 332k pairs of decoded n-best hypotheses and ground-truth transcription in various asr domains. it has shown great effectiveness in learning the mapping from hypotheses to transcription by parameter-efficient llm finetuning @xcite , which significantly enhances the asr result and outperforms typical lm rescoring methods @xcite . however, ger paradigm is also observed to suffer from two limitations. first, llms are unaware of the source speech during ger process, which could lead to results that do not match the source speech content. for example, as shown in fig. 1 (left), the source speech reads the word "think" at the beginning and followed by "he", which is correctly recognized by the 1-best hypothesis. then during the ger process, llm removes the word "think", as this structure of verb plus noun at the beginning of a sentence is not rigorous according to grammar. however, this is not expected as it violates the source speech content. second, we observe that n-best hypotheses usually only vary in a few tokens. for example, as shown in fig. 1 (right), all the tokens in candidates are the same except "enjoys"/"enjoy"/"joins". in this case, it would be information redundant to leverage all of the hypotheses for predicting the ground-truth transcription, which could confuse the llms about which tokens to focus on for correction and thus lead to sub-optimal ger performance. motivated by the above observations, we propose clozeger, a new paradigm for asr generative error correction. first, we introduce a popular multimodal llm, speechgpt @xcite , to receive source speech as an extra input to the ger paradigm. with the powerful cross-modal ability of speechgpt, we can now constrain ger to comply with the source speech while correcting the errors in decoded hypotheses. then, in order to remove the input information redundancy, we reformat it as a cloze test (i.e., a special multiplechoice question) with logits calibration @xcite , where the identical parts across n-best hypotheses are set as the context and the varying parts are set as blanks (each with several options provided). with such clear instructions for error correction, it would be easier for llms to perform context reasoning and choose the right answer for each blank rather than predicting the entire sentence from redundant n-best inputs 3 . finally, we add a simple post-processing stage to correct the errors in cloze context (i.e., identical parts across n-best list) to further improve the correction result. our contributions are summarized as follows: ## related work large language models. there is recently a surge of research interests in transformer-based llms, such as chatgpt (openai, 2022), gpt-4 (openai, 2023) and llama @xcite . benefiting from the huge model size and abundant training data, llms can well understand the linguistic structures and semantic meanings behind textual data, which shows remarkable performance on a wide range of nlp tasks @xcite @xcite . more recently, researchers have started to explore the potential of llms on multimodal tasks by incorporating other modalities into llms @xcite @xcite @xcite . among them, speechgpt @xcite ) is one of the most popular multimodal llms that represent speech and text using a unified tokenizer, which enables us to add source speech into the original n-best hypotheses input of the ger paradigm. ## methodology in this section, we present our proposed clozeger paradigm in detail. we first introduce the preliminary knowledge of ger in §3.1, and then we investigate to introduce source speech to ger paradigm with multimodal llm ( §3.2). finally, we present the new task format of clozeger in §3.3. ## conclusion in this paper, we propose clozeger, a new paradigm for asr generative error correction. first, we introduce a multimodal llm (i.e., speechgpt) to receive source speech as extra input to improve the fidelity of correction output. then, we reformat ger as a cloze test with logits calibration to remove the input information redundancy and simplify ger with clear instructions. experimental evidence shows that clozeger achieves a new breakthrough over vanilla ger on 9 popular asr datasets. further analysis verifies the effectiveness of different modules in our framework.