F ast C orrect 2: Fast Error Correction on Multiple Candidates for Automatic Speech Recognition (2021)

Abstract: Error correction is widely used in automatic speech recognition (ASR) to post-process the generated sentence, and can further reduce the word error rate (WER). Although multiple candidates are generated by an ASR system through beam search, current error correction approaches can only correct one sentence at a time, failing to leverage the voting effect from multiple candidates to better detect and correct error tokens. In this work, we propose FastCorrect 2, an error correction model that takes multiple ASR candidates as input for better correction accuracy. FastCorrect 2 adopts non-autoregressive generation for fast inference, which consists of an encoder that processes multiple source sentences and a decoder that generates the target sentence in parallel from the adjusted source sentence, where the adjustment is based on the predicted duration of each source token. However, there are some issues when handling multiple source sentences. First, it is non-trivial to leverage the voting effect from multiple source sentences since they usually vary in length. Thus, we propose a novel alignment algorithm to maximize the degree of token alignment among multiple sentences in terms of token and pronunciation similarity. Second, the decoder can only take one adjusted source sentence as input, while there are multiple source sentences. Thus, we develop a candidate predictor to detect the most suitable candidate for the decoder. Experiments on our inhouse dataset and AISHELL-1 show that FastCorrect 2 can further reduce the WER over the previous correction model with single candidate by 3.2% and 2.6%, demonstrating the effectiveness of leveraging multiple candidates in ASR error correction. FastCorrect 2 achieves better performance than the cascaded re-scoring and correction pipeline and can serve as a unified post-processing module for ASR.

## introduction error correction has been applied in automatic speech recognition (asr), which post-processes the outputs of the asr system to achieve lower word error rate (wer) @xcite @xcite . taking the recognized sentence from the asr system as source and the ground-truth sentence as target, asr correction can be formulated as a sequence-to-sequence problem and modeled with autoregressive @xcite or non-autoregressive @xcite generation. the key challenge in asr error correction is to detect and correct the error tokens. current approaches only correct one sentence at a time, which might be sub-optimal since the correction model can only guess the error token based on the context information of a single sentence. considering that beam search is commonly used in asr, multiple candidates are usually generated and available for error correction. we argue that multiple candidates contain the voting effect, which refers to that the tokens from multiple sentences can verify the correctness with each other. for example, if the beam search candidates with 3 sentences are "i have cat", "i have hat", "i have bat", then the first two tokens are likely to be correct since they are the same among all beam candidates. the inconsistency on the last token shows that: 1) this token may need correction, and 2) the pronunciation of the ground-truth token may end with "aet". this voting effect can be utilized to boost the asr correction by helping the model detect error token and giving some clues about the pronunciation of ground-truth token. in this paper, we propose fastcorrect 2, an error correction model that takes multiple asr candidates as input to leverage this voting effect for better correction accuracy. in order to satisfy the latency constraint for industrial deployment, we leverage non-autoregressive (nar) generation @xcite @xcite @xcite for fast inference. fastcorrect 2 consists of an encoder that processes multiple source sentences, a duration predictor to predict the number of target tokens corresponding to each source token, and a decoder that generates the target sentence in parallel from the adjusted source sentence, where the adjustment is based on the predicted duration of each source token. we describe the challenges when supporting multiple candidates and introduce our corresponding designs to address these challenges as follows. we conduct experiments on internal asr datasets and public aishell-1 datasets to verify the effectiveness of fastcorrect 2. experiment results show that our method 1) achieves better correction accuracy over previous nonautoregressive correction model on single sentence, and 2) achieves better performance than the conventional cascaded re-scoring and correction pipeline and can serve as a unified post-processing module for asr. the contributions of fastcorrect 2 are summarized as follows: ## background in this section, we briefly review the two postprocessing methods for asr: error correction and re-scoring. ## fastcorrect 2 in this section, we introduce fastcorrect 2 in detail. in order to leverage the voting effect in multiple candidates, we first propose an alignment algorithm to align the length-variant candidates into the same length while maximizing the token and pronunciation similarity on each alignment position. then we introduce the detailed architecture of our model, consisting of an encoder with a pre-net to handle the multiple candidates input, a duration predictor to predict the number of target tokens corresponding to each source token in each candidate, and a candidate predictor to find out the easiest candidate to correct, which is then adjusted according to the predicted duration and used as the decoder input for parallel generation. we introduce the alignment algorithm and the correction model in the following subsections. ## experiment settings in this section, we show the datasets, asr models, settings of fastcorrect 2, as well as the baseline systems used in experiments. ## results in this section, we first report the accuracy and latency of fastcorrect 2 for asr error correction, and then perform ablation study to verify the effectiveness of each module in fastcorrect 2. ## conclusion in this work, we propose fastcorrect 2 to leverage multiple candidates for asr error correction, where the candidates are generated by the asr model through beam search and contain voting effect to help better detect and correct error tokens. in order to leverage this voting effect from multiple candidates, we propose a novel alignment algorithm to align the multiple candidates while maximizing the token matching score and pronunciation similarity score on each alignment position. accordingly, we make several modifications on a previous non-autoregressive error correction model to make it suitable for the input with multiple candidates, including an encoder with a pre-net to handle the concatenation of multiple candidates, a duration predictor to predict the token duration of each candidate, and a candidate predictor to choose the easiest candidate as the decoder input for correction. experiment results show that fastcorrect 2 improves the correction accuracy over previous non-autoregressive correction model with single candidate, demonstrating the effectiveness of multiple candidates for error correction. besides, fast-correct 2 can be used as a unified post-processing module for asr, achieving comparable or better accuracy with the cascaded post-processing pipeline (correction and re-scoring) while speeding up for 3 times.