Non-Autoregressive C hinese ASR Error Correction with Phonological Training (2022)

Abstract: Automatic Speech Recognition (ASR) is an efficient and widely used input method that transcribes speech signals into text. As the errors introduced by ASR systems will impair the performance of downstream tasks, we introduce a post-processing error correction method, PhVEC, to correct errors in text space. For the errors in ASR result, existing works mainly focus on fixed-length corrections, modifying each wrong token to a correct one (one-to-one correction), but rarely consider the variable-length correction (one-to-many or many-to-one correction). In this paper, we propose an efficient non-autoregressive (NAR) method for Chinese ASR error correction for both cases. Instead of conventionally predicting the sentence length in NAR methods, we propose a novel approach that uses phonological tokens to extend the source sentence for variable-length correction, enabling our model to generate phonetically similar corrections. Experimental results on datasets of different domains show that our method achieves significant improvement in word error rate reduction and speeds up the inference by 6.2 times compared with the autoregressive model.

## introduction errors introduced by automatic speech recognition (asr) usually affect the performance of downstream tasks such as phonetic search, speech translation, etc. in recent years, asr error correction techniques have been proposed @xcite @xcite to refine the asr output and correct errors in text space. without loss of generality, we study chinese asr error correction in this paper. given the asr result of an utterance, the goal of error correction is to generate a sentence with the wrongly recognized words corrected. thus the asr error correction can be modeled as a machine translation problem under conventional autoregressive sequence-to-sequence (seq2seq) framework @xcite @xcite . however, the autoregressive models suffer from inefficient decoding since the generation of each target token depends on previously generated characters (figure 1 (a) ). furthermore, without considering the phonetic similarities, the method is prone to generate corrections with totally different pronunciation, as for the example, the error character ("表", biao) should be corrected into ("不 要", bu yao) but the seq2seq model ignores phonological features and corrects it to a phonetically different correction ("不许", bu xu). to speed up prediction, recent studies propose to take non-autoregressive (nar) methods @xcite for error correction, which generates target tokens in parallel. most nar approaches make fixed-length predictions that generate same-length output as the source input by directly tagging on the source text @xcite . some works further leverage phonological features to correct the asr errors caused by similar pronounced characters @xcite . such methods have made great improvements in correcting simple errors, but cannot handle samples with different lengths of source and target, referred to as variable-length error correction. see figure 1 (b) for illustration. the method successfully finds out the erroneous character ("表", biao), and substitutes it with its phonological feature "biao" for error correction. however, it eventually generates a wrong correction ("标", biao) under the constraint of fixed length prediction. to overcome above constraint while preserving efficient prediction, nar solutions for variablelength prediction are proposed @xcite . they first build a length predictor to estimate fertility, i.e., the number of target tokens tagging with phonetics 𝑦 1 𝑦 2 𝑦 3 𝑦 4 你 标 难 过 𝑥 1 ′ 𝑥 2 ′ 𝑥 3 ′ 𝑥 4 ′ 你 biao 难 过 (c). nar with length predictor encoder 𝑦 1 𝑦 2 𝑦 3 𝑦 4 宝 宝 难 过 𝑥 1 ′ 𝑥 2 ′ 𝑥 3 ′ 𝑥 4 ′ 表 表 难 过 length predictor 0 2 1 1 decoder error corrector (d). phonology based nar (our proposed) error detector 𝑥 1 ′ 𝑥 2 ′ 𝑥 3 ′ 𝑥 4 ′ 𝑥 5 ′ 𝑥 6 ′ 𝑥 7 ′ 𝑥 8 ′ aligned to each source token, then up-sampling or dropping some source tokens accordingly for parallel correction. however, the method is fragile because incorrect length prediction will distort the meaning of the sentence and may directly lead to false prediction. see figure 1 (c) for example, the length predictor produces a wrong alignment (0 2 1 1), indicating that the decoder will drop the first source character and generate two target characters for the second source character. according to the alignment, the intermediate result turns to be a meaningless sentence ("表表难过", means "watch watch sad") and leads to a wrong correction. to address the issues mentioned above, we propose a novel nar model named phvec (phonology-based variable-length error correction). the model incorporates the phonological features of error characters to enable variable-length prediction. concretely, phvec contains a detection network and a correction network based on a pre-trained language model @xcite . the detection network predicts the correctness of each token, and the correction network generates correction result. instead of using a length predictor to predict the fertility for each source token, we insert the phonological tokens (chinese pinyin foot_0 ) to the source sequence as placeholders after each detected erroneous character. during prediction, each source token (either the character token or pinyin token) can generate zero or one target token. with the guidance of the phonological token, the model will generate characters with similar pronunciations. as figure 1 (d ) shows, the pinyin token "b" produces target character ("不", bu) with the same consonant letter, and "a o" produces target character ("要", yao) with the same vowel letter. we delete the repeated characters in the final sequence. we evaluate our methods on multiple datasets with varying degrees of asr word error rates. experimental results show that our phvec obtains significant improvement (over 10% word error rate reduction) on standard benchmarks compared with existing nar-based asr error correction methods at comparable speed. even compared with the ar baseline transformer and bart @xcite ) models, phvec can still have an 8.55% and 2.39% word error rate reduction while keeping a 6.2x speed-up. ## method chinese asr error correction can be formalized as the following task. given a speech recognition sequence @xmath0, x 2 , ..., x n ) of n chinese characters, the goal is to correct it into another sequence of m characters @xmath1, y 2 , ..., y m ). note that the target sequence length m does not have to be equal to the source sequence length n. there exist three types of asr errors in transforming from x to y : substitution, deletion, and insertion. as illustrated in figure 1(d ), our proposed asr error correction model is composed of an error detector and an error corrector, both perform a nar tagging. the error detector takes x as input to predict the correctness of each token. the error corrector takes the combination of the source tokens and the phonological features of problematic tokens as input for correction. during error correction, each token of the extended input can be tagged to a chinese character or a blank token " ", so that the model can support variable-length correction. since different pinyin letters may generate the same character, we eliminate continuous repetitive characters in the final result. in this paper, we use chinese pinyin as the phonological feature. the learning of phvec is conducted end-to-end, with the error detector and corrector optimized jointly. ## training data generation we follow the common practice in error correction @xcite @xcite to synthesize the training corpus with simulated asr errors. the simulated asr results are generated by replacing the chinese characters or words of clean sentences into problematic ones. specifically, we generate the noisy text in three steps: (i) sampling some candidate characters and replace them with pinyin. (ii) adding noise to the original pinyin and generate new valid ones (iii) producing new chinese characters or words based on the updated pinyin. in the first step, the candidate words are obtained from a confusion set @xcite that contains words prone to be mis-recognized. in the second step, we design three strategies of pinyin perturbation for simulating substitution errors, deletion errors and insertion errors, as shown in figure 3 . note that when generating deletion and insertion errors, some random letters may be inserted to construct valid pinyin sequence. in the last step, we replace the noisy pinyin with corresponding word candidates, and select the sentence ranked highest by a n-gram language model as the final simulated asr result. ## experiments we carry out experiments on two chinese asr error correction datasets. we use word error rate (wer) and word error reduction rate (werr) to evaluate our error correction performance. ## conclusion in this paper, we propose a non-autoregressive chinese asr error correction network with phonolog-ical training. our method first detects the problematic characters, then adds the phonological features of them to adjust the input length, thus generating a variable-length sequence for error correction. the phonological features enable our model to produce similar-pronounced corrections, and support variable-length correction in a non-autoregressive mode. experiments show that our method is superior to the autoregressive method while maintaining a 6.2x speed-up. as a future work, we plan to extend phvec to other languages and use corresponding phonological tokens to correct the variable length errors caused by pronunciation. yun zhao, xuerui yang, jinchao wang, yongyu gao, chao yan, and yuanfu zhou. 2021. bart based semantic correction for mandarin automatic speech recognition system. corr, abs/2104.05507.