Robust ASR Error Correction with Conservative Data Filtering (2024)

Abstract: Error correction (EC) based on large language models is an emerging technology to enhance the performance of automatic speech recognition (ASR) systems.Generally, training data for EC are collected by automatically pairing a large set of ASR hypotheses (as sources) and their gold references (as targets).However, the quality of such pairs is not guaranteed, and we observed various types of noise which can make the EC models brittle, e.g. inducing overcorrection in out-of-domain (OOD) settings.In this work, we propose two fundamental criteria that EC training data should satisfy: namely, EC targets should (1) improve linguistic acceptability over sources and (2) be inferable from the available context (e.g. source phonemes).Through these criteria, we identify low-quality EC pairs and train the models not to make any correction in such cases, the process we refer to as conservative data filtering.In our experiments, we focus on Japanese ASR using a strong Conformer-CTC as the baseline and finetune Japanese LLMs for EC.Through our evaluation on a suite of 21 internal benchmarks, we demonstrate that our approach can significantly reduce overcorrection and improve both the accuracy and quality of ASR results in the challenging OOD settings.

## introduction automatic speech recognition (asr) is the task of transcribing human speech into readable text, which is of practical use in various applications. in contrast to the traditional hybrid approach @xcite , modern asr systems are trained in an end-to-end manner using a large parallel corpus of acoustic speech paired with gold transcriptions @xcite . despite their huge success, end-to-end asr systems have limited linguistic knowledge due to the difficulty of leveraging unpaired text-only data which exist in abundance @xcite . error correction (ec) is an effective strategy to correct linguistic errors produced by such asr systems @xcite . recently, large language models (llms) pretrained on massive text-only data have shown promising results for this purpose @xcite . while several works explore the zeroshot or in-context learning capability of llms @xcite , finetuning llms with sufficient ec training data remains critical to impart the knowledge of asr-specific error patterns and desired corrections @xcite @xcite @xcite generally, training data for ec are collected by automatically pairing the asr hypothesis (source) and its gold transcription (target), and the task is formulated as sequence transduction from the source to target @xcite . however, the quality of such pairs is not guaranteed: in fact, we observed various types of noise which require incorrect, unnecessary, or uninferable corrections that are unreasonable to be predicted from the source. we show some illustrative examples in training ec models on such noisy data can amplify overcorrection, which is a typical problem in current ec @xcite . however, existing works largely overlook the existence of such noise and apply minimal data filtering, e.g. simply discard pairs with large edit distance @xcite . in this study, we propose two fundamental criteria that ec training data should satisfy in general. specifically, we ensure that ec targets c1. improve linguistic acceptability over sources c2. are inferable from the available context (e.g. source phonemes) based on these criteria, we identify low-quality ec pairs and train the models to avoid making any correction on them. such conservative behavior is asr hypothesis (source w s ) gold reference (target w t ) clean に雑音を蒸したもの に雑音を付加したもの [ni zatsuon o fuka shita mono] [ni zatsuon o fuka shita mono] (to which noise is steamed) (to which noise is added) noisy incorrect/ unnecessary しかし一対一の場合ですと しかし一対一場合ですと [shikashi ittaiichi no baai desuto] [shikashi ittaiichi baai desuto] (but in case of one-to-one) (but in case one-to-one) uninferable 男の人はぐらいですかね 男の人の方がいいですかね [otokonohito wa gurai desukane] [otokonohito noho:ga ii desukane] (would a male person be about) (would a male person be better) often crucial to reduce overcorrection and improve robustness, esp. in the out-of-domain (ood) settings @xcite . the overall flow of our data filtering strategy is shown in figure 1 . in our experiments, we focus on japanese asr using an internal conformer-ctc as the baseline @xcite . for ec, we finetune opensource japanese llms, namely swallow-mistral 7b foot_0 and sarashina-2 7b foot_1 , and evaluate the performance on 21 internal benchmarks comprised of various domains. through our experiments, we confirm that our approach can significantly reduce overcorrection and robustly improve asr results in the most challenging ood settings. ## related work in the existing literature, ec primarily focuses on the in-domain setup where models are trained and evaluated over the same domain @xcite @xcite @xcite . recently, @xcite proposed a low-resource ood setup where ec models are finetuned on a limited amount of target domain data to generalize beyond in-domain data. however, target domains of ec are conceptually broad or even open-ended, so it is desirable that ec models work reliably in any target domain without prior knowledge or finetuning. in this study, we focus on the most challenging zero-resource ood setup to develop general-purpose ec models which work out of the box in a variety of domains. despite the recent progress, overcorrection remains a major challenge in ec, esp. in the ood setup. to alleviate this issue, constrained decoding @xcite restricts or biases the correction towards retaining the original asr hypotheses. @xcite use a representative data source and partially train the models to copy the input to induce conservative behavior. in complementary to their approach, we focus on the quality of ec data and apply sophisticated data filtering, which is a novel aspect of our approach that works much more effectively than existing filtering based on edit distance @xcite @xcite . typically, asr errors originate from confusing phonetically similar words and phrases. therefore, supplementing ec models with phonetic/acoustic information can help improve their performance @xcite @xcite . in this study, we use the source phonemes as an additional input, which can be easily handled by the text-based llms. when available, the full n-best hypotheses can be used as input to provide richer clues on where the asr systems are confused @xcite . however, for both simplicity and computational efficiency, we only use the 1-best hypothesis (i.e. top asr prediction) in our experiments. ## methods ec can be formulated as a sequence transduction task from the asr hypothesis (source) to the gold reference (target). formally, let (w s , w t ) denote the source and target sequence pair. in the simplest setting, the ec model is trained to estimate p ec (w t | w s ) with the expectation of transforming an error-prone source into a clean target. s as an additional input, in which case the ec model estimates p ec (w t | w s , w s ). source phonemes are obtained from our asr system ( §4) and represented in hiragana, one of the japanese syllabaries, which can be easily consumed by japanese llms. below is an example: criteria 1: ec targets should improve linguistic acceptability over sources. the main objective of ec is to resolve linguistic errors in the asr system's predictions and improve linguistic acceptability. while the gold reference usually contains cleaner text, this is not always the case, e.g. due to speaker disfluency in spontaneous speech or noisy transcriptions. in addition, japanese is a language with rich orthographic variation where multiple valid spellings exist @xcite . for instance, the correction is not necessary if the source transcribes a bottle as 瓶 [bin] while the target transcribes as ビン [bin], since both spellings are equally acceptable. to improve robustness, ec models should only focus on apparent mistakes and resolve them accurately. one simple way to express this criteria is based on the following equation: here, p(w s ) and p(w t ) denote the likelihoods of the source and target, which can be computed using any language model. in this study, we simply use the base japanese llm. c 1 denotes the threshold, set to 1 by default, which can control the strength of the filter. intuitively, (w s , w t ) that do not satisfy this criteria do not sufficiently improve the linguistic acceptability, indicating the correction is incorrect or unnecessary. criteria 2: ec targets should be inferable from the available context. existing works assume that ec targets are generally inferable from the source. however, this is not always the case: in fact, expert evaluation revealed that about one-third of the errors cannot be corrected from the source alone @xcite . this is mainly attributed to the large phonetic discrepancy between the source and target, e.g. caused by environmental noise or incapability of the asr system. a robust ec model should only make the correction when it is inferable from the available context. to express this criteria, we quantify the degree of inferability from the source phonemes using the following equation: here, p ec is a baseline ec model trained only using the source phonemes as input. in this study, we finetune the base japanese llm following the procedure described in §4. again, c 2 denotes the threshold which can be set to 1 by default. intuitively, (w s , w t ) that do not satisfy this criteria cannot be easily inferred from the available context, namely source phonemes in our case. it is worth noting that edit distance is not a suitable measure of inferability. for instance, the uninferable example in based on the above criteria (c1 and c2), we identify low-quality ec pairs and train the models to avoid making any correction on them by simply replacing the target with source (w @xmath0: see figure 1 for an illustration. we found this approach more effective than discarding the noisy pairs, since the model is explicitly trained to be conservative on noisy or otherwise ambiguous examples. note that both criteria are defined based on the likelihood ratio between the source and target (eq. 1, 2). in figure 2 , we plot the distribution of the (log-)likelihood ratio for each criteria in our training data, using swallow-mistral 7b as the llm. 4 we can verify that a non-negligible portion of the pairs do not satisfy the criteria, suggesting noisy pairs are prevalent in ec training data. for additional examples of the filtered/non-filtered pairs, we refer the reader to appendix a. out of the whole training data, our asr baseline predicts the exact gold reference (i.e. w @xmath1. therefore, the ec model effectively learns to make a correction (i.e. w s ̸ = w t ) in only 66% of the cases. of these effective pairs, 34% are classified as noisy based on our c1 filter, 33% based on c2 filter, and 42% when combined. while this results in even fewer examples to learn from, we can expect the model to focus on clearer errors and improve ood robustness. our approach is also in line with the principle that data quality can be more important than quantity for llm alignment @xcite . ## experimental setup asr system for the asr baseline, we use an internal conformer-ctc developed for commercial use cases. the acoustic model is a ctc @xcite with 240-dimensional logmel-derived features every 40 milliseconds as input, consisting of 10 conformer layers @xcite , followed by an output layer of 42 japanese phonemes including the blank symbol. for inference, a static graph for graph decoding is created using a word n-gram model and a dictionary representing the mapping between words (w s ) and their phoneme sequences (w s ). in total, our training data consists of 8000 hours of transcribed speech with little or no overlap between our benchmarking domains. ec model for ec, we finetune two japanese llms, namely swallow-mistral 7b @xcite and a more recent sarashina-2 7b, on a subset of the asr training data ensuring a 1:1 mixture of read and spontaneous speech. for finetuning, we use lora @xcite with rank @xmath2. the effective batch size is set to 128 on a single a100 gpu, and the learning rate is 5e-4 annealed with a cosine scheduler. all ec models are trained for a total of 1000 steps, since we observed more steps led to overfitting in the ood setup. for inference, we use greedy decoding, which we found to be efficient yet effective. for c2 filtering, we train the phoneme-based ec model to predict the target w t only using the source phonemes w s as input. otherwise, models are trained with both the source phonemes w s and the source hypothesis w s as input. as an ablation study, we compare the performance of ec models without any data filtering (no filter), with c1 and c2 filtering applied independently (c1/c2 only), and with both filtering applied in combination (c1+c2). in addition, to confirm that noisy pairs are less effective for ec training, we also experiment with an inverse filtering of c1+c2, considering the noisy pairs as clean and vice versa (inv. c1+c2). evaluation we evaluate ec performance on 21 internal benchmarks comprised of various domains. details of each benchmark are provided in as for the evaluation metrics, we primarily focus on character error rate (cer ↓ ) which is standardly used for asr. to quantify the degree of overcorrection, we also measure the percentage of source hypotheses altered after ec (%ec ↓ ). fi-nally, we measure the percentage of hypotheses where the linguistic acceptability is improved after ec (%la ↑ ). to measure %la, we compare the masked language modeling score @xcite of the hypothesis before and after ec using japanese deberta v2 large foot_4 . while deberta is relatively small compared to recent llms, it can take into account the full (bidirectional) context of the hypothesis and effectively assess its linguistic acceptability @xcite . ## results and discussion in focusing on swallow-mistral 7b, there is no single approach which outperforms all others due to the diversity of the test sets. however, we can still draw several conclusions from the overall metrics. first, compared to the original asr results, we can verify that ec without data filtering drastically worsens cer on average (11.84 → 12.51). this is mainly attributed to the overcorrection problem, as we can see a large portion of the hypotheses (43.0% on average) are altered by ec. such aggressive behavior can be helpful in some occasions (e.g. test 13) but generally too risky in the ood setup, leading to modest or even severe performance degradation (e.g. in test 1, 4, 16, 18, to count a few). in contrast, by applying our c1 filtering, we can substantially alleviate the degradation of cer (11.84 → 11.86) by almost halving the frequency of corrections (22.1% on average). this shows that ec can be kept more accurate and conservative by training on cleaner pairs which improve linguistic acceptability. our c2 filtering also has a similar benefit and makes the ec model more robust in the ood setup, outperforming the original asr results in 57.1% (12/21) of the test cases. in addition, by combining our c1+c2 filtering, we can further cut down the frequency of corrections to 13.9% on average. through this conservative behavior, we could significantly improve the ood robustness of ec and reduce the original cer in 71.4% (15/21) of the test sets. this result demonstrates that both c1 and c2 filters help ec focus on clear and fixable asr errors whilst ignoring more controversial ones. to verify that clean (rather than noisy) portions of the data contribute to this improvement, we also experimented with the inverse filtering of c1+c2. generally, we confirmed that inverse filtering worsens cer on average (11.84 → 12.13) and only improves upon the original asr in 28.6% (6/21) of the test sets. therefore, noisy pairs are much less effective for accurate ec. while the frequency of correction is drastically suppressed (8.2% on average), this is largely attributed to the difficulty of learning from noisy examples and overlooking clear errors. in a few cases (e.g. test 5), we found inverse filtering to be quite competitive, which suggests that noisy pairs still include useful examples for some domains. we expect that our filtering can be improved for such domains by appropriately tuning the thresholds (e.g. lowering c 1 and c 2 ) to include useful pairs of borderline quality. finally, in terms of the linguistic acceptability (%la), we generally see improvement through ec: this indicates that ec is at least successful in resolving linguistic errors and improving asr quality, even if by deviating from the ground truth @xcite . naturally, our c1 filtering consistently strengthens this desirable property by explicitly taking this criteria into account (eq. 1). as an additional experiment, we also evaluated the results of ec with data filtering based on maximum edit distance @xcite @xcite . in this approach, ec pairs with normalized edit distance above a certain threshold are simply discarded from the training data. 6 we chose the commonly used thresholds of 0.5 and 0.25, which discard 1% and 5% of the whole training data, respectively. the results are shown in finally, to verify that our claims hold for a different japanese llm, we also experimented using sarashina-2 7b. as discussed in appendix c, we can draw similar conclusions with even better performance, achieving an average cer of 11.41 in the best case and outperforming the original asr results in 85.7% (18/21) of the test sets. therefore, our approach is generalizable using other llms and we can expect to further improve performance by leveraging more powerful llms. ## conclusion ec is an emerging technology to boost the performance of asr by harnessing the power of llms. however, current ec remains brittle, often degrading performance due to overcorrection in the ood setup, which hinders its practical application. in this study, we first focused on the quality of ec training data and proposed a method to identify noisy data based on two fundamental criteria. second, we revealed that ec data contains a considerable proportion of such noisy pairs, which can be effectively handled through our conservative data filtering. finally, we demonstrated that our approach can significantly alleviate the overcorrection problem and improve the robustness of ec in the challenging zero-resource ood setup. in contrast to the existing filtering methods (e.g. based on edit distance), we expect the quality of our data filtering to keep improving as the underlying llms become more powerful and accurate, which is a notable trend in the current literature. in future work, we also plan to control for other important factors of data quality, such as diversity and representativeness @xcite , to further improve the robustness of ec. overall, we expect our approach to be a foundational step towards developing general-purpose ec models applicable in any domain of interest, facilitating the utilization of llm technology in the real-world scenarios. mike lewis, luke zettlemoyer, and omer levy. 2023. lima: less is more for alignment. in thirty-seventh conference on neural information processing systems. linchen zhu, wenjie liu, linquan liu, and edward lin. 2021. improving asr error correction using nbest hypotheses. in 2021 ieee automatic speech recognition and understanding workshop (asru), pages 83-89. ieee.