RED - ACE : Robust Error Detection for ASR using Confidence Embeddings (2022)

Abstract: ASR Error Detection (AED) models aim to post-process the output of Automatic Speech Recognition (ASR) systems, in order to detect transcription errors. Modern approaches usually use text-based input, comprised solely of the ASR transcription hypothesis, disregarding additional signals from the ASR model. Instead, we utilize the ASR system’s word-level confidence scores for improving AED performance. Specifically, we add an ASR Confidence Embedding (ACE) layer to the AED model’s encoder, allowing us to jointly encode the confidence scores and the transcribed text into a contextualized representation. Our experiments show the benefits of ASR confidence scores for AED, their complementary effect over the textual signal, as well as the effectiveness and robustness of ACE for combining these signals. To foster further research, we publish a novel AED dataset consisting of ASR outputs on the LibriSpeech corpus with annotated transcription errors.

## introduction automatic speech recognition (asr) systems transcribe audio signals, consisting of speech, into text. while state-of-the-art asr systems reached high transcription quality, training them requires large amounts of data and compute resources. fortunately, many high performing systems are available as off-the-shelf cloud services. however, a performance drop can be observed when applying them to specific domains or accents (khandelwal et al., 2020; mani et al., 2020), or when transcribing noisy audio. moreover, cloud services usually expose the asr models as a black box, making it impossible to further fine-tune them. asr error detection (aed) models are designed to post-process the asr output, in order to detect transcription errors and avoid their propagation to downstream tasks (errattahi et al., 2018). aed models are widely used in interactive systems, to engage the user to resolve the detected errors. for example, aed systems can be found in google docs voice typing, where low confidence words are underlined, making it easier for users to spot errors and take actions to correct them. modern nlp models usually build upon the transformer architecture (vaswani et al., 2017). however, no transformer-based aed models have been proposed yet. recently, the transformer has been applied to asr error correction (mani et al., 2020; liao et al., 2020; leng et al., 2021a,b), another asr post-processing task. these models use only the transcription hypothesis text as input and discard other signals from the asr model. however, earlier work on aed (not transformer-based) has shown the benefits of such signals (allauzen, 2007; pellegrini and trancoso, 2009; chen et al., 2013) and specifically the benefits of asr wordlevel confidence scores (zhou et al., 2005), which are often provided in addition to the transcribed text (jiang, 2005; qiu et al., 2021; li et al., 2021). in this work we focus exclusively on aed and propose a natural way to embed the asr confidence scores into the transformer architecture. we introduce red-ace, a modified transformer encoder with an additional embedding layer, that jointly encodes the textual input and the word-level confidence scores into a contextualized representation (figure 2 ). our aed pipeline first quantizes the confidence scores into integers and then feeds the quantized scores with the transcribed text into the modified transformer encoder (figure 1 ). our experiments demonstrate the effectiveness of red-ace in improving aed performance. in addition, we demonstrate the robustness of red-ace to changes in the transcribed audio quality. finally, we release a novel dataset that can be used to train and evaluate aed models. ## red-ace following recent trends in nlp, we use a pretrained transformer-based language model, leveraging its rich language representation. red-ace is based on a pre-trained bert (devlin et al., 2019), adapted to be confidence-aware and further fine-tuned for sequence tagging. concretely, our aed model is a binary sequence tagger that given the asr output, consisting of the transcription hypothesis words and their corresponding wordlevel confidence scores, predicts an error or noterror tag for each input token. @xcite our aed pipeline is illustrated in figure 1 . first, we quantize the floating-point confidence scores into integers using a binning algorithm. @xcite next, the quantized scores and the transcription text are fed into a confidence-aware bert tagger. in bert, each input token has 3 embeddings: token, segment and position. @xcite to adapt bert to be confidence-aware, we implement an additional dedicated embedding layer, indicating the confidence bin that the input token belongs to. we construct a learned confidence embedding lookup matrix @xmath0, where b is the number of confidence bins and h is bert's embedding vector's size. for a given token, its input representation is constructed by summing the corresponding bert's @xcite we discuss words to tokens conversion in §a. @xcite typical confidence scores range between 0.0 to 1.0. we perform experiments with simple equal-width binning and quantile-based discretization (equal-sized buckets), as well as different bin numbers. more details in §a.1. @xcite we refer the reader to devlin et al. (2019) for more details. embeddings with its confidence embedding (figure 2 ). this allows the model to learn a dedicated dense representation vector for each confidence bin, as well as naturally combine it with the final contextualized representation of each input token. ## dataset creation and annotation to train and evaluate aed models, we generate a dataset with labeled transcription errors. labeling of asr errors. we decode audio data using an asr model and obtain the transcription hypothesis. then, we align the hypothesis words with the reference (correct) transcription. specifically, we find an edit path, between the hypothesis and the reference, with the minimum edit distance and obtain a sequence of edit operations (insertions, deletions and substitutions) that can be used to transform the hypothesis into the reference. every incorrect hypothesis word (i.e needs to be deleted or substituted) is labeled as error, the rest are labeled as noterror. audio data source. we use the librispeech corpus (panayotov et al., 2015), containing 1000 hours of transcribed english speech from audio books. @xcite the corpus contains clean and other pools, where clean is of higher recording quality. @xcite asr models. in this work we focus exclusively on a black-box setting, where the exact implementation details of the asr and the confidence models are unknown. this setting is particularly relevant since many applications rely on strong performance of black-box asr models which are exposed as cloud services. we use google cloud speech-to-text api as our candidate asr model. @xcite in our main experiments we select the default asr model. @xcite to ensure the generalization ability of red-ace, we repeat our main experiments using a different asr model, in this case we choose the video model. data release. since google cloud requires a paid subscription and since the underlying asr models may change over time, we make our dataset publicly available. @xcite this ensures full reproducibility of our results (in case the asr models change) and makes it easier for researchers to train and evaluate aed models, removing the need to run inference on paid cloud-based asr models or train dedicated models in order to transcribe audio. ## experimental setup our experiments examine the informativeness of the confidence scores as well as the effectiveness of red-ace in combining them with text. we provide extensive implementation details in §a.1. [cls] input tokens encoder concat (token rep + confidence scores) predicted errors error detection layer token embeds segment embeds position embeds c0 e0 c1 e1 c2 e2 c3 e3 c4 e4 c5 e5 c6 e6 2019) as a masked language model (mlm) in order to infill them. predicts error for input words that are not in the top k bert's suggestions. @xcite bert we fine-tune bert (devlin et al., 2019) for sequence tagging (only on text, without adding red-ace). as transformers have not beeen applied for aed yet, we choose bert as a pretrained lm following cheng and duan (2020), who applied it for grammatical error detection (ged) and achieved the highest performance in the nlptea-2020 shared task (rao et al., 2020). bert & c predicts error if bert predicts error and confidence is below a threshold. @xcite bert | c predicts error if bert predicts er-ror or confidence is below a threshold. @xcite bert c we fine-tune bert jointly with the confidence scores by concatenating the score value to the token's contextualized representation produced by bert (directly before it is fed into the sequence tagger). bert's last hidden layer dimension is increased by 1, and the corresponding value populated with the token's confidence score. an illustration is provided in figure 4 . ## results we next observe that bert & c performs worse than bert on all metrics. when comparing bert | c to bert we observe the expected increase in recall (bert's errors are a subset of the errors from bert | c) and a decrease in precision, f1 decreases on clean and increases on other. the results on bert c are particularly surprising. similarly to red-ace, bert c trains bert jointly with the scores. however, unlike red-ace, bert c performs worse than bert. this demonstrates the effectiveness and importance of our modeling approach, that represents the scores using a learned dense embedding vectors. as red-ace is the only method that successfully combines the scores with text, we focus the rest of the analysis on comparing it to the text-based bert tagger. in the main settings ( finally, we analyze the robustness settings from uncertainty calibration adapts a models prediction probabilities to better reflect their true correctness likelihood (guo et al., 2017). we provide the brier scores (evaluating calibration) for our dataset in aed. we provide a brief summary of relevant aed papers, for a more thorough review of aed we refer the reader to errattahi et al. (2018). modern transformer-based approaches have not addressed the aed task directly. a few attempts were made to apply transformers for asr error correction, using a sequence-to-sequence models to map directly between the asr hypothesis and the correct (reference) transcription (mani et al., 2020; liao et al., 2020; leng et al., 2021a,b). to the best of our knowledge, our work is the first to address aed using the transformer and to introduce representation for asr confidence scores in a transformer-based asr post-processing model. ## conclusion we introduced red-ace, an approach for embedding asr word-level confidence scores into a transformer-based asr error detector. red-ace jointly encodes the scores and the transcription hypothesis into a contextualized representation. our experiments demonstrated that the asr word-level confidence scores are useful on top of the transcription hypothesis text, yet it is not trivial to effectivelly combine these signals. we showed that performing such combination using red-ace leads to significant performance gains, as well as increased robustness to changes in the audio quality, which can be crucial for real-world applications. in addition, we published a novel aed dataset that allows researchers to train and evaluate aed models, without the need to run asr models. it also ensures the full reproducibility of our results in case google cloud models will change over time. in future work, we would like to leverage additional signals from the asr model (such as alternative hypotheses), as well as explore the benefits of confidence scores for error correction models.