Optimized Tokenization for Transcribed Error Correction (2023)

Abstract: The challenges facing speech recognition systems, such as variations in pronunciations, adverse audio conditions, and the scarcity of labeled data, emphasize the necessity for a post-processing step that corrects recurring errors. Previous research has shown the advantages of employing dedicated error correction models, yet training such models requires large amounts of labeled data which is not easily obtained. To overcome this limitation, synthetic transcribed-like data is often utilized, however, bridging the distribution gap between transcribed errors and synthetic noise is not trivial. In this paper, we demonstrate that the performance of correction models can be significantly increased by training solely using synthetic data. Specifically, we empirically show that: (1) synthetic data generated using the error distribution derived from a set of transcribed data outperforms the common approach of applying random perturbations; (2) applying language-specific adjustments to the vocabulary of a BPE tokenizer strike a balance between adapting to unseen distributions and retaining knowledge of transcribed errors. We showcase the benefits of these key observations, and evaluate our approach using multiple languages, speech recognition systems and prominent speech recognition datasets.

## introduction speech recognition systems has undergone remarkable progress in the last several years, largely due to the growing adoption of the self-supervision paradigm. @xcite @xcite . these systems can attain low error rates given sufficient model capacity and quality labeled data, yet they are susceptible to errors caused by audio and speech factors, such as speakers pronunciation, background noise, recording quality and orthographic errors which occurs mostly in nonphonetic languages. figure 1 : the best performing bpe hyperparameters for seven european correction models. phonetic (isolating, low morpheme-word ratio) languages tend to use short tokens with small vocabularies. morphological-rich (bundles several morphemes in each word, inflectional patterns) requires longer tokens and larger vocabularies to memorize inflected words and complex structures. to this end, researchers suggested text-to-text error correction methods specifically designed to correct transcribed errors @xcite @xcite . these transformer-based methods typically requires a massive and diverse training data (i.e., transcribed audio segments and their corresponding humanauthored transcriptions) which is rarely at reach. to bridge that gap, synthetic data is frequently incorporated to ensure a sufficient amount of training examples. a common approach is to introduce random perturbations or use predefined confusion sets @xcite @xcite . nevertheless, this approach might result with perturbations that are unrelated to transcribed errors, thereby failing to accurately replicate the distribution of such errors. a promising solution would involve generating a set of synthetic examples that follows the error distribution of transcribed data, and utilize it to train a correction model. such model needs to find a balance between memorizing the transcribe-like patterns by fitting the synthetic training data, and generalizing to unseen distributions. transformer-based correction models mostly use subword tokenizers such as bpe @xcite , which merges tokens based on a learned set of rules proportional to the tokenizer's vocabulary size. prior research showed that small vocabularies consisting of short tokens, such as individual characters, may assist with spelling errors, natural noise and generalize to unseen distributions @xcite . however, it may also lead to degraded performance in some settings @xcite . alternatively, it has been shown that large vocabularies can affect the model's memorization capabilities @xcite , which can be significant for recurring transcribed error patterns. to address these shortcomings, we generate examples that mimic the error distribution of transcribed texts by incorporating information derived from transcribed errors.we then balance between memorizing erroneous patterns and generalizing to unseen distributions using language-specific adjustments to the vocabulary of bpe tokenizers. these adjustments limit the vocabulary size and the maximum length of vocabulary tokens. interestingly, we find that the optimal vocabulary limitations differs between languages (figure 1 ). our experiments reveal a correlation between the vocabulary size and the phonetic complexity of the languages. phonetically simpler languages tend to perform better with smaller bpe vocabularies. furthermore, we have noticed that languages with rich morphological structures benefit from a longer token length limitation. one possible explanation is that non-phonetic languages with extensive morphology structures require memorizing error patterns that consist of longer sequences. the field of research concerning the maximum length of individual vocabulary tokens, irrespective of vocabulary size, is currently lacking in extensive study. our contributions can be summarized as follows: however, these methods require a significant amount of training data, consisting of pairs of transcribed segments and their corresponding groundtruth transcriptions. unfortunately, publicly available resources are often insufficient, and obtaining labeled data can be expensive and time-consuming. to overcome this challenge, a commonly used approach involves generating synthetic training examples. these examples are intentionally noised to resemble transcribed sequences and are then combined with the labeled examples for training the correction model. the perturbations can be introduced through random noise @xcite @xcite , predefined confusion sets like homophone dictionaries @xcite , or by using predictions from a pre-trained model @xcite . we hypothesize that simply incorporating random perturbations or aiming for a similar overall error rate as a labeled example set is insufficient. instead, it is crucial to closely align with the distribution of transcribed errors. throughout this study, we provide empirical evidence to support the validity of this hypothesis. robustness to perturbations. speech recognition systems commonly include a language model in their post-processing stage. the likelihood score produced by the language model allows prioritizing multiple transcribed candidates @xcite @xcite , and act as a correction mechanism to potentially corrupt transcriptions. however, the transcribed examples may contain character-level errors such as insertions, deletions, and substitutions, which can undermine the effectiveness of the language model @xcite @xcite . previous studies have demonstrated the benefits of employing small vocabularies, such as characterlevel representations, when dealing with noisy or misspelled texts @xcite @xcite . inspired by this line of research, we conduct experiments with low-granularity representations to correct errors. our research is also related to a body of work that focuses on creating robust representations through the regularization of the tokenization process @xcite . our work stands out in that our objective is to discover an optimal segmentation that is specifically tailored for transcribed errors in a particular language. ## methodology in this work, we create transcribed errors correction models that acts as a post-processing step, correcting the outputs of a speech recognition model. we derive the error distribution from transcribed labeled data and leverage it to create synthetic examples. to capture an informative error distribution rather than random one, we transcribe audio segments using fine-tuned xls-r models and compare the transcribed outputs with ground-truth transcriptions. based on the extracted distribution, we introduce perturbations to a large set of text sequence, resulting in synthetic examples that resemble transcribed data. these synthetic examples are subsequently utilized to train our error correction models, as well as bpe tokenizers. we aim to control the model's ability to generalize to unseen distributions and memorize transcribed error patterns by adjusting the bpe vocabulary. concretely, we experiment with the vocabulary size and the maximum length of each token. ## experimental setup transcribed error correction. we conduct experiments with seven european languages and utilize correction models trained on synthetically generated examples. our experiments includes correcting texts transcribed using several speech recognition models. we employ bpe tokenizers with vocabulary sizes ranging from 500 to 25k tokens, and a token length limitation varying from 4 to 12991 16 characters. for a coarse-grained segmentation comparison, we utilize a character-based tokenizer consisting of the characters found in the training sets. additionally, we compare our approach to randomly generated synthetic noise at the character level. we follow the procedure described in @xcite to create randomly noised sentences along with their corresponding ground truth counterparts. generalizing to unseen distributions. we conduct two experimental settings to evaluate the generalization abilities of our approach. firstly, we perform a cross-dataset evaluation to test the performance on datasets not included in the training process. secondly, we evaluate our approach on samples transcribed using a different speech recognition model, whisper @xcite , to assess its ability to capture unique patterns specific to speech recognition. this evaluation is particularly important since our approach is trained using errors obtained by xls-r models, which have different training objectives than whisper and may yield different error distributions. ## conclusions. in this paper, we propose a transcribed errors correction models trained on synthetic data generated using the errors derived from transcribed examples. we introduce a noise protocol that leverages the error distribution extracted from transcribed samples, and demonstrate the importance of adjusting the vocabulary size and the maximum token length of the models' bpe tokenizer. moreover, we show that such vocabulary adjustments varies between languages, as they are correlated with their levels of phonetic and morphological properties. we highlight the concerns and emphasize the limitations associated with training error correction models using synthetic data generated by applying random noise. we showcase the generalization abilities and the performance gains of our proposed approach using several datasets and speech recognition models. the claims made in the paper are supported by a comprehensive experimental study. we hope that this research will inspire further exploration of language-specific adjustments in various natural language processing tasks and domains.