BTS : Back T ran S cription for Speech-to-Text Post-Processor using Text-to-Speech-to-Text (2021)

Abstract: With the growing popularity of smart speakers, such as Amazon Alexa, speech is becoming one of the most important modes of human-computer interaction. Automatic speech recognition (ASR) is arguably the most critical component of such systems, as errors in speech recognition propagate to the downstream components and drastically degrade the user experience. A simple and effective way to improve the speech recognition accuracy is to apply automatic post-processor to the recognition result. However, training a post-processor requires parallel corpora created by human annotators, which are expensive and not scalable. To alleviate this problem, we propose Back TranScription (BTS), a denoising-based method that can create such corpora without human labor. Using a raw corpus, BTS corrupts the text using Text-to-Speech (TTS) and Speech-to-Text (STT) systems. Then, a post-processing model can be trained to reconstruct the original text given the corrupted input. Quantitative and qualitative evaluations show that a post-processor trained using our approach is highly effective in fixing non-trivial speech recognition errors such as mishandling foreign words. We present the generated parallel corpus and post-processing platform to make our results publicly available.

## introduction automatic speech recognition (asr) is a technology that converts human voice into text. with the emergence of deep learning, the performance of asr has been improved considerably. consequently, many firms are applying asr to their business models @xcite . although several excellent commercial api systems are available, such as google cloud speech api @xcite and naver's clova † corresponding author @xcite , most small-and mediumsized companies are building their own asr software using open-source tools such as kaldi @xcite owing to the need for domain-specific systems as well as security of in-house industrial data @xcite . in addition, many companies are operating on conventional asr architectures, such as gaussian mixture models (gmms) @xcite and hidden markov models (hmms) @xcite , which are based on acoustic and language models. however, a drawback of the above-mentioned method is that words that are not in the dictionary are misrecognized as incorrect words owing to the out-of-vocabulary (oov) problem. as this method is a statistics-based method, satisfactory performance is achieved only when a massive voice database is available . probability values for sequences of words that are not present in the training corpus are estimated to be unstable, and it is difficult to sufficiently reflect the context because n values are constrained in n-grams. moreover, the entry barrier is high because it is difficult for nonprofessionals to handle the model. to alleviate these limitations, asr studies have recently been conducted using pretrained model (pm)-based transfer learning @xcite @xcite . this methodology shows superior performance compared to methods based on the conventional asr architecture; however, it has two main limitations in terms of applying it to real-world services. first, from the data aspect, this methodology requires a large amount of training data for pretraining to service the asr software. as it is strongly dependent on the data size, it is difficult to apply it to a low-resource language (lrl), such as korean. furthermore, as the latest studies are based on a high-resource language (hrl) with sufficient training data, the same performance cannot be achieved if the same model is applied to an lrl without any special processing. second, from the service environment aspect, this methodology requires service circumstances with sufficient computing power (e.g., gpu) to process large-scale data. it is difficult to establish a sufficient hardware environment to provide services, except for large companies such as google and facebook. in other words, as training a model involves many parameters and a large amount of data, companies that do not have sufficient server or gpu environments will find it difficult to configure the service environment and improve performance using the latest model @xcite . therefore, it is important to ensure that companies with insufficient environments can provide services while performing well against lrls. to this end, instead of pm-based transfer learning, a new method for improving asr performance is required. to alleviate these limitations, some studies have attempted to improve the performance of the asr model through various pre-processing and postprocessing methods without changing the model @xcite @xcite @xcite . this approach does not require a large amount of data for pretraining the model and it can be applied to any model as well as models that can provide sufficient service with a cpu @xcite , such as the vanilla transformer @xcite . in this regard, this method can alleviate the above-mentioned limitations in terms of the data and service environment. hence, this method is particularly important from the viewpoint of lrls. accordingly, we propose back transcription (bts), a fully automated data construction method for a sequence-to-sequence (s2s)-based postprocessor model that does not require human intervention or model modification. the contributions of this study are as follows. ## related work asr post-processing is a research field that aims to improve performance by correcting the asr errors rather than changing the model architecture. the two main methodologies for asr post-processing in the field of speech recognition are the conventional methodology and the sequence-to-sequence (s2s) methodology. ## conclusion and future work we proposed bts, which can automatically generate a parallel corpus from raw corpora to train asr post-processors. by combining tts and stt systems, asr noise was injected into the raw text, and the post-processing model was trained in a denoising manner. quantitative and qualitative evaluations showed that our approach can effectively handle challenging asr errors, such as foreign word conversion. in the future, we plan to investigate different noising strategies that reflect real-world asr errors and make the denoising process more challenging. demonstrating the effectiveness of bts in additional languages from various language families is another important direction for future research.