Hyper- BTS Dataset: Scalability and Enhanced Analysis of Back T ran S cription ( BTS ) for ASR Post-Processing (2024)

Abstract: The recent advancements in the realm of Automatic Speech Recognition (ASR) post-processing have been primarily driven by sequence-to-sequence paradigms. Despite their effectiveness, these methods often demand substantial amounts of data, necessitating the expensive recruitment of phonetic transcription experts to rectify the erroneous outputs of ASR systems, thereby creating the desired training data. Back TranScription (BTS) alleviates this issue by generating ASR inputs from clean text via a Text-to-Speech (TTS) system. While initial studies on BTS exhibited promise, they were constrained by a limited dataset of just 200,000 sentence pairs, leaving the scalability of this method in question. In this study, we delve into the potential scalability of BTS. We introduce the “Hyper-BTS” dataset, a corpus approximately five times larger than that utilized in prior research. Additionally, we present innovative criteria for categorizing error types within ASR post-processing. This not only facilitates a more comprehensive qualitative analysis, which was absent in preceding studies, but also enhances the understanding of ASR error patterns. Our empirical results, both quantitative and qualitative, suggest that the enlarged scale of the Hyper-BTS dataset sufficiently addresses a vast majority of the ASR error categories. We make the Hyper-BTS dataset publicly available.

## introduction a large-scale dataset-based nlp research paradigm, which is based on foundation models @xcite such as @xcite , and prompt tuning using natural-language prompts @xcite has recently been of interest in both the academia and industry. such large-scale models have proven that there is efficiency in the usage of large-scale datasets, and include a scaling law model @xcite , which theoretically demonstrates their justification. there is also increasing application of this promising research paradigm in the automatic speech recognition (asr) field. aside from traditional speech recognition architecture-based research such as gaussian mixture models (gmms) @xcite , and hidden markov models (hmms) @xcite based on acoustic and language models, model-centric asr research using transfer learning based on pretrained models is currently being widely conducted @xcite @xcite . model-centric asr research requires the configuring of many parameters for the pre-training of models, as well as a sufficiency of computing power (e.g., gpu) to process large-scale datasets. thus, despite its proven efficiency, insufficiency of computing power in real-world service scenarios limits the performance of this asr model approach. in other words, since many parameters and data are required when training a model, companies that do not have sufficient server or gpu environments have difficulty configuring service environments and improving performance using the model-centric asr approach @xcite . conversely, a different approach, termed "datacentric" has also emerged, which aims to improve asr model performance by improving the data quality or pre-processing and post-processing without model modification @xcite @xcite . this alleviates the previous limitations (of computation cost and non-scalable human annotation) because it does not modify the model, and enables its application to lightweight models such as the vanilla transformer, which can be sufficiently processed by a single cpu @xcite . there has been a recent endeavor in the datacentric asr post-processor approach known as back transcription (bts) @xcite . bts, an automatic data construction method, has been devised for use as substitute for publicly available training data, for asr post-processor based on a sequence-to-sequence model (converting input sequences into target sequences) and to eliminate the requirement to build parallel corpora by human-annotators. specifically, this method integrates text-to-speech (tts) with speech-to-text (stt) efficiently for building a pseudo-parallel corpus (see detail in appendix a). however, in a current bts study, model training was performed using only a 200,000 parallel corpus in korean. while this may be a significant amount from the point of view of low-resource neural machine translation (nmt), it is very small in comparison with the recent research flow utilizing large-size data. in addition, only the method and demo system were disclosed in the bts study, but no dataset was released with the work. therefore, to improve the performance of an asr post-processor, based on bts technology, we take advantage of the existing research flow to build large-capacity data and present a hyper-bts dataset that is five times the size of the existing bts study, with a one million-text large-capacity dataset. further, to activate the relevant research interest, we make it publicly accessible, dividing the data into training, validation, and test datasets. to the best of our knowledge, this is the first time a parallel corpus for an asr post-processor has been made public. by opening the data in this way, asr post-processor research can be triggered, and the problems with the existing commercial asr api systems can be studied and improved. existing commercial asr apis currently present problems such as spacing, conversion of numbers, and pronunciation boundary errors. therefore, it is inevitable that asr post-processor recognition results will contain unexpected errors. in other words, there is room for performance improvement using asr post-processor, and additionally, precise error analysis is required. despite the acknowledgement of the existence of recognition errors, there are currently no precise criteria for categorizing output error types from asr systems. many studies related to large-scale language models @xcite have through their works attempted to de-velop a model @xcite for improving asr systems. however, analysis of the types of errors output by asr systems and guidelines on research and design are insufficient, as existing studies simply analyze the advantages and disadvantages of generated results without benchmarking the results against some set of standards. in this study, we propose novel criteria of error type categorization of asr post-processor specialized in korean, in terms of bts work also based on korean. we present this set of criteria to be used for the direction of further work in enhancing asr post-processor performance. in addition, based on our defined error types, we perform an in-depth qualitative analysis of the hyper-bts dataset-based asr post-processor to verify whether actual error correction is performed well. through this, we suggest methods that can be employed to improve the performance of asr postprocessor systems. the contributions of this study are as follows: ## conclusion in this study, we conducted a thorough analysis of results from rigorous experiments after developing the hyper-bts dataset and training an automatic speech recognition (asr) post-processor. both quantitative and qualitative outcomes validate the effectiveness of the hyper-bts dataset in enhancing the performance of the asr postprocessor. recognizing the broader implications of our research, we are committed to facilitating unrestricted access to this dataset for both industry professionals and academic researchers. additionally, we pioneered a robust quality control mechanism by formulating novel guidelines anchored in the categorization of asr post-processor error types, thereby aiming to elevate the qualitative dimensions of asr post-processing.