KEBAP : K orean Error Explainable Benchmark Dataset for ASR and Post-processing (2023)

Abstract: Automatic Speech Recognition (ASR) systems are instrumental across various applications, with their performance being critically tied to user satisfaction. Conventional evaluation metrics for ASR systems produce a singular aggregate score, which is insufficient for understanding specific system vulnerabilities. Therefore, we aim to address the limitations of the previous ASR evaluation methods by introducing the Korean Error Explainable Benchmark Dataset for ASR and Post-processing (KEBAP). KEBAP enables comprehensive analysis of ASR systems at both speech- and text levels, thereby facilitating a more balanced assessment encompassing speech recognition accuracy and user readability. KEBAP provides 37 newly defined speech-level resources incorporating diverse noise environments and speaker characteristics categories, also presenting 13 distinct text-level error types. This paper demonstrates detailed statistical analyses of colloquial noise categories and textual error types. Furthermore, we conduct extensive validation and analysis on commercially deployed ASR systems, providing valuable insights into their performance. As a more fine-grained and real-world-centric evaluation method, KEBAP contributes to identifying and mitigating potential weaknesses in ASR systems.

## introduction automatic speech recognition (asr) is a task that recognizes speech and converts it into text, and it is getting more and more attention with the development of voice interface applications and devices such as alexa, siri, and cortana @xcite @xcite . in the real world, the asr result has a trade-off between recognition accuracy 1 and user readability. even figure 1 : an example of using kebap to evaluate weaknesses in asr results. the above approach with conventional evaluation methods lacks explanatory power. evaluation using kebap below enables a detailed description of the weaknesses. if the asr model accurately recognizes the input voice, the user's readability may decrease. this is because humans do not always utter perfect sentences in the real world (e.g., incomplete utterances, sighs, etc.). to achieve balanced asr results in this trade-off situation, it is required to consider both recognition accuracy and user readability. in terms of recognition accuracy, various asr evaluation metrics such as word error rate (wer) (woodard and nelson) and character error rate (cer) @xcite have been prevalent. also, readability is considered in asr post-processing (asrp) tasks, where the goal is to improve the clarity and comprehension of speech recognition outputs without modifying the underlying model architecture @xcite @xcite . the asrp task relies on quantitative metrics, such as bleu @xcite and gleu @xcite , similar to the asr evaluation using wer and cer. however, it is crucial to recognize that even if quantitative evaluation scores are similar, the qualitative aspects of the asr results may not necessarily align. conventional research methods, which focus on accuracy or user readability, compute quantitative scores based on the degree of alignment between inputs and outputs. this approach falls short in classifying potential error types or pinpointing the model's specific weaknesses, thus lacking explanatory power for real-world asr model outputs. this deficiency hinders the establishment of clear directions for model improvement. to this end, datasets that aim to enhance the explanatory power of asr evaluations by considering noisy environments or speaker characteristics have been published recently @xcite @xcite . however, these datasets still focus on accuracy and provide a limited set of error types, thereby leading to insufficiency in diagnosing specific weaknesses within asr models. therefore, we introduce the novel korean error explainable benchmark dataset for asr and postprocessing (kebap) foot_0 . it encompasses speechlevel distraction-based resources and text-level error types relevant to real-world asr applications. these diverse error types of kebap can lead to improved explanatory capability compared to the previous examination methods, as illustrated in figure 1 . in particular, speech-level noise types are bifurcated into two categories: noisy environments and speaker characteristics, comprising 37 distinct types. additionally, kebap includes 13 types of textual errors pertinent in asr contexts. the dataset stands out in its authenticity since all speech samples are recorded by human speakers, and background noises are derived from real-world environments. also, we annotate the difficulty levels to all types, enhancing the interpretability of the asr model. we employ kebap to conduct an empirical analysis of the correlations between speech-level noise types and textual error types. moreover, leveraging chatgpt (openai-blog, 2022), we explore the potential of language models in discovering the vulnerabilities of asr models. our observations highlight kebap's significant interpretability of asr model diagnostics and shed light on the pressing need for research on diagnostic tasks for asr systems. our work sets the stage for more real-world-oriented evaluations of asr systems and can contribute to the advancements in this domain. ## efficacy validation for kebap in this section, we assess the specific capabilities of commercialized asr models using kebap. to achieve this, we conduct a detailed correlation analysis of commercialized systems such as google cloud speech-to-text @xcite and clova speech (chung, 2019). we examine the correlation between speech-level noise types and text-level errors, aiming for a granular understanding. we comprehensively validate the model's capabilities by considering both speech-and text-level aspects. we verify whether the llm possesses the necessary qualities as a diagnostic model through the error type classification task. ## conclusion in the real-world, asr results involve a trade-off between recognition accuracy and user readability, thus requiring a balanced consideration of these factors. to provide guidance for improving model performance, it is necessary to enhance interpretability, which entails considering both speech-level accuracy and text-level user readability. to this end, we propose korean error explainable benchmark dataset for asr and post-processing (kebap) for diagnosing and validating models by segmenting error types while considering both speech-and text-level. to facilitate the construction process, we utilize a gec dataset that includes text-level errors and structure the process into validation, recording, synthesis of background noise, and difficulty tagging stages, employing consensus labeling within each stage to enhance the efficiency and quality of the task. we performed a detailed diagnostic analysis of the commercialization systems using kebap. furthermore, the proposed task falls into a domain that is challenging for chatgpt to cover, and it indicates the need for further research to achieve a closer approximation to real-world diagnostics. we demonstrated that kebap contributes to enhancing the interpretability of the model's weaknesses.