New Datasets and Controllable Iterative Data Augmentation Method for Code-switching ASR Error Correction (2023)

Abstract: With the wide use of automatic speech recognition(ASR) systems, researchers pay more attention to the ASR error correction task to improve the quality of recognition results. In particular, ASR in bilingual or multilingual settings, namely code-switching ASR, has greater challenges and research value. In this paper, we first present code-switching ASR correction datasets obtained from solid ASR systems and automatic annotators. The datasets contain Chinese-English code-switching dialogues of bilingual speakers in Singapore, Malaysia, and Hong Kong. Based on this task, we propose a controllable iterative (CI) data augmentation method for improving the performance of mainstream ASR error correction systems. With a small amount of training data, our proposed method has the ability to iteratively produce abundant pseudo parallel data from the monolingual corpus for Chinese-English code-switching ASR correction. Results of experiments show that our method achieves the best performance compared with the rule-based, back-translation-based data augmentation methods and large language model ChatGPT.

## introduction recently, automatic speech recognition (asr) system has achieved impressive success with the adoption of advanced deep neural network @xcite . learning from training data, the asr system can match human transcribers for some critical tasks. it has been widely applied in several areas, such as live captioning, intelligent appliance control @xcite . in some bilingual and multilingual societies, people often speak more than one language in a conversation. the speech phenomenon of switching language among sentences is called codeswitching. to explore the code-switching phenomenon, code-switching tasks in many different language pairs have been introduced @xcite , including the code-switching asr task. employing multilingual asr systems to solve code-switching asr is proposed in recent work @xcite . however, the performance of systems in many cases is still far from satisfactory. errors introduced by asr systems usually affect the performance of downstream tasks. to improve the recognition performance, the codeswitching asr error correction task is necessary. in this paper, we focus on code-switching asr error correction between chinese and english, which is common in the chinese communities of asia area. to enable the research and development of code-switching asr error correction, we propose two datasets seame-c and ascend-c that are constructed from two chinese-english codeswitching asr datasets seame @xcite and ascend @xcite . in a challenge for the code-switching asr error correction is the lack of a large scale of training data. moreover, we can not even collect sufficient high-quality chinese-english code-switching data. considering these challenges, we propose a controllable iterative (ci) data augmentation method that can iteratively generate abundant code-switching asr error correction instances from the monolingual corpus with small-scale labeled training data. to verify the effectiveness of our proposed method, we implement two data augmentation baselines and evaluate the performance of these methods on proposed datasets at first. with a similar scale of augmented data, our proposed controllable iterative method achieves the best performance in both maxmatch (m 2 ) scorer @xcite and mer metrics on seame-c and ascend-c datasets. it performs well for both mt5 @xcite and mbart @xcite correction methods. besides, considering the strong ability of large language models (llm), we compare the performance of our method with the most powerful llm chatgpt. experiments show that our method outdistances the chatgpt method, which shows the effectiveness of our method in the chinese-english code-switching asr error correction task. besides, we find that this task is challenging, and llm method is far from achieving satisfactory results at now. our contributions are summarized as follows: the resources are publicly available foot_0 . ## related work code-switching asr, the task aiming to recognize the speech that contains more than one language within the sentences, has been increasingly reported in speech technology and linguistic studies recently @xcite . however, to our knowledge, there is no research on the codeswitching asr error correction task at present. in this section, we introduce two relevant fields: asr error correction and data augmentation method in text error correction. with the successes of asr systems, the asr error correction has become a prevalent task in recent years. the idea of regarding the asr error correction as the machine translation and applied advanced neural machine translation models is adopted in several previous approaches @xcite . d'haro and banchs (2016) first introduce machine translation to improve automatic transcription. considering the machine translation models require only parallel text pairs, @xcite propose an approach to utilizing text-only data by training a correction model to explicitly correct the errors. the joint modeling method shows great performance in this task @xcite . re-ranking asr n-best hypotheses is also a popular topic for error recovery @xcite . to address the problem of lack of training data, data augmentation method is widely adopted in text error correction area. many works @xcite adopt pre-defined rules to generate sentences with errors. inspired by back-translation procedure for machine translation @xcite , works @xcite are proposed to train a model to generate erroneous sentences. recently, researchers focus on the exposure bias problem of data augmentation method and proposed methods to utilize the augmented data in more effective ways @xcite . ## code-switching asr error correction datasets in this section, we provide details about our two datasets seame-c and ascend-c. specifically, we give the construction process of our datasets at first. then, we do an analysis of our datasets and show some characteristics of them. in this work, we adopt the same corpus splitting as the original asr datasets. in addition, we remove some bad cases that are too short and eliminate monolingual sentences in the validation and test sets to ensure the quality of evaluation. our datasets provide quality resources for the research and development of code-switching asr error correction. ## controllable iterative data augmentation method to address the problem of lacking sufficient training data, we propose the controllable iterative data augmentation method. due to the shortage of high-quality chinese-english code-switching corpus, our method is designed to directly generate training instances from monolingual texts. our method consists of two parts: sequence editing model and pseudo data generation algorithm. sequence editing model is the fundamental structure of our method. it adopts the transformer architecture and the predefined editing operations called transformations to generate raw augmented data from seed corpus. pseudo data generation algo-rithm plays a crucial role in the proposed method. it adopts the sequence editing model to iteratively produce pseudo training data under the control of confidence bias. with our designed transformations, the sequence editing model is able to directly generate training instances from monolingual texts. figure 1 shows an example of the process of generating pseudo data in our controllable iterative method. ## experimental setup in this section, we present our experimental setup including datasets, asr error correction model, and data augmentation baseline. more details of implementation are presented in appendix d. ## conclusion in this paper, we explore the code-switching asr error correction task. to enable the research, we first propose two chinese-english code-switching asr correction datasets. to address the problem of lacking sufficient training data, we propose the controllable iterative data augmentation method that can generate code-switching asr error correction instances from the monolingual corpus iteratively. experiment results show that our method achieves the best performance compared with the rule-based augmentation method, back-translation augmentation method, and chatgpt method. we further find that combining the data generated by both rulebased method and controllable iterative method can further improve the performance of the error correction model. in future work, we will apply our data augmentation method to other code-switching tasks and also test our method in other language pairs.