D ialect M o E : An End-to-End Multi-Dialect Speech Recognition Model with Mixture-of-Experts (2024)

Abstract: “Dialect speech recognition has always been one of the challenges in Automatic Speech Recog-nition (ASR) systems. While lots of ASR systems perform well in Mandarin, their performancesignificantly drops when handling dialect speech. This is mainly due to the obvious differencesbetween dialects and Mandarin in pronunciation and the data scarcity of dialect speech. In thispaper, we propose DialectMoE, a Chinese multi-dialects speech recognition model based onMixture-of-Experts (MoE) in a low-resource conditions. Specifically, DialectMoE assigns inputsequences to a set of experts using a dynamic routing algorithm, with each expert potentiallytrained for a specific dialect. Subsequently, the outputs of these experts are combined to derivethe final output. Due to the similarities among dialects, distinct experts may offer assistance inrecognizing other dialects as well. Experimental results on the Datatang dialect public datasetshow that, compared with the baseline model, DialectMoE reduces Character Error Rate (CER)for Sichuan, Yunnan, Hubei and Henan dialects by 23.6%, 32.6%, 39.2% and 35.09% respec-tively. The proposed DialectMoE model demonstrates outstanding performance in multi-dialectsspeech recognition.”

## introduction the application domains of speech recognition technology are extensive, encompassing diverse fields such as voice assistants, smart homes, and automotive voice interaction, among others. thanks to the advancements in deep learning, automatic speech recognition (asr) systems have made remarkable strides in recognizing mandarin speech @xcite @xcite . dialect serves as a prevalent mode of everyday communication among the chinese populace. however, the performance of asr systems remains limited in dialect speech, posing a significant challenge in the field of speech recognition technology @xcite due to the inherent variations and distinct characteristics in pronunciation among dialects and mandarin. therefore, improving the accuracy and adaptability of chinese asr systems is significant and meaningful for multidialect. our study mainly focuses on chinese dialects, the proposed method can also be generalized to other dialects. chinese dialects are typically classified into ten main categories, each exhibiting notable differences in pronunciation, tone, vocabulary, and grammar @xcite . chinese is a tonal language, where each character corresponds to a specific tone, a feature that is prevalent in most of its dialects as well. the pronunciation of a given chinese character with different tones imparts markedly distinct meanings. this underscores the profound significance of tones in the comprehension of chinese phonetics. @xcite . figure 1 depicts the tonal distinctions among standard mandarin, yunnan dialect, and sichuan dialect. it is evident that notable in tone between standard mandarin and yunnan dialect as well as sichuan dialect in the second, third, and fourth tones. however, yunnan dialect and sichuan dialect exhibit a pronunciation similarity in specific tones. the change in tone reveals differences and similarities between standard mandarin and various dialects. hence, considering both the differences and similarities in pronunciation among various dialects alongside standard mandarin becomes crucial for the advancement of chinese speech recognition systems. in recent years, numerous researches have focused on tackling the challenge of poor performance in dialect speech recognition models @xcite @xcite . the traditional way is based on different modeling methods to improve the effect of dialect speech recognition. @xcite employed an adaptive method that utilizes a pronunciation vocabulary with dialect data to capture differences between standard and dialect pronunciations. @xcite proposed a novel method for modeling chinese characters based on radicals, effectively addressing the issue of dialect modeling difficulty. this method significantly reduces the required size of radical dictionaries compared to ordinary character dictionaries. recently, multitask-based methods have been widely used in the task of dialect speech recognition. compared with the traditional method, the multi-task learning method is more efficient. @xcite proposed constructing a dialect classification model and a separate speech recognition model for each dialect. the dialect classification model is used to select the corresponding dialect speech recognition model. @xcite proposed a multi-task training strategy that combines dialect classification with dialect speech recognition, bridging the substantial gap between mandarin and dialect acoustic properties. however, these investigations are contingent upon extensive dialectal datasets and do not examine the potential influence of commonalities among various dialects on model performance. to construct a reliable dialect speech recognition model in low-resource conditions, jiang (2023) introduced a transfer learning-based approach, it involves a model trained on mandarin and fine-tunes with small-scale dialect data. however, relying solely on transfer learning may not adequately capture the distinctions between dialects and mandarin. @xcite proposed the aformer model with multi-stage training strategy, which can capture diverse acoustic information in different training stage, enabling the model to effectively adapt to dialect data. the aforementioned studies focus on the training strategy and model expansion, they do not fully consider the differences and similarities between dialects and mandarin. in this paper, we present dialectmoe, a multi-dialect speech recognition model based on mixture-of-experts (moe), aimed at improving the performance of multi-dialects speech recognition in low-resource conditions. dialectmoe is architecturally structured with dual encoders: a dialect encoder and a general encoder. the main contributions of the paper include: ## conclusion in this manuscript, we present a multi-dialectal speech recognition model based on moe termed dialect-moe. structurally, it incorporates a dual-encoder architecture, wherein the general encoder is dedicated to acquiring general acoustic representations, and the dialect encoder is specialized for acquiring acoustic representations across various dialects. a refinement in the dynamic routing strategy within the moe c c l 2 0 2 4 layer of the universal encoder has been introduced to enable the selection of appropriate experts based on the acoustic information specific to the dialect in the input sequence. furthermore, we propose a threestage training methodology to facilitate dialectmoe in learning distinct tasks at different phases, thereby enhancing its adaptability and performance across varying aspects of the multi-dialectal speech recognition task. experimental results demonstrate that the proposed dialectmoe model achieves remarkable performance in multi-dialects speech recognition tasks.