Wenzhou Dialect Speech to M andarin Text Conversion (2025)

Abstract: The Wenzhou dialect is a Chinese dialect that is significantly distinct from Mandarin, the official language of China. It is among the most complex Chinese dialects and is nearly incomprehensible to people from regions such as Northern China, thereby creating substantial communication barriers. Therefore, the conversion between the Wenzhou dialect and Mandarin is essential to facilitate communication between Wenzhou dialect speakers and those from other Chinese regions. However, as a low-resource language, the Wenzhou dialect lacks publicly available datasets, and such conversion technologies have not been extensively researched. Thus, in this study, we create a parallel dataset containing Wenzhou dialect speech and the corresponding Mandarin text and build benchmark models for Wenzhou dialect speech-to-Mandarin text conversion. In particular, we fine-tune two self-supervised learning-based pretrained models, that is, TeleSpeech-ASR1.0 and Wav2Vec2-XLS-R, with our training dataset and report their performance on our test dataset as baselines for future research.

## introduction china has 56 ethnic groups over a wide geographical span, resulting in a significant variety of chinese dialects. in addition to mandarin, which is the official standard chinese, active chinese dialects can generally be divided into nine categories: wu chinese 1 , cantonese, jin chinese, min chinese, hakka, xiang chinese, gan chinese, hui chinese, and pinghua (chinese academy of social sciences and city university of hong @xcite . some speakers of these dialects experience difficulty communicating with mandarin speakers, and these dialects have significant differences between branches. this study focuses on the wenzhou di-1 https://www.ethnologue.com/language/wuu/ figure 2: differences in word usage the wenzhou dialect is considered the most difficult dialect in china because of its unique phonetic system, which differs markedly from those of mandarin and other chinese dialects @xcite . figures 1 2 3 present simple examples illustrating the differences between the wenzhou dialect and mandarin. figure 1 shows the differences in pronunciation between wenzhou dialect and mandarin. although the same chinese characters are used in figure 1 , the pronunciation of the wenzhou dialect is completely different from that of mandarin. figure 2 illustrates the differences in word usage between the wenzhou dialect and mandarin. in the figure, the orange underlined parts are specific to the wenzhou dialect, with certain chinese characters no longer used in modern mandarin. figure 3 illustrates the differences in figure 3 : differences in grammar terms of grammar between the wenzhou dialect and mandarin. in the wenzhou dialect, the verb is placed before the adverb, which is the opposite to that in case of mandarin. moreover, the order of chinese characters within the noun differs between the two. because of these differences, the wenzhou dialect is typically incomprehensible to people from regions such as northern china. therefore, the conversion between the wenzhou dialect and mandarin is essential to facilitate communication between wenzhou dialect speakers and people from other regions. this study focuses on a method for converting wenzhou dialect speech to mandarin text. neural network-based end-to-end frameworks have achieved remarkable success in speech-totext tasks, such as automatic speech recognition (asr) @xcite @xcite and speech-to-text translation (st) @xcite @xcite . certain advanced models perform very well on high-resource languages owing to the richness and diversity of available data. in contrast, low-resource languages face many challenges, particularly some dialects lacking data support and complex phonetic systems. the wenzhou dialect is also a low-resource language with no publicly available datasets, and methods for converting wenzhou dialect speech into mandarin text are yet to be explored. in an attempt to address these challenges, this study makes the following three key contributions: ## related work this section presents a review of related work on the ssl-based pretrained models, telespeech-asr1.0 and wav2vec2-xls-r, which form the basis of our benchmark models. low-resource languages lack the resources to support the scale of annotated data required to train neural network models. ssl has achieved tremendous success across various fields, including natural language processing and speech processing. ssl facilitates neural networks in learning rich feature representations from large amounts of unlabeled multilingual data @xcite . models pretrained by ssl can be fine-tuned on various downstream tasks @xcite , and the resulting fine-tuned models enable strong downstream performance even with limited access to annotated data. ## benchmark dataset for converting the wenzhou dialect to mandarin this study constructs a benchmark dataset for converting the wenzhou dialect to mandarin because currently there is no publicly available dataset for this task. although we have presented the wenzhou dialect texts for convenience in the three examples (figures 1 2 3 ) used for comparisons of the wenzhou dialect and mandarin in the introduction section, the wenzhou dialect does not have a formal writing system. even native wenzhou dialect speakers might not know the meaning of the orangeunderlined chinese character in figure 2 , some of which are no longer in use in modern mandarin. therefore, we construct a parallel dataset comprising the wenzhou dialect speech (not text) and the corresponding mandarin text. we construct the dataset from a local news video program in the wenzhou dialect as per the following two steps: (1) extraction of the wenzhou dialect audio and (2) annotation of the corresponding mandarin text. in step 1, we first collect 144 news videos hosted and broadcast in the wenzhou dialect uploaded on youtube 5 . subsequently, we extract the wenzhou dialect audio from these videos and manually segment the audio into sentences. in step 2, we extract the mandarin subtitles embedded in the videos using paddle optical character recognition (ocr) tools 6 , followed by manual correction to ensure quality. after the two steps, we obtain approximately 18 hours of the wenzhou dialect speech and the corresponding mandarin text. the dataset comprises 8,068 audio files, with an average duration of approximately 8 seconds per file. to divide the data into training, validation, and test datasets, we shuffle the order of video ids and split the dataset based on the video id as follows: 76 videos for training, 33 videos for validation, and 35 videos for testing. of the 35 videos for testing, only audio files with durations greater than 5 seconds are extracted as test samples. this corresponds to 5,037, 1,004, and 2,027 audio files for training, validation, and testing, respectively, as summarized in ## experiments in this section, we present two benchmark models for the wenzhou dialect speech-to-mandarin text conversion and evaluate their performance on the constructed benchmark dataset. 5 the videos can be downloaded from https://www.youtube.com/playlist?@xmath0 ## conclusion this study constructed the first publicly available dataset for converting the wenzhou dialect speechto-mandarin text. further, we also developed benchmark models, namely, the fine-tuned models of ssl-based pretrained models, and evaluated and reported their performance on the constructed dataset as baselines for future research. the evaluations demonstrated the effectiveness of fine-tuning the ssl-based pretrained models using our dataset. however, the results revealed significant challenges, particularly in terms of capturing the phonetic and syntactic intricacies of the wenzhou dialect. to address these limitations, we plan to explore strategies such as the incorporation of decoding architectures to better capture syntactic structures and account for word order differences in future work. in addition, we plan to leverage scaling pretrained models with larger cross-lingual capabilities, such as xeus @xcite , to enrich representations for handling complex phonetic systems of the wenzhou dialect.