Speech-to-Speech Translation for a Real-world Unwritten Language (2023)

Abstract: We study speech-to-speech translation (S2ST) that translates speech from one language into another language and focuses on building systems to support languages without standard text writing systems. We use English-Taiwanese Hokkien as a case study, and present an end-to-end solution from training data collection, modeling choices to benchmark dataset release. First, we present efforts on creating human annotated data, automatically mining data from large unlabeled speech datasets, and adopting pseudo-labeling to produce weakly supervised data. On the modeling, we take advantage of recent advances in applying self-supervised discrete representations as target for prediction in S2ST and show the effectiveness of leveraging additional text supervision from Mandarin, a language similar to Hokkien, in model training. Finally, we release an S2ST benchmark set to facilitate future research in this field.

## introduction speech-to-speech translation (s2st) aims at translating speech from one language into speech in another language. s2st technology can not only enable communication between people speaking different languages but also help knowledge sharing across the world. while more than 40% of the languages in the world do not have text written forms foot_2 , s2st for unwritten languages still remains a research area with little exploration mainly due to the lack of training data. the majority of the previous work on this topic conducts experiments on datasets built from applying tts on s2t corpora to generate synthetic target speech for model training @xcite . @xcite presents the first textless s2st system trained on real s2st data, while it only investigates translation between high-resource and similar language pairs (english↔spanish, english↔french). in this work, we take taiwanese hokkien as an example of an unwritten language and study s2st between english (en) and taiwanese hokkien. taiwanese hokkien (hereafter hokkien) is one of the official languages in taiwan spoken by over 70% of the population (approximately 15.8 million people). hokkien lacks a unified writing system that is widely adopted by its native speakers, though a few possible writing systems exist, e.g. chinese characters (hanji), or romanization systems such as pehōe-jī (poj) and tâi-lô, etc. in addition, hokkien is a tonal language that has complex tone sandhi rules @xcite . @xcite investigates mandarin-taiwanese hokkien s2st with a cascaded template matching approach. in our work, we focus on en↔hokkien, a distant language pair, and build one-stage s2st systems. we take advantage of the discrete unit-based s2st approach @xcite to translate source speech into target discrete units, where we convert the target speech into a sequence of integers by a self-supervised speech encoder. first, to support @xmath0, we extend hubertbased discrete unit extraction @xcite and examine the feasibility of unit-to-waveform generation @xcite for tonal languages. second, we leverage the unit-based speech normalization technique proposed in @xcite to remove the non-linguistic variations in speech from multiple speakers. the original study takes advantage of synthetic speech generated from tts as the reference target for normalization, while we build the normalizer with real hokkien speech data. last but not least, we study two s2st model training strategies, speech-to-unit translation (s2ut) with a single decoder @xcite or a two-pass decoding process @xcite that leverages mandarin (zh) as a written language similar to hokkien to provide extra text supervision. as no en↔hokkien s2st dataset is available, we also leverage mandarin to assist the s2st data creation process and create a 60-hr human annotated training set and an open benchmark set. nevertheless, this is still a low-resource problem. to tackle the data scarcity issue, we further apply en↔zh mt to create weakly supervised data @xcite and learn a joint embedding space for english and hokkien through mandarin to support data mining from unlabeled english and hokkien data @xcite . the contributions of this work are as follows: ## related work conventionally, s2st can be achieved via the concatenation of three systems: automatic speech recognition (asr), machine translation (mt) and text-to-speech synthesis (tts) @xcite . in recent years, the advancement from end-to-end speech-to-text translation (s2t) @xcite or text-to-speech translation (t2st) @xcite have simplified the s2st pipeline into two stages, which reduces error propagation issues and improves efficiency @xcite . most recently, researchers have built one-stage s2st systems that can be categorized in several aspects. first, systems that model directly from source to weakly supervised data created from tts @xcite or a cascaded pipeline with asr and mt models @xcite is often combined with the s2st data. in addition, self-supervised pre-training with large-scale unlabeled data also effectively improves s2st model performance @xcite . ## methodology in this section, we first present two types of backbone architectures for s2st modeling. then, we describe our efforts on creating parallel s2st training data from human annotations as well as leveraging speech data mining @xcite and creating weakly supervised data through pseudolabeling @xcite . ## experimental setup in this section, we describe the data, model training details, as well as baseline systems and the evaluation protocol. all experiments are conducted using fairseq @xcite . ## conclusions we present the first en↔hokkien s2st systems, where hokkien is an oral language that does not have standard and widely adopted text writing systems, i.e. an unwritten language. to tackle the challenges of speech translation for unwritten languages and the lack of parallel training data, we present an end-to-end study. first, we explore three options of training data creation including human annotation, weakly supervised data from pseudolabeling and data mining. second, we investigate two modeling choices including direct speech-tounit translation with a single speech unit decoder and two-pass decoding that leverages extra supervision from target text. experimental results show that leveraging a similar high-resource written language (mandarin in the case of hokkien) is effective in both the data creation process and model training. finally, we release the benchmark dataset and asr evaluation model to facilitate research in this field. in the future, we aim to expand study and establish an s2st model building strategy that works for a diverse set of unwritten languages.