Can Synthetic Speech Improve End-to-End Conversational Speech Translation? (2024)

Abstract: Conversational speech translation is an important technology that fosters communication among people of different language backgrounds. Three-way parallel data in the form of source speech, source transcript, and target translation is usually required to train end-to-end systems. However, such datasets are not readily available and are expensive to create as this involves multiple annotation stages. In this paper, we investigate the use of synthetic data from generative models, namely machine translation and text-to-speech synthesis, for training conversational speech translation systems. We show that adding synthetic data to the training recipe increasingly improves end-to-end training performance, especially when limited real data is available. However, when no real data is available, no amount of synthetic data helps.

## introduction the growing globalization of our society requires effective technologies that foster communication among individuals of varying language backgrounds. speech translation is an important technology that fosters everyday communication among individuals from different language backgrounds, bridging cultural and linguistic barriers. the technology has improved dramatically in recent years thanks to deep learning, but most gains have been demonstrated on formal settings such as parliamentary speeches, prepared monologues, and university lectures. informal conversations pose significant challenges due to the lack of training data. conversations deviate from formal written language and include informal expressions, slang, overlapping speech, incomplete sentences, varying intonation, pace, and emotion, which are typically not present in standard speech translation datasets. to capture these nuances, conversational datasets are essential for training models to understand and translate reallife spoken language accurately. however, creating this type of data usually involves individuals talking on the telephone for hours about various topics, followed by multiple annotation stages involving segmenting the long-form speech into chunks, transcribing the various chunks, and then translating them into the target language. executing these tasks is tedious, time-consuming, and expensive. this motivates a new approach of utilizing synthetic data from generative models. generative figure 1 : overview of approach in generating three-way parallel data (foreign speech, foreign text, english text) for training end-to-end conversational speech translation systems. models present an attractive solution by being able to produce large amounts of synthetic conversational speech quickly in a flexible and cost-effective manner. when used as a data augmentation technique, this synthetic data can potentially improve the performance of speech translation systems in conversational speech domains. over the years, machine translation has seen significant advances. machine translation (mt) models utilizing the transformer architecture when trained on large data sets across multiple languages generalize better and can generate adequate and fluent translations in near real-time. in addition, textto-speech synthesis (tts) has attained significant strides resulting in high-quality synthesized voices that closely mimic human speech. in this study, we leverage the advances from mt and tts to show that a synthetic conversational speech dataset that is easier and cheaper to create can be used for the task of conversational end-to-end speech translation. to do this, we create a dataset of synthetic speech by back-translating monolingual text from the target language to the source language, and then generating the speech in the source language speech from the back-translated text using a tts system (figure 1 ). we seek to answer the following questions: synthetic data the use of synthetic data has been extensively studied for text-based machine translation. sennrich et al. (2016a) shows that generating synthetic source sentences from target monolingual data through back-translation helps boost neural machine translation performance. amin et al. (2021) investigate the use of synthetic data for training rnn-t asr models via a multi-stage training pipeline with continual learning. rossenbach et al. (2020) show that training attention-based asr systems on synthetic data leads to huge improvements in word-error-rate (wer). rossenbach et al. (2021) compare the benefits of training with synthetic data for four asr architectures, namely -attention encoder-decoder (aed), hybrid asr, ctc, and monotic rnn-t. fang and feng ( 2023 ) train a target-to-unit model to map the target text to source speech units @xcite . they then utilize a unit vocoder to map the source units into a waveform. @xcite ; @xcite expand these augmentation methods to lowresource and accented asr, respectively. ## data creation methods we use the term three-way parallel data to refer to the aligned source speech, source transcript, and target translation that is necessary for end-to-end speech translation model training. we first describe the manually-created 3-way parallel data (referred to as real in subsequent sections) used for baseline models. we then explain the synthetically generated 3-way parallel data (which we refer to as synth in subsequent sections) used for data augmentation. ## experiments and results this section outlines the various experiments and results of this study. we use the espnet @xcite toolkit for all experiments. we evaluate all systems with bleu @xcite , as implemented by sacrebleu foot_4 @xcite . ## takeaways we summarize our findings here: (1) when there is a small amount of real speech available, including a large amount of synthetic speech leads to higher performance gains in end-to-end training; (2) when there is a large amount of real data available, including synthetic data leads to minimal performance gains; ## conclusion we investigated whether using synthetic data generated from backtranslation and text-to-speech synthesis for end-to-end conversational speech translation improves performance. incorporating synthetic data into a conversation speech translation training recipe helps improve the overall system's performance especially when there is limited real speech available. when models are trained on only synthetic data, we find that models do not generalize beyond their training domain. this mismatch between the synthetic training data and real-world data leads to suboptimal performance when models trained on only synthetic data are applied to real speech. this highlights the importance of incorporating at least some real speech data during training to bridge the domain gap effectively. in the case where no real speech is available, no amount of synthetic data helps. there are several open questions worth examining as future work: