Generating Synthetic Speech from S poken V ocab for Speech Translation (2023)

Abstract: Training end-to-end speech translation (ST) systems requires sufficiently large-scale data, which is unavailable for most language pairs and domains. One practical solution to the data scarcity issue is to convert text-based machine translation (MT) data to ST data via text-to-speech (TTS) systems. Yet, using TTS systems can be tedious and slow. In this work, we propose SpokenVocab, a simple, scalable and effective data augmentation technique to convert MT data to ST data on-the-fly. The idea is to retrieve and stitch audio snippets, corresponding to words in an MT sentence, from a spoken vocabulary bank. Our experiments on multiple language pairs show that stitched speech helps to improve translation quality by an average of 1.83 BLEU score, while performing equally well as TTS-generated speech in improving translation quality. We also showcase how SpokenVocab can be applied in code-switching ST for which often no TTS systems exit.

## introduction end-to-end (e2e) speech-to-text translation (st) models require large amounts of data to train (sperber and paulik, 2020). despite the emerging st datasets (cattoni et al., 2021; wang et al., 2021), their size is considerably smaller compared to text-based machine translation (mt) data. a common remedy to tackle the data scarcity issue is to leverage text-based mt data in training st systems. common approaches include multi-task learning (anastasopoulos and chiang, 2018; ye et al., 2021), transfer learning & pretraining (bansal et al., 2019; wang et al., 2020) and knowledge distillation (inaguma et al., 2021; tang et al., 2021). a more straightforward alternative is to convert text-based mt data to st via text-to-speech (tts) synthesis engines (pino et al., 2019; jia et al., 2019). @xcite our code is available at https://github.com/ mingzi151/spokenvocab this method is less commonly used despite its simplicity and effectiveness, foot_0 mainly due to practical reasons: (i) tts models have slow inference time and may incur monetary costs; (ii) the conversion is required for each mt datasets. recently, lam et al. (2022) proposed to generate synthetic speech without using tts models. however, their approach is based on real st data, and thus cannot be extended to mt data. in this work, we propose a simple, effective and efficient data augmentation approach to convert mt data to st data on-the-fly. the idea is to prepare a set of spoken words, forming a spoken vocabulary (spokenvocab) bank, and then generate synthetic speech by retrieving and stitching spoken words based on a text sequence, as shown in figure 1 . @xcite our experiments show that this method is as effective as tts-generated speech, at a much lower computational and financial cost. for instance, augmenting st data on-the-fly with 100k of stitchconverted mt data, boosts translation quality by an average of 1.83 bleu over 3 language pairs from must-c (cattoni et al., 2021) with no additional cost, memory, or speed footprints. comparing the real st data vs. our converted version from the same transcripts, to our positive surprise, revealed that our synthetic data outperforms its real counterpart by 0.41 bleu score. we conduct thorough experiments to examine spokenvocab in boosting translation and further showcase its use and benefit in the context of code-switching (cs) st. we hope this simple technique to ease the use of mt data for st in practice as well as other tasks where synthetic speech is useful. ## spokenvocab we describe our methodology in creating effective synthetic st data based on mt data in this section. the core step is the preparation of a spokenvocab bank offline and stitching sounds on-the-fly. concretely, we first use a tts engine to convert items in a word vocabulary to speech to obtain a set of spokenvocab offline. @xcite next, we can configure the tts engine to generate different speaker voices and thus curate a spokenvocab bank in which each set corresponds to a "speaker". the purpose is to simulate, to the greatest extent, a realistic speech dataset consisting of various speakers. at training, assume we have access to an mt dataset and each pair denoted @xmath0, @xmath1, respectively. given such a pair, we choose one voice foot_3 from the bank, and produce synthetic speech by fetching corresponding audio snippets by words in s from the bank and stitching them together. during stitching, we deploy cross-fade, a well-known technique to smooth transitions between two independent audio clips. @xcite pairing it with t yields a synthetic st instance. @xcite 3 experiments we first present the st system ( §3.1) and tts systems ( §3.1.2) used in this study. we then describe the st and mt datasets ( §3.1.3), followed by providing implementation details ( §3.1.4). next we explain how spokenvocab is designed ( §3.2) and report translation results ( §3.3). lastly, we illustrate how our method can be applied to cs st ( §3.5). ## conclusion in this work, we proposed a simple, fast and effective data augmentation technique, spokenvocab for st. this provides an alternative for converting mt data to st data with tts systems which comes with monetary and computation costs in practice. our approach generates synthetic speech on-thefly during training, with no cost or footprint. we have shown that speech stitched from spokenvocab works as effective as tts-generated speech, and unlike tts system, it could directly be applied as a data augmentation tool in code-switching st. our approach can be used in other content-driven speech processing tasks as an uncompromising and easy-to-use augmentation technique.