Leveraging Synthetic Audio Data for End-to-End Low-Resource Speech Translation (2024)

Abstract: This paper describes our system submission to the International Conference on Spoken Language Translation (IWSLT 2024) for Irish-to-English speech translation. We built end-to-end systems based on Whisper, and employed a number of data augmentation techniques, such as speech back-translation and noise augmentation. We investigate the effect of using synthetic audio data and discuss several methods for enriching signal diversity.

## introduction resource scarcity and the scattered nature of the data are crucial challenges for low-resource languages @xcite @xcite . in this sense, irish is considered a low-resource language and significantly lacking in speech and language tools and resources @xcite . researchers have been employing various data augmentation techniques to improve the quality of low-resource textual machine translation (mt) systems. among these techniques is using synthetic data generated by back-translation @xcite @xcite @xcite , or large language models @xcite . similarly, in the area of speech, @xcite showed that models trained solely on synthetic audio datasets can generalize their performance to human voice data. nevertheless, @xcite revealed a consistent decrease in the diversity of the outputs of language models trained on synthetic textual data. we observe that leveraging synthetic audio data generated by text-to-speech (tts) models can be beneficial for training speech translation models, especially for low-resource languages. however, it can lack the diversity found in authentic audio signals in terms of pitch, speed, and background noise. speech translation systems can be cascaded systems or end-to-end systems @xcite . cascaded systems use two models, one for automatic speech recognition (asr) and one for textual machine translation (mt). end-to-end speech translation systems use one model for the whole process; hence, it is more challenging. in this work, we present end-to-end speech translation models. in addition to describing our system submitted to iwslt 2024, this work presents the following contributions: ## authentic data the organizers of the iwslt shared task, provided the iwslt-2023 dataset, which consists of training, dev, and test portions. we used both the training and dev portions for training, and the test portion for evaluation. we also used the irish portion of the fleurs datasets. moreover, we employed the bilingual audio-text data available at the bitesize website for teaching irish. ## synthetic data this section explains diverse approaches for creating synthetic data for speech translation. we describe each approach, as well as its advantages and disadvantages. ## experiments our experiments fine-tune whisper @xcite for the task of irish-to-english speech translation. we experiment with a number of data augmentation techniques, such as speech back-translation (source-side synthetic audio data generation), and audio data augmentation with noise and vad. ## evaluation and results to evaluate our systems, we calculated bleu @xcite , chrf++ @xcite , and ter @xcite , as implemented in the sacrebleu library 9 @xcite . for semantic evaluation, we used an embedding-based approach, calculating and comparing cosine similarity between the vector embeddings of each reference and the equivalent translation generated by the model. we report the average of semantic similarity across utterances. we used two models with sentence-transformers @xcite , "allmpnet-base-v2" (semantic 1) and "all-minilm-l12-v2" @xcite (semantic 2). as we fine-tuned all the models for approximately two epochs, we report the evaluation of the best performing checkpoint. for inference, we used faster-whisper 10 with the default vad arguments. we also compared the results without vad, and found that applying vad at inference time is better for all the models (cf. appendix a). we used 5 for "beam size" and 2 for "no repeat ngram size". as model c uses the same training data as model b while the choice of augmentation techniques were based on manual observation of the characteristics of the authentic data and the synthetic data, the achieved improvements encourage further investigation. in the future, we would like to conduct more experiments that employ other data augmentation techniques. moreover, we would like to measure the effect of adding synthetic audio data compared to augmenting the authentic data only. finally, as the main purpose of this research is to understand the best practices of using synthetic audio data (i.e. data generated by tts models) to improve speech translation quality, we will conduct further study on mimicking authentic data characteristics to enhance the effect of data augmentation with synthetic audio data.