Finetuning End-to-End Models for E stonian Conversational Spoken Language Translation (2024)

Abstract: This paper investigates the finetuning of end-to-end models for bidirectional Estonian-English and Estonian-Russian conversational speech-to-text translation. Due to the limited availability of speech translation data for Estonian, we created additional training data by web scraping and synthesizing data from speech recognition datasets using machine translation. We evaluated three publicly available end-to-end models: Whisper, OWSM 3.1, and SeamlessM4T. Our results indicate that fine-tuning with synthetic data enhances translation accuracy by a large margin, with SeamlessM4T matching or surpassing cascaded speech translation systems that use state-of-the-art speech recognition and machine translation models.

## introduction estonian language, spoken by around one million native speakers, has benefited significantly from the estonian language technology program in the last decades @xcite . this initiative has fostered advancements in several key areas, such as automatic speech recognition (asr) @xcite and machine translation (mt) @xcite . these improvements are largely due to investments in collecting relevant training data and the successful application of large multilingual pretrained models. another crucial area of language technology is spoken language translation, which is essential for maintaining smaller languages like estonian in today's digital world. this technology enables native speakers of a small language to access foreign language content more easily and allows for the broader dissemination of native language content. however, one of the significant challenges in developing these technologies is the lack of adequate training data for estonian, particularly in conversational speech. this shortage hampers the ability to further enhance and refine speech translation tools. in this study, we explore the finetuning of three publicly available end-to-end models for bidirectional estonian-english and estonian-russian conversational speech translation tasks and evaluate their accuracy against the cascaded spoken language translation approach. given the scarcity of speech translation datasets containing significant amounts of conversational speech for these translation directions, we explore two methods to generate additional data: synthesizing speech translation training data from asr training data using machine translation, and scraping data (e.g., videos with subtitles) from the internet. we evaluate these models and finetuning approaches using automatic metrics (bleu and bleurt) on realistic conversational speech evaluation sets. the main contribution of this paper is demonstrating that leading large publicly available end-toend multilingual speech translation models can be fine-tuned to excel in translation tasks involving relatively low-resource languages by using synthetic data generated from diverse asr training data. another innovative aspect of the paper is showing that openai's whisper, originally trained only for translating into english, serves as an effective base model that can be finetuned for other speech translation directions. additionally, we release an evaluation set for estonian-english-russian spoken language translation, which includes conversational speech recordings "from the wild", complete with manual transcripts and professionally produced translations foot_0 . the best-trained speech translation models are publicly available foot_1 . an example of an estonian tv news broadcast with english and russian subtitles generated by our finetuned whisper model is available at https: //www.youtube.com/watch?@xmath0. ## available models in this section, an overview of publicly accessible models suitable for speech translation in the targeted translation directions of our study will be provided. ## methodology the main focus of our work is finetuning publicly available speech translation models using additional data. since there are no conversational speech translation datasets that include estonian, we experiment with generating additional data on our own using two methods: web scraping and data synthesis. we compare the performance of all three existing speech translation models before and after finetuning with the same data. although whisper is originally trained to perform only multilingual speech recognition and speech translation to english, it has been shown that it can perform speech translation to other directions with surprisingly high accuracy by changing only the prefix of the decoder. for example, @xcite showed that by only modifying the prompt, whisper can achieve 18.1 bleu score on the english-german speech translation test set from the must-c corpus @xcite . therefore, we were relatively confident that whisper can be finetuned for all translation directions that we were interested in. the design of whisper's prompt does not support the specification of alternative translation directions. consequently, we finetuned whisper using extra speech translation data by employing the "transcribe" prompt, where the language specified in the prompt matched the intended target language. at the inference stage, the expected target language was set in the prompt, but the source language remained unspecified to the model. on all datasets, whisper was finetuned foot_4 for three epochs over the additional translation datasets. a learning rate schedule with a peak rate of 1e-04 was used, with 500 warmup steps and a linearly decaying schedule towards 0 after the warmup. an effective batch size of 64 was used. stochastic weight averaging (swa) @xcite with a learning rate of 1e-05 was applied during the last epoch. adam optimizer was used. the owsm 3.1 ebf model underwent finetuning over five epochs, utilizing a batch size of 320 and a maximum learning rate of 2.0e-04, accompanied by a warmup phase of 600 steps. a label smoothing technique was employed with a smoothing factor of 0.1. during training, a multitask encoder-decoder/ctc loss method was used (with source language transcript as supervision for the ctc head), setting the ctc loss weight at 0.3. the majority of these hyperparameters were adopted directly from the espnet's training recipe for the owsm 3.1 ebf model without further adjustments. the seamlessm4t model was finetuned using a batch size of 48, peak learning rate of 1e-06 with 100 warmup steps. this finetuning setup integrated automatic early stopping that measured the model's loss on heldout training data after every 1000 model updates and stopped training when the loss didn't improve during the last 10 evaluations. this usually happened during the second epoch. for whisper and owsm, the training data was compiled to segments of maximally 30 seconds in length, which usually involved concatenating the transcripts of several adjacent utterances from the long-form training audio, together with the corresponding audio chunks (including the audio between transcription end and start times). the seam-lessm4t model was finetuned using the original utterances and/or subtitle segments. all finetuning experiments were conducted using four nvidia a100 (80gb) gpus.