Improving the Quality of IWLST 2024 Cascade Offline Speech Translation and Speech-to-Speech Translation via Translation Hypothesis Ensembling with NMT models and Large Language Models (2024)

Abstract: This paper presents HW-TSC’s submission to the IWSLT 2024 Offline Speech Translation Task and Speech-to-Speech Translation Task. The former includes three translation directions: English to German, English to Chinese, and English to Japanese, while the latter only includes the translation direction of English to Chinese. We attend all three tracks (Constraint training, Constrained with Large Language Models training, and Unconstrained training) of offline speech translation task, using the cascade model architecture. Under the constrained training track, we train an ASR model from scratch, and then employ R-Drop and domain data selection to train the NMT model. In the constrained with Large Language Models training track, we use Wav2vec 2.0 and mBART50 for ASR model training initialization, and then train the LLama2-7B-based MT model using continuous training with sentence-aligned parallel data, supervised fine-tuning, and contrastive preference optimization. In the unconstrained training track, we fine-tune the whisper model for speech recognition, and then ensemble the translation results of NMT models and LLMs to produce superior translation output. For the speech-to-speech translation Task, we initially employ the offline speech translation system described above to generate the translated text. Then, we utilize the VITS model to generate the corresponding speech and employ the OpenVoice model for timbre cloning.

## introduction recent advances in deep learning allow us to address traditional nlp tasks in a new and significantly different manner. one such task is speech translation, involving automatic speech recognition (asr) @xcite system and machine translation (mt) @xcite system. another task is speech-to-speech translation (s2s), which involves asr system, mt system, and textto-speech (tts) @xcite system. recent trends tend to utilize a single neural network to directly translate input speech from one language to text or speech in another language, bypassing intermediate symbolic representations. the results shows that the performance of end-to-end models is nearing that of cascade solutions, but the effectiveness comparison between the two technologies remains unclear. both methods face specific challenges. the primary challenge with the end-to-end approach is the lack of training data, while the cascade method has to go through the asr, mt and even tts processes, leading to the errors accumulation. due to the data insufficiency in end-to-end training, we ultimately chose the cascade approach on the iwslt 2024 offline speech translation task and speech-to-speech translation task. for the iwslt offline speech translation task, we apply different training strategies across the three tracks, adapting to diverse data and model conditions. in the constrained training track, we initiate training with an asr model from scratch, followed by the utilization of r-drop @xcite and domain data selection @xcite techniques to train the nmt model. within the constrained with large language models (llms) training track, we commence asr model training initialization using wav2vec 2.0 @xcite and mbart50 @xcite . subsequently, we train the llama2-7b-based @xcite mt model through continual pre-training with sentence-aligned parallel data @xcite , supervised fine-tuning @xcite , and contrastive preference optimization (cpo) @xcite . in the unconstrained training track, we fine-tune the whisper model @xcite for speech recognition, and then ensemble @xcite the translation outputs of nmt models and llms to generate superior translation result. for the iwslt s2s translation task, we initially employ the offline speech translation system described above to generate the translated text. next, we utilize the vits @xcite model to generate the corresponding speech and employ the openvoice @xcite model for timbre cloning. in comparison to last year, our cascade offline speech translation system and s2s translation system is performing significantly better, particularly following translation hypothesis ensembling with nmt models and llms. ## tts model several recent end-to-end tts models enabling single-stage training and parallel sampling have been proposed, but their sample quality does not match that of two-stage tts systems. vits @xcite is a parallel end-to-end tts method that generates more natural sounding audio than current two-stage models. the method adopts variational inference augmented with normalizing flows and an adversarial training process, which improves the expressive power of generative modeling. in the s2s translation system, we first use the speech translation system to generate the translation text, and then use the vits model to generate the corresponding speech. to improve the similarity of synthesized audio's timbre to that of the source language audio, we also use openvoice @xcite model for timbre cloning. it is a versatile voice cloning approach that requires only a short audio clip from the reference speaker to replicate their voice and generate speech in multiple languages. ## experiments and results the only difference between our s2s translation system and speech translation system is the addition of tts and timbre cloning modules. since we did not perform additional training on these two modules, we only present the experimental results of the speech translation system. we utilize the open-source fairseq @xcite for training the nmt model, the open-source alma @xcite for fine-tuning llm model. we assess the asr models using the word error rate (wer) and evaluate the mt models using case-sensitive sacrebleu @xcite and comet scores. our asr system is evaluated on the test sets of tst-com, while our mt system is evaluated on the test sets of tst-com and tst2022. the cascade system based on the nmt model perform better in the bleu metric, while the cascade system based on the llm model perform better in the comet metric. when ensembling the translation results of both nmt and llm, the cascade system is performing well in both bleu and comet. ## conclusion this paper presents our cascade speech translation system and s2s translation system in the iwslt 2024 evaluation. we try several asr model training strategies and achieve good performance. for the mt system, we explore two research directions based on nmt and llm, and enhanced them through various technical means. finally, we achieve further improvements by ensembling the translation results of nmt models and llms. for the tts, we directly use open source models to generate speech and timbre clones. our experimental results show that llm-based asr and mt are promising research directions.