CMU ’s IWSLT 2024 Simultaneous Speech Translation System (2024)

Abstract: This paper describes CMU’s submission to the IWSLT 2024 Simultaneous Speech Translation (SST) task for translating English speech to German text in a streaming manner. Our end-to-end speech-to-text (ST) system integrates the WavLM speech encoder, a modality adapter, and the Llama2-7B-Base model as the decoder. We employ a two-stage training approach: initially, we align the representations of speech and text, followed by full fine-tuning. Both stages are trained on MuST-c v2 data with cross-entropy loss. We adapt our offline ST model for SST using a simple fixed hold-n policy. Experiments show that our model obtains an offline BLEU score of 31.1 and a BLEU score of 29.5 under 2 seconds latency on the MuST-C-v2 tst-COMMON.

## introduction this paper presents cmu's submission to the iwslt 2024 @xcite simultaneous speech translation (sst) task, focusing on streaming english speech to german text translation. recent advancements in large language models (llms) have demonstrated their potential to be a strong backbone for offline st @xcite . in this year's submission, we build an end-to-end offline st model with wavlm @xcite and llama2-7b-base @xcite following the practice of lst @xcite . then we adapt the offline model for simultaneous translation. we prepare our end-to-end st model in the following steps: ## system description as shown in figure 1 , our offline st models consists of three primary components: a speech encoder, an adapter, and a llm decoder. for the speech encoder, we employ the wavlm model foot_1 , which has been pre-trained on 94,000 hours data including librilight @xcite , voxpopuli @xcite and gigaspeech @xcite . we use the output of last encoder layer as the speech representation. the modality adapter consists of two components: a length adapter and a modality adapter. the length adapter consists of two 1-dimensional convolutional layers with a kernel size of 5, a stride size of 2, padding of 2, and a hidden size of 1024. the modality adapter is a linear layer that projects the output of the length adapter to the embedding space of llm. we use llama2-7b-base as the llm decoder. the llm decoder takes the output of the modality adapter and autoregressively generate the target text translation. ## experimental setup we use the adamw optimizer with a cosine learning rate decay and a warmup ratio of 0.2. the learning rate commences at 2e-4 for the first training stage and is reduced to 2e-5 for the second stage. we train the first stage for 6 epochs and train the second stage for 1 epoch. we employ an early stopping strategy with a patience of 6 epochs, evaluating every 1000 steps in stage 1 and every 200 steps in stage 2. the batch size is set to 128 for both stages. all models are trained on 4 nividia a6000 gpus with deepspeed's zero training strategy. the training times for the first and second stages are approximately 29 hours and 9 hours, respectively. we select the checkpoints with the lowest dev loss for testing. for offline testing, we use a beam size of 4 to generate translations. in the simultaneous testing scenario, we set the start seconds to 2, indicating the initial wait time before processing speech chunks. we employ a hold-n strategy with n set to 7, meaning that the last 7 tokens of each chunk are withheld until more context is available. the beam size is set to 4, and the chunk size is set to 2500ms. we evaluate translation quality using sacre-bleu @xcite . we evaluate translation latency for sst with average lagging (al) @xcite and length-adaptive average lagging (laal) @xcite using simuleval toolkit @xcite . ## results from st to sst, we observe a 5% quality degradation (31.1 to 29.5 sacrebleu). however, this comes with significant latency improvements. the average lagging (al) decreases from 5.85 to 1.96 seconds, a 66.5% reduction. the length adaptive average lagging (laal) improves from 5.85 to 2.22 seconds, a 62.1% decrease. we also investigate the impacts of different llms and speech encoders, as shown in our analysis of the performance between different llms used as decoders shows that the tower llm foot_4 , subjected to continued pre-training on a curated multilingual dataset of 20 billion high-quality tokens, exhibits a marked performance advantage over llama2 in the initial stage of training. however, during the second stage, when the llm backend is trainable, tower quickly overfits, implying potential overlap between the must-c corpus and the data involved in tower's pretraining. tower ## conclusion in this paper, we describe the submission of cmu's english to german simultaneous speech-to-text translation systems for the iwslt 2024 simultaneous track. we start by building a offline speech-totext system which leverages self-supervised speech and text foundation models. we then adapt this offline model for streaming inference, enabling simultaneous speech-to-text translation.