S imul S eamless: FBK at IWSLT 2024 Simultaneous Speech Translation (2024)

Abstract: This paper describes the FBK’s participation in the Simultaneous Translation Evaluation Campaign at IWSLT 2024. For this year’s submission in the speech-to-text translation (ST) sub-track, we propose SimulSeamless, which is realized by combining AlignAtt and SeamlessM4T in its medium configuration. The SeamlessM4T model is used ‘off-the-shelf’ and its simultaneous inference is enabled through the adoption of AlignAtt, a SimulST policy based on cross-attention that can be applied without any retraining or adaptation of the underlying model for the simultaneous task. We participated in all the Shared Task languages (English->German, Japanese, Chinese, and Czech->English), achieving acceptable or even better results compared to last year’s submissions. SimulSeamless, covering more than 143 source languages and 200 target languages, is released at: https://github.com/hlt-mt/FBK-fairseq/.

## introduction simultaneous speech-to-text translation (simulst) is the task in which a model has to provide a textual translation into the target language while continuously receiving an incremental speech input in the source language. simulst poses additional difficulties to standard offline st, as it has to find the optimal balance between translation quality and output latency, which is the time delay between an utterance being spoken and the corresponding translation being emitted. this balance -often referred to as "qualitylatency tradeoff" -depends on the application scenario @xcite , which can span many domains such as online meetings, lectures, conference talks, and live shows. due to the growing interest in simulst technologies, this task has been included in the iwslt evaluation campaigns 1 since 2020. the increasing interest has led to numerous direct and cascade models participating in the challenge every year @xcite @xcite , all vying for the title of the best approach to realize a simulst system from scratch. more recently, the practice of using models without ad-hoc training for the simultaneous scenario has become widespread @xcite @xcite @xcite , demonstrating that competitive or even superior results can be achieved compared to systems specifically tailored for simulst @xcite . among the strategies used to repurpose standard (offline) st models for simulst @xcite @xcite , alignatt @xcite emerged as the best one, achieving new state-of-the-art results. alignatt exploits speechtranslations alignments based on cross-attention scores to guide the simultaneous inference, overcoming the limitations of the previous approach relying on attention @xcite . alongside the increased interest in the simulst task, especially during the last year, we have witnessed an explosion in the use of large models @xcite , including speech foundation models @xcite @xcite . these models are now commonly used alone or in combination with large language models @xcite for generic st tasks. among these, seam-lessm4t @xcite ) has emerged as one of the most promising multimodal and multilingual models, covering more than 143 source languages and 200 target languages. for this year's submission to the iwslt evaluation campaign on simultaneous translation, we, therefore, propose to combine the best of both 1 https://iwslt.org/ 1 120 worlds to obtain a multilingual model without any training or adaptation for the simulst task. this results in simulseamless, consisting of the seam-lessm4t model used "off-the-shelf" repurposed for simultaneous inference using alignatt. from empirical results on the task, we show that simulseamless can achieve acceptable or even better results compared to last year's participants, despite not being retrained or fine-tuned either for the simultaneous task or on paired data in the evaluated languages. moreover, simulseamless is a generic multilingual model that can be used for any allowed translation direction supported by the underlying seamlessm4t model, covering more than 143 source languages and 200 target languages. the code is released under the apache 2.0 licence at https://github.com/hlt-mt/fbk-fairseq/ blob/master/fbk_works/simulseamless.md . ## simulseamless similarly to previous years @xcite , we participated in the simultaneous translation evaluation campaign, focusing on the speech-to-text translation sub-track. for this year's submission, we opted for the use of the new seam-lessm4t model, which is allowed for the task, foot_0 as the underlying model of the simulst policy alig-natt. this policy can be applied to any standard (i.e., offline-trained) model without the need for retraining or adaptation. in the following, both these elements and their combination are explained in detail. seamlessm4t. seamlessm4t @xcite (or massively multilingual & multimodal machine translation) is a family of models based on pre-trained models including w2v bert 2.0, and nllb @xcite , whose encoder and decoder respectively are used for the speech-to-text modality. w2v-bert is a conformer-based model @xcite composed of 24 layers, with a total of ∼600m parameters, and trained on 1 million hours of open speech audio data to learn self-supervised speech representations. it processes the audio features obtained by applying 80-dimensional mel filterbanks to the audio waveform. the w2v-bert encoder is followed by a length adapter based on a modified version of the m-adaptor @xcite , which is a transformer-based model @xcite that is in charge of compressing the speech representation (by a factor of 8) through attention pooling. the compressed input representations are then fed to the nllb decoder, in its 1.3b parameters configuration, to produce the translations. the final model was obtained after training on both manual and automatically aligned speech translation data with a total of 406,000 hours. alignatt. alignatt @xcite is a simulst policy that relies on cross-attention to make decisions about whether to emit translated words or wait for additional information in the simultaneous scenario. at each time step, the cross-attention scores are exploited to obtain audiotranslation alignments by uniquely assigning the predicted words to the audio frames (encoder states) having the maximum attention score. then, it is checked, for each word, if it has been aligned with one of the last f frames, which is the parameter handling the latency of the model. if this is true, the emission is stopped, otherwise, the next word is evaluated. the idea behind alignatt is that, if a word is aligned with one of the last received audio frames, the encoded information could be unstable and/or not sufficient to reliably predict that word. conversely, if a word mostly attends to a more stable and earliest-received encoded information, it can be safely predicted. with this formulation, alignatt simplifies the previous edatt policy @xcite ) by eliminating the dependency on additional hyper-parameters while achieving com-2 121 petitive or even better results. seamlessm4t + @xmath0. since alignatt is applicable to any standard st models without the need for re-training or adaptation, we chose to apply it directly to the seam-lessm4t model in its medium configuration, realizing simulseamless. this solution is completely different from seamlessstreaming @xcite , which is obtained through an expensive ad-hoc finetuning of the seamless model for the simultaneous task based on emma -efficient monotonic multi-head attention @xcite . since seamlessm4t already covers all the languages evaluated in the simultaneous track, the model is used completely "off-the-shelf". the simulseamless model is shown in figure 1 . ## experimental settings we used the available checkpoint of the seam-lessm4t model provided on huggingface in its "medium" configuration, foot_1 with a total of 1.2b parameters. the results are reported on the benchmarks used for the submission, which is must-c @xcite v2.0 tst-common for en-{de, ja, zh}, and the dev set provided for the task for cs-en. the scores are computed using the simuleval toolkit @xcite . 4 translation quality is evaluated using bleu score with sacrebleu @xcite 5 . latency is reported using average lagging (al) @xcite since it is the metric used for the final scoring. length adaptive average lagging (laal) @xcite and average token delay (atd) @xcite are also evaluated and included in the final results since they are official metrics reported for the task. 6 both latency and bleu scores are computed at the character level for chinese and japanese while the standard 13a tokenizer is used for sacrebleu, and word-level latency is computed for the other languages. additionally, computationally aware metrics are presented to account for the real elapsed time, which also considers the computational cost of running the underlying model. the inference was run using a single gpu nvidia v100 with 16gb of ram. 1 2 3 4 5 6 7 8 9 10 11 12 27 28 layer bleu en-de (a) english to german 1 2 3 4 5 6 7 8 9 10 11 12 20 21 22 23 layer bleu en-zh en-ja (b) english to chinese and japanese 1 2 3 4 5 6 7 8 9 10 11 12 17 18 19 layer bleu cs-en (c) czech to english figure 3: translation quality (bleu↑) scores of simulseamless on must-c v2.0 tst-common for english (en) to german (de), japanese (ja), and chinese (zh), and on the iwslt 2024 dev set for czech (cs) to english by varying the decoder layer from which crossattention scores are extracted from. for the alignatt policy, we set the size of the speech chunk processed by the model at each time step to 1s for english to german and czech to english, 800ms for english to chinese, and 400ms for english to japanese. to achieve latency close to an al of 2s required for the submission, we set 3 122 1 2 3 4 5 6 7 8 9 10 11 12 2 1.8 1.9 2.1 2.2 2.3 layer al (s) en-de en-zh en-ja cs-en figure 4: latency (al↓) scores of simulseamless on must-c v2.0 tst-common for english (en) to german (de), japanese (ja), and chinese (zh), and on the iwslt 2024 dev set for czech (cs) to english by varying the decoder layer from which cross-attention scores are extracted from. the hyper-parameter handling the latency f to 1 for en-ja and en-zh, 6 for en-de, and 9 for cs-en. the cross-attention scores are normalized framewise before applying alignatt to avoid the crossattention weights being skewed to some frame representation, as shown in figure 2 . ## conclusions we introduced fbk's system designed for participation in the iwslt 2024 evaluation campaigns in simultaneous translation and, specifically, the speech-to-text sub-track (simulst). our submission is characterized by the "off-the-self" use of the seamlessm4t model for direct speech translation, repurposed for the simultaneous scenario by means of alignatt. alignatt is a simulst policy that leverages cross-attention scores to guide simultaneous inference without any further modification or adaptation of the underlying model. the combination of seamlessm4t and alignatt results in simulseamless, which supports all translation pairs of the evaluation campaign (english to german, japanese, and chinese, and czech to english). simulseamless, to be released upon paper acceptance, achieves acceptable or even superior results compared to last year's participants. moreover, it can be used for any language pairs enabled by the underlying seamlessm4t model, potentially covering more than 143 source languages and 200 target languages.