NAVER LABS E urope Submission to the Instruction-following Track (2025)

Abstract: In this paper we describe NAVER LABS Europe submission to the instruction-following speech processing short track at IWSLT 2025. We participate in the constrained settings, developing systems that can simultaneously perform ASR, ST, and SQA tasks from English speech input into the following target languages: Chinese, Italian, and German. Our solution leverages two pretrained modules: (1) a speech-to-LLM embedding projector trained using representations from the SeamlessM4T-v2-large speech encoder; and (2) LoRA adapters trained on text data on top of Llama-3.1-8B-Instruct. These modules are jointly loaded and further instruction-tuned for 1K steps on multilingual and multimodal data to form our final system submitted for evaluation.

## introduction large language models (llms) have demonstrated remarkable success across various text-based natural language processing tasks @xcite @xcite @xcite , motivating research into extending them to other modalities. this has led to the development of multimodal llms capable of processing speech, audio, images and video @xcite @xcite @xcite @xcite @xcite @xcite . this year iwslt instruction-following speech processing track focuses on the leveraging of llms and speech foundation models (sfm) to build solutions capable to perform multilingual tasks from english speech input and textual multilingual instructions @xcite . naver labs europe (nle) * equal contribution. our submitted systems leverage all the available data from the constrained settings, together with data automatically obtained using both backbones. we train two types of systems in parallel: this system paper is organized as follows. section 2 describes the preprocessing applied to the data used in this challenge. sections 3 and 4 describe our training pipeline and experimental settings, respectively. section 5 presents our experiments and discussion. section 6 presents the submitted system. section 7 concludes the paper. ## data for training our models, we leverage the data from the constrained setting: covost2 @xcite , @xcite and spokensquad @xcite . with the agreement of the organizers, we also take advantage of the seamlessm4t-v2-large to produce extra synthetic speech data @xcite and multilingual text data (seamless mt). llama-3.1-8b-instruct is used to rephrase sqa answers. acl 60-60 @xcite is used for validation and evaluation only. we now present our data preprocessing (section 2.1) and prompt format (section 2.2). ## training pipeline our training pipeline is illustrated in figure 1 . we first train a speech projector on speech tasks (a), and text lora weights on textual tasks (b). these modules are then reloaded and adapted together on both speech and textual tasks (c). in this section we describe the key components used and the data sampling strategy. foundation models for speech, we leverage seamlessm4t-v2-large model, extracting speech representations for all our audio data from its 24th speech encoder layer (i.e. the last layer). prior to training, we average every 3 consecutive frame vectors, reducing significantly the sequence length. this simple trick allows us to train our models with larger batches, while maintaining good performance in speech tasks. all our models are built on top of a frozen llama-3.1-8b-instruct. ## experimental settings codebase we train our models using an internal fork of torchtune (torchtune maintainers and contributors, 2024), which allows us to process interleaved representations of text and high-dimensional vectors within the user turn during instruction tuning. the high-dimensional vectors pass through our speech projector, while the text prefix and suffix user prompts are processed by the llm embedding layer. the obtained speech and text embeddings are both concatenated and fed into the first layer of the llm which is trained on the masked input with standard cross-entropy loss. different learning rate schedulers and optimizers are employed for the speech projector and the lora weights, allowing for more controlled and effective training of these distinct model components. inference settings we perform inference using torchtune, with a batch size of 1 and greedy decoding. the maximum number of new tokens was limited to 300. unless explicitly stated otherwise, this decoding strategy was consistently applied across all experimental settings. additional discussion regarding multimodal inference is present in appendix section b.1. ## experiments we now present our results for asr, st, and sqa. section 5.1 introduces the models used in our experiments, followed by results and discussion in section 5.2. ## submitted model therefore, we decided to select the a.1-based model as our primary submission model. for producing the decoding of the test set, we transform the instructions into our own prompt format, and submit the output of the same model, with the same decoding settings for all splits: greedy decoding using 150 as maximum number of tokens. a brief post-submission discussion is provided in appendix section c.2.1. ## conclusion in this paper, we presented nle's submission to the instruction-following speech processing short track at iwslt 2025, constrained setting. we developed multimodal models that simultaneously performed asr, st, and sqa tasks from english speech input into chinese, german and italian. our approach is simple yet effective: we decoupled training, training a speech projector on speech-to-text tasks, and lora adapters on text-to-text tasks. then, both modules are loaded and the resulting multimodal model was instruction-tuned for a few steps on multilingual and multimodal data to produce the final system submitted for evaluation. en de it zh # lines 8,026 3,272 783 627