Cross-Lingual Transfer Learning for Speech Translation (2025)

Abstract: There has been increasing interest in building multilingual foundation models for NLP and speech research. This paper examines how to expand the speech translation capability of these models with restricted data. Whisper, a speech foundation model with strong performance on speech recognition and English translation, is used as the example model. Using speech-to-speech retrieval to analyse the audio representations generated by the encoder, we show that utterances from different languages are mapped to a shared semantic space. This shared embedding space can then be leveraged for zero-shot cross-lingual transfer in speech translation. By fine-tuning the Whisper decoder with only English-to-Chinese speech translation data, improved performance for translation to Chinese can be obtained for multiple languages, in addition to English. Furthermore, for languages related to those seen in training it is possible to perform speech translation, despite the model never seeing the language in training, or being able to perform transcription.

## introduction speech translation (st) systems directly generate transcriptions in the target language from spoken utterances in a different language and have various applications @xcite . with the growing demand for multilingual models, it is crucial to develop translation systems that support multiple languages, both as source and target. however, data collection for training st systems is more challenging than for neural machine translation (nmt) and automatic speech recognition (asr) tasks. unlike nmt, where the same text corpus can be used for both translation directions @xcite , st systems face challenges due to their asymmetric inputoutput nature. for instance, data for translating audio in language x into text in english (@xmath0, largely due to the higher global demand for english translations. moreover, high-resource language pairs have more available data than low-resource pairs. given the high cost of collecting diverse data pairs for st systems, understanding what is required to build a multilingual st model and expand its capability to more languages is essential. in this work, we use openai's whisper @xcite as a case study to explore the behavior of multilingual speech foundation models. whisper is pre-trained to support speech recognition in 100 languages and translation from 99 languages into english (@xmath1. the encoder can extract semantic information from the acoustic features. we hypothesise that the features in different languages are aligned within a shared semantic space, and this alignment could enable the model to support translation from multiple source languages, a key feature for expanding multilingual st capabilities. whisper's decoder acts as a language model that generates tokens conditioned on the encoder outputs. by supporting multiple languages at the token level, the decoder facilitates translation into various target languages. this flexibility allows us to test and expand its st capabilities to new target languages, which we verify through zero-shot and fine-tuning experiments. in this work, we explore how to extend whisper's capability in speech translation, expanding its supported translation language pairs. first, we evaluate the level of language invariance in the embeddings produced by the whisper encoder using a speech-to-speech retrieval task @xcite . second, we expand the translation to a new target language by fine-tuning whisper, the results show a level of cross-lingual transferability among the source languages. third, we show that whisper can translate spoken utterances from previously unseen languages into english texts, indicating its ability to map unseen languages into a shared speech embedding space. figure 1 : illustration of whisper's decoding process for asr and speech translation tasks. whisper supports speech recognition in 100 languages and speech translation from any language into english (orange (german, de, input) and purple (french, f r, input) text blocks). fine-tuning on english-to-chinese, @xmath2, speech translation data enables the model to acquire additional speech translation capabilities (such as @xmath3. the @xmath4, independent of the target language. unlike text models, whisper's pre-training for speech translation only uses english as the target language. recently, @xcite have demonstrated that whisper exhibits emergent capabilities in unseen speech translation directions through prompt engineering at inference for (@xmath5. in this study, we conduct a more comprehensive investigation into whisper's cross-lingual transferability. whisper's utterance embeddings are not explicitly aggregated, again unlike text models. additionally, speech representations are much longer than text tokens. these differences add to the difficulty of auto-alignment in the speech encoder space. in the speech area, @xcite learned multimodal multilingual speech embeddings by fine-tuning from a pre-trained xls-r model @xcite . they used the labse text encoder model @xcite , which produces aligned embedding spaces across languages, as the teacher model during training. for each given language, the proposed samu-xlsr model generates utterancelevel speech embeddings and is trained to minimize the cosine loss relative to the teacher model's output. through knowledge distillation, the model can produce an aligned speech embedding space. @xcite followed a similar idea to align the space produced by the speech encoder with a pre-trained multilingual text encoder. our work differs in that whisper is not explicitly trained to match a text encoder space; instead, we rely on the speech translation pre-training target to achieve automatic alignment. moreover, whisper generates speech embeddings at a frame-level granularity rather than at the utterance level, enabling more fine-grained representations. ## conclusions this work demonstrates how to extend speech translation capabilities in whisper. whisper's decoder, supporting diverse language tokens, allows for effective expansion to new target languages. our experiments reveal high recall rates in speechto-speech retrieval, indicating that whisper's encoder captures language-invariant features across languages. fine-tuning whisper on english-to-chinese (@xmath6.9 for three other source languages. in addition, whisper can successfully translate speech from some previously unseen languages into english, despite high wers. these results confirm that whisper maps utterances into a shared embedding space, enabling effective cross-lingual transfer for speech translation.