T -Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation (2022)

Abstract: We present a new approach to perform zero-shot cross-modal transfer between speech and text for translation tasks. Multilingual speech and text are encoded in a joint fixed-size representation space. Then, we compare different approaches to decode these multimodal and multilingual fixed-size representations, enabling zero-shot translation between languages and modalities. All our models are trained without the need of cross-modal labeled translation data.Despite a fixed-size representation, we achieve very competitive results on several text and speech translation tasks. In particular, we significantly improve the state-of-the-art for zero-shot speech translation on Must-C. Incorporating a speech decoder in our framework, we introduce the first results for zero-shot direct speech-to-speech and text-to-speech translation.

## introduction most, if not all, current state-of-the-art text and speech translation systems are based on a sequenceto-sequence approach and an attention mechanism to connect the encoder and decoder. such models require labeled data to be trained end-to-end. for text-to-text (t2t) translation, this labeled data, called bitexts, is available in large amounts for a number of language pairs, in particular since large-scale bitext mining initiatives like paracrawl @xcite and ccmatrix @xcite . finding training data for speech-to-text (s2t) translation is more challenging, but several data collection efforts exist, like mtedx @xcite , covost @xcite , and must-c @xcite . finally, speech-tospeech (s2s) translation suffers from scarcity of end-to-end labeled data and current s2s systems are limited to a very small number of language pairs. very recent works start to consider mining labeled data for s2s, e.g. @xcite . unsupervised representation learning is very successfully used to initialize the encoder and/or decoder of a sequence-to-sequence model, thereby lowering the amount of labeled data needed to train or fine-tune the model end-to-end. approaches include for instance xlm @xcite , xlsr @xcite , wav2vec @xcite , data2vec @xcite and mslam @xcite . in this work, we propose a new modular architecture for text and speech translation, which is based on a common fixed-size multilingual and multimodal internal representation, and encoders and decoders which are independently trained. we explore several variants of teacher-student training to learn text and speech encoders for multiple languages, which are compatible with the embedding space of the laser encoder @xcite . in contrast to preceding works on multilingual and multimodal representations, we also train text decoders for multiple languages which are able to generate translations given the joint representation. finally, we demonstrate that it is possible to train a speech decoder using raw audio only. figure 1 visualizes the overall approach. we show that these encoders and decoders can be freely combined to achieve very competitive performance in t2t, s2t and (zero-shot) s2s translation. in summary, our contributions are as follows. multilingual and multimodal representations building multilingual representation for text or speech is key to develop state-of-the-art models based on these modalities. conneau and lample (2019) introduce a multilingual pre-training method with good cross-lingual transfer capabilities. conneau et al. (2020) extend the wav2vec2 (baevski et al., 2020) architecture to the multilingual setting introducing a multilingual pre-trained model for speech. more recently, bapna et al. @xcite pre-train a multilingual encoder model handling both speech and text in order to benefit from cross-modal transfer between speech and text. an important obstacle to good joint speech/text representations is the length mismatch between audio and text. on the other hand, several works have studied how to encode sentences in a fixedsize representation @xcite @xcite . in the multilingual setting, these works highlight that paraphrases and translations are close in the sentence embedding space, enabling large-scale bitext mining. recently, @xcite extended the existing laser model @xcite built for multilingual text to the speech modality for several spoken languages. they show that this joint speech/text fixed-size representation can be efficiently used for large-scale mining of speech against text and even speech against speech. ## exploring training strategies the purpose of this work is to build a common fixed-size representation for multilingual speech and multilingual text that can be decoded in text and speech in different languages. we want to build language-specific encoders and decoders compatible with this fixed-size representation. plugging one encoder with one decoder from different modalities and/or different languages enables performing zero-shot cross-modal translation. to this end, we first study how to efficiently decode fixed-size sentence representation for text. second, we study how to improve similarity for sentence embeddings between languages. after an ablation study on the japanese-english text translation direction, we extend the best training strategy to several other languages and a new modality, speech. ## overall architecture text student encoders we now want to train several text students for different languages, in order to plug, at test time, these encoders to different decoders to perform translation tasks. we decide to use laser english embeddings as our teacher. this english space has proven to have good semantic properties: paraphrases are close in the embedding space, and makes it a good teacher for english translations. moreover, most of mt data involve english translations that we will use to learn our text students. we focus on 7 languages, namely, german, french, spanish, catalan, japanese, turkish, and mongolian. we use ccmatrix bi-texts to learn our text students, and bi-texts mined with laser3 @xcite for mongolian. ## results and discussion text-to-text translation as presented in section 4, we test different strategies to train an english decoder. when training a decoder with raw text data, we use 15 billion english sentences extracted from ccnet @xcite . when training with additional bi-text data, we use bi-texts from ccmatrix @xcite , and the english part of the bi-texts for the auxiliary auto-encoding loss in order to have a good balance between bitexts and raw data. we present the results for textto-text translation for xx-en directions in in finally, we trained decoders for german, french, spanish, turkish and mongolian in order to be able to translate from any of our languages to any other. a detailed analysis of the translation tasks with these new decoders can be found in the appendix. similar to what we noticed with our english decoder, we obtain excellent zero-shot crosslingual transfer: the german decoder has never seen japanese embeddings before and japanese has never been aligned to german. however, the ja-de results are competitive compared to state-of-the-art translation models trained in an end-to-end way with much more data. speech-to-text translation then, we tried to plug the decoders trained on text data to our speech encoders in order to perform zero-shot speech-totext translation. we trained independent speech student encoders for german, french, turkish, japanese and mongolian spoken languages on the covost 2 training set. for catalan and spanish, we trained a single speech student encoder for both languages as they have high language similarity. we report direct speech translation results in in our framework, the de-en speech translation direction benefits from cross-modal transfer while all other directions benefit from both cross-modal and cross-lingual transfer as the decoder has been trained on text and has only seen english and german embeddings. in this zero-shot cross-modal setting, we notice that the results are really competitive compared to supervised baselines trained end-to-end. moreover, the supervised baselines use speech translation data, whereas our approach does not need speech translation data but only transcriptions. except for turkish, which has a really different morphological structure compared to english, speech translation results are close to their supervised counterpart trained with xlsr. an interest- ## conclusion in this work, we studied how to build a common fixed-size representation for text and speech in different languages, to perform zero-shot cross-modal translation. by imposing a fixed-size representation and aligning explicitly languages and modalities, we have overcome the sentence length mismatch between audio and text, and obtained multilingual and multimodal representations compatible with decoders trained on other languages and/or modalities in a zero-shot setting. we were able to build text and speech encoders for multiple languages compatible with text decoders for multiple languages as well as an english speech decoder. our zero-shot cross-modal translation results for direct speech-totext, text-to-speech and speech-to-speech translation define a new zero-shot state-of-the-art baseline. to the best of our knowledge, this is the first work tackling zero-shot direct text-to-speech and speechto-speech translation. finally, we highlighted the modularity of our architecture; all type of data can be used to train decoders (unlabeled text or speech data ; t2t, s2t, s2s translation data; speech transcription data). using more types of training data may further enhance the robustness of the decoder to other languages or other modalities.