Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation (2022)

Abstract: End-to-end Speech Translation (ST) aims at translating the source language speech into target language text without generating the intermediate transcriptions. However, the training of end-to-end methods relies on parallel ST data, which are difficult and expensive to obtain. Fortunately, the supervised data for automatic speech recognition (ASR) and machine translation (MT) are usually more accessible, making zero-shot speech translation a potential direction. Existing zero-shot methods fail to align the two modalities of speech and text into a shared semantic space, resulting in much worse performance compared to the supervised ST methods. In order to enable zero-shot ST, we propose a novel Discrete Cross-Modal Alignment (DCMA) method that employs a shared discrete vocabulary space to accommodate and match both modalities of speech and text. Specifically, we introduce a vector quantization module to discretize the continuous representations of speech and text into a finite set of virtual tokens, and use ASR data to map corresponding speech and text to the same virtual token in a shared codebook. This way, source language speech can be embedded in the same semantic space as the source language text, which can be then transformed into target language text with an MT module. Experiments on multiple language pairs demonstrate that our zero-shot ST method significantly improves the SOTA, and even performers on par with the strong supervised ST baselines.

## introduction end-to-end speech translation (st) aims at designing a single model to directly learn the mapping between source language speech and target language text, and has attracted much attention recently due to its advantages of no error propagation and lower decoding latency @xcite @xcite . however, the training of end-to-end models requires large-scale and high-quality parallel st data, which are expensive and difficult to obtain. public datasets such as @xcite and covost @xcite are quite limited in scale and languages. in contrast, the datasets for automatic speech recognition (asr) and machine translation (mt) are easier to access in practice. therefore, zero-shot st, which learns an end-to-end model using only asr and mt data, is a direction worth exploring. as illustrated in figure 1 , the key of zero-shot st is to learn an appropriate shared semantic space for source language speech and text, after which the model can translate from the common space using an mt module without relying any st data. there are some attempts to achieve zero-shot st using multi-task learning that implicitly aligns speech and text @xcite . due to the lack of supervised objective functions for cross-modal alignment, the speech and text representations cannot be well aligned in these methods and the results lag behind supervised settings by a significant margin. inspired by recent research on learning shared semantic space for speech and text @xcite @xcite , we aim to design a supervised cross-modal alignment task that explicitly maps speech and text into a common feature space. however, our preliminary study indicates that the existing cross-modal alignment methods that bridge the gap between speech and text in a continuous space do not work well in the zero-shot st task because in this condition, it is difficult to completely align two modalities to the same distribution in a high-dimensional continuous space. to address this issue, we propose a novel discrete cross-modal alignment (dcma) method that employs a shared discrete vocabulary space to accommodate and match both modalities of speech and text. with the shared discrete vocabulary across the two modalities, the source language speech and the corresponding text are mapped to the same virtual token, ensuring representational consistency. specifically, the st model consists of a speech encoder and a text decoder. we introduce a vector quantization module between the speech encoder and the text decoder to discretize the continuous representations into a finite set of virtual tokens. the asr data are used to provide supervision that maps the speech and its corresponding text into the same virtual token of the shared codebook. in addition to the vector quantization module, the speech encoder is decoupled into an acoustic encoder and a semantic encoder. a shared memory module following the speech encoder is introduced to project variable-length input features of both source language speech and text into fix-sized ones. machine translation is jointly trained to learn the mapping between the fix-sized features on the source side and the text on the target side. to further enhance the speech encoder, masked language model (mlm) and connectionist temporal classification (ctc) are employed as auxiliary tasks in which all parameters are shared. experimental results on the benchmark dataset must-c demonstrate that our discrete alignment method can significantly improve the performance of zero-shot speech translation. the contributions of this paper are as follows: cross-modal alignment in end-to-end st cross-modal alignment aims at aligning representations of speech and text to extract common features. some recent works have pointed out that the representation gap between speech and text is a major obstacle to speech translation. there are many proposals to bridge the gap, including introducing an alignment task @xcite @xcite , mixup strategy @xcite and multimodal pretraining @xcite @xcite @xcite . additionally, some more sophisticated modules such as adaptive feature selection @xcite , shrink mechanism @xcite and shared memory module @xcite have been proposed to address the length inconsistency problem. these methods align the representations of two modalities in continuous feature space. although these methods work effectively in supervised settings, our preliminary study indicates that semantic encoder ② shared memory module ④ text decoder embedding i love you ich liebe dich text input speech input contextual feature (speech) contextual feature (text) semantic memory (text) semantic memory (speech) mlm loss ctc loss mt nll loss wav2vec2 cnn acoustic encoder 1. m. m. … … align loss distribution across codebook (text) distribution across codebook (speech) figure 2 : overview of our model dcma. the speech encoder is decoupled into acoustic encoder and semantic encoder. we adopt a shared memory module to project feature sequence into a fixed length and a vector quantization module to perform cross-modal alignment in discrete space. the text decoder is used to translate from the discrete common space output by vector quantization module. the part above the dashed line illustrates the data flow of the mt task from source language text into target language text. the asr data is employed to design the cross-modal alignment loss. the data flow of the bottom part can be used to perform the st task. the alignments in continuous space do not work well under zero-shot scenario because it is difficult to match two continuous distributions without strong supervision. ## conclusion in this paper, we propose a novel alignment method dcma to enable zero-shot st. the key part of our approach is to discretize the continuous vectors to a finite set of virtual tokens and use asr data to map the corresponding speech and text to the same virtual token in the shared codebook. our experiments demonstrate that our method can learn an effective end-to-end st model without any parallel st data. it significantly improves the existing sota and achieves competitive performance compared to the supervised models.