WACO : Word-Aligned Contrastive Learning for Speech Translation (2023)

Abstract: End-to-end Speech Translation (E2E ST) aims to directly translate source speech into target text. Existing ST methods perform poorly when only extremely small speech-text data are available for training. We observe that an ST model’s performance closely correlates with its embedding similarity between speech and source transcript. In this paper, we propose Word-Aligned COntrastive learning (WACO), a simple and effective method for extremely low-resource speech-to-text translation. Our key idea is bridging word-level representations for both speech and text modalities via contrastive learning. We evaluate WACO and other methods on the MuST-C dataset, a widely used ST benchmark, and on a low-resource direction Maltese-English from IWSLT 2023. Our experiments demonstrate that WACO outperforms the best baseline by 9+ BLEU points with only 1-hour parallel ST data. Code is available at https://github.com/owaski/WACO .

## introduction end-to-end speech translation (e2e st) directly translates speech in a source language to text in a target language, without intermediate pipelines. e2e st has witnessed significant progress in translation quality @xcite @xcite @xcite @xcite . however, existing e2e st methods still perform poorly when only a limited amount of parallel st data are available. how can we build a highly performant st model with extremely low resource, e.g. only 1 hour of parallel data (approximately a few hundred utterances)? we analyze the encoder representations from a directly learned st model 1 and find that its average embeddings from speech and transcript are similar at the sequence level but still not aligned 1 we use xstnet @xcite as an example. well on the word level (figure 1a ). an ideal st model should encode a speech utterance closely aligned with the representations of corresponding words in its transcript text (figure 1b ). prior methods attempt to use additional parallel data from machine translation (mt) and automatic speech recognition (asr) to align speech and text representations. however, most do not explicitly reduce the word level representation gap among speech and text. we hypothesize that such misalignment at the word level between corresponding speech and transcript text is a critical cause of the inferior st performance. we further observe such a misalignment phenomenon is severe when st data is extremely low. we did a pilot study by training direct st models using different sizes of st data (1/5/10/388 hours) . as shown in figure 2 , we find that the translation performance highly correlates with the word-level embedding similarity between speech and transcript text. with fewer parallel st data, the cross-modal similarity drops simultaneously with the bleu score and almost reaches 0 given 1-hour st training data. this observation suggests that the model can map both modalities into a partially aligned semantic space given sufficiently large st data but fails when st data is extremely small. in this paper, we propose waco, a word-level contrastive learning method for extremely lowresource speech-to-text translation. its key idea is to promote representation similarity among corresponding words in speech and text and to push non-corresponding representations further apart in embedding spaces. furthermore, it can use additional large asr data to improve word-level representation learning. we conduct experiments on the must-c benchmark with varying data sizes and a real-world low-resource direction maltese-english (mt-en) from iwslt 2023 low-resource track. we also analyze the word-level representation similarity from the learned models. our contributions are: end-to-end st due to error propagation and high latency in cascaded st systems, bérard et al. (2016); duong et al. (2016) first proposed to translate source speech into target text directly without generating the intermediate transcript. the major difficulty in training end-to-end st systems is the lack of direct st data. though many st datasets @xcite were proposed in recent years, the amount of st data is still much less than that of mt and asr. to overcome the data scarcity problem, methods including data augmentation @xcite @xcite , self-training (pino et al., 2020), multitasking @xcite @xcite and pre-training @xcite @xcite @xcite @xcite @xcite have been proposed. waco is a novel approach that can be applied in existing multi-tasking and pre-training frameworks to improve st performance. ## proposed method: waco in this section, we describe our approach to develop effective speech translation models with extremely low-resource parallel data. ## conclusion in this work, we propose waco to align wordlevel speech and text embeddings. experiments demonstrate the effectiveness of our method in both low-resource and regular st settings. analysis shows that our method can achieve better speechtext alignment which correlates well with its st performance.