DTW -Align: Bridging the Modality Gap in End-to-End Speech Translation with Dynamic Time Warping Alignment (2025)

Abstract: End-to-End Speech Translation (E2E-ST) is the task of translating source speech directly into target text bypassing the intermediate transcription step. The representation discrepancy between the speech and text modalities has motivated research on what is known as bridging the modality gap. State-of-the-art methods addressed this by aligning speech and text representations on the word or token level. Unfortunately, this requires an alignment tool that is not available for all languages. Although this issue has been addressed by aligning speech and text embeddings using nearest-neighbor similarity search, it does not lead to accurate alignments. In this work, we adapt Dynamic Time Warping (DTW) for aligning speech and text embeddings during training. Our experiments demonstrate the effectiveness of our method in bridging the modality gap in E2E-ST. Compared to previous work, our method produces more accurate alignments and achieves comparable E2E-ST results while being significantly faster. Furthermore, our method outperforms previous work in low resource settings on 5 out of 6 language directions.

## introduction end-to-end speech translation (e2e-st) is the task of translating speech in a source language directly into text in a target language. e2e-st gained success and attention as an alternative to cascaded solutions where an automatic speech recognition (asr) and a machine translation (mt) models are combined @xcite @xcite @xcite @xcite . cascaded solutions benefit from abundant asr and mt data but might suffer from error propagation and high latency, which can be solved by e2e-st. however, training e2e-st models is not straightforward due to the representation discrepancy between the speech and text modalities. previous work has achieved state-of-the-art results by aligning speech and text representations at the word or token level, either using an alignment tool @xcite or by generating the alignment automatically during training @xcite . the closest to our work is cross-modal mixup via optimal transport (cmot), which uses optimal transport for finding speech and text alignments. although cmot achieves state-of-the-art results, it does not guarantee producing monotonic alignments or ensure that each text token is assigned to at least one frame. this contradicts the expected structure of speechtext alignment and can lead to noisy alignments. furthermore, cmot introduces a significant training time overhead. in this work, we introduce dtw-align, a method for aligning speech and text embeddings during training using an adaptation of dynamic time warping @xcite . figure 1 shows an example alignment generated using dtw-align and cmot, which illustrates that our method generates monotonic alignments and guarantees that all tokens are aligned, while cmot does not. we demonstrate the effectiveness of our method in bridging the modality gap with mixup training @xcite . similarly to previous work @xcite , we train on a mixup of aligned speech and text representations, however, instead of discretely selecting either a speech or a text embedding, we linearly interpolate speech and text embeddings @xcite . our experiments show that our method is faster and produces more accurate alignments. furthermore, it achieves comparable results to cmot on 6 language directions from the covost2 dataset, while training significantly faster. we also evaluate our method in a low resource setting where training can be more vulnerable to alignment noise, and we show that our method leads to a statistically significant improvement over cmot in 5 out of 6 language directions. ## results and discussion ## analysis although our method substantially outperfroms cmot in terms of alignment accuracy, it does not yield improvements in st performance. we attribute this to two factors: the amount of training data, which makes training more robust under noise and the low mixup probability value, which is set to 0.2. in §6.1 we measure the performance of cmot, dtw-align-discrete and dtw-align in a simulated low resource scenario of 10h per lan- ## conclusion we introduce a method that eliminates the requirement for an external forced alignment tool by dynamically aligning speech and text embeddings during training based on dynamic time warping (dtw). compared to state-of-the-art approaches, our method matches or exceeds bleu score results while being significantly faster. we further demonstrate that using dtw-align is more robust and data efficient in low resource settings. in addition, compared to hubert-transformer baseline, our method improves performance by 1.2 and 2.4 bleu points in high and low resource settings respectively with minimal overhead in the training time. finally, unlike cmot, our method can produce both token and word level alignments, which makes it compatible with previous work that requires word level alignments @xcite @xcite , therefore, it can bring a boost to the ongoing efforts on bridging the modality gap in e2e-st or other speech-to-text tasks.