A da T ran S : Adapting with Boundary-based Shrinking for End-to-End Speech Translation (2023)

Abstract: To alleviate the data scarcity problem in End-to-end speech translation (ST), pre-training on data for speech recognition and machine translation is considered as an important technique. However, the modality gap between speech and text prevents the ST model from efficiently inheriting knowledge from the pre-trained models. In this work, we propose AdaTranS for end-to-end ST. It adapts the speech features with a new shrinking mechanism to mitigate the length mismatch between speech and text features by predicting word boundaries. Experiments on the MUST-C dataset demonstrate that AdaTranS achieves better performance than the other shrinking-based methods, with higher inference speed and lower memory usage. Further experiments also show that AdaTranS can be equipped with additional alignment losses to further improve performance.

## introduction end-to-end speech translation (st), which directly translates source speech into text in another language, has achieved remarkable progress in recent years @xcite @xcite @xcite . compared to the conventional cascaded systems @xcite , the end-to-end models are believed to have the advantages of low latency and less error propagation. a well-trained end-to-end model typically needs a large amount of training data. however, the available direct speech-translation corpora are very limited @xcite . given the fact that data used for automatic speech recognition (asr) and machine translation (mt) are much richer, the paradigm of "pre-training on asr and mt data and then fine-tuning on st" becomes one of the approaches to alleviate the data scarcity problem @xcite . it has been shown that decoupling the st encoder into acoustic and semantic encoders is ben-eficial to learn desired features @xcite . initializing the two encoders by pre-trained asr and mt encoders, respectively, can significantly boost the performance @xcite . however, the modality gap between speech and text might prevent the st models from effectively inheriting the pre-trained knowledge @xcite . the modality gap between speech and text can be summarized as two dimensions. first, the length gap -the speech features are usually much longer than their corresponding texts @xcite . second is the representation space gap. directly fine-tuning mt parameters (semantic encoder and decoder) with speech features as inputs, which learned independently, would result in sub-optimal performance. previous work has explored and proposed several alignment objectives to address the second gap, e.g., cross-modal adaption @xcite , cross-attentive regularization @xcite and cross-modal contrastive learning @xcite . a shrinking mechanism is usually used to address the length gap. some leverage continuous integrate-and-fire (cif) @xcite to shrink the long speech features @xcite , but they mostly work on simultaneous st and need extra efforts to perform better shrinking. others mainly depend on the ctc greedy path @xcite , which might introduce extra inference cost and lead to sub-optimal shrinking results. adatrans uses a new shrinking mechanism called boundary-based shrinking, which achieves higher performance. through extensive experiments on the must-c (di gangi et al., 2019) dataset, we show that ada-trans is superior to other shrinking-based methods with a faster inference speed or lower memory usage. further equipped with alignment objectives, adatrans shows competitive performance compared to the state-of-the-art models. acoustic encoder semantic encoder decoder shrink ctc module ctc loss st loss boundary predictor source speech target text softmax pred loss guide figure 1: adatrans architecture, where the blue modules are initialized with the asr model and the orange modules are initialized with the mt model. the ctc module (dotted) can be removed during inference. ## related work numerous techniques have been proposed to adapt speech and text representations in order to mitigate the modality gap in end-to-end st. @xcite introduce noise into the text input, while @xcite ; @xcite employ phoneme sequences for text input. these approaches reduce differences between speech and text input by extending text representations. conversely, some research aims at shrinking lengthy speech input by presenting various shrinking strategies @xcite @xcite @xcite . our work falls within this category, proposing a new approach that offers enhanced effectiveness and efficiency. others prioritize aligning the speech-text representation space @xcite @xcite @xcite . ctc alignment's role in enhancing st is particularly pertinent to our research. earlier studies have demonstrated that integrating an additional ctc loss for multi-task learning or pre-training assists in st model training @xcite . some also leverage ctc alignment to shrink speech inputs @xcite @xcite . moreover, @xcite and @xcite delve deeper, optimizing ctc objectives for st. ## conclusion this work proposes a new end-to-end st model called adatrans, which uses a boundary predictor trained by signals from ctc output probabilities, to adapt and bridge the length gap between speech and text. experiments show that adatrans per-forms better than other shrinking-based methods, in terms of both quality and efficiency. it can also be further enhanced by modality alignment objectives to achieve state-of-the-art results.