S eg A ugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentations (2023)

Abstract: End-to-end Speech Translation is hindered by a lack of available data resources. While most of them are based on documents, a sentence-level version is available, which is however single and static, potentially impeding the usefulness of the data. We propose a new data augmentation strategy, SegAugment, to address this issue by generating multiple alternative sentence-level versions of a dataset. Our method utilizes an Audio Segmentation system, which re-segments the speech of each document with different length constraints, after which we obtain the target text via alignment methods. Experiments demonstrate consistent gains across eight language pairs in MuST-C, with an average increase of 2.5 BLEU points, and up to 5 BLEU for low-resource scenarios in mTEDx. Furthermore, when combined with a strong system, SegAugment obtains state-of-the-art results in MuST-C. Finally, we show that the proposed method can also successfully augment sentence-level datasets, and that it enables Speech Translation models to close the gap between the manual and automatic segmentation at inference time.

## introduction the conventional approach for speech translation (st) involves cascading two separate systems: an automatic speech recognition (asr) model followed by a machine translation (mt) model. however, recent advances in deep learning @xcite , coupled with an increased availability of st corpora @xcite have enabled the use of end-to-end models @xcite . although end-to-end models can address several shortcomings of cascaded models, such as slow inference times, error propagation, and information loss, they are limited by a data bottleneck @xcite . this bottleneck arises from the inability of end-toend models to directly leverage data from the more resourceful tasks of asr and mt, which restricts them from consistently matching the performance of the cascaded models @xcite @xcite . the majority of st corpora are based on document-level speech data, such as must-c @xcite and mtedx @xcite , which are derived from ted talks, with duration times of 10 to 20 minutes. these documentbased data are processed into shorter, sentencelevel examples through a process called manual segmentation, which relies on grammatical features in the text. still, this sentence-level version is single and static and is potentially limiting the utility of the already scarce st datasets. to address this limitation, we propose segaug-ment, a segmentation-based data augmentation method that generates multiple alternative sentencelevel versions of document-level speech data (fig 1 ). segaugment employs shas @xcite , an audio segmentation method that we tune to yield different re-segmentations of a speech document based on duration constraints. for each new segmentation of a document, the corresponding transcript is retrieved via ctc-based forced alignment @xcite , and the target text is obtained with an mt model. our contributions are as follows: ## relevant research specaugment @xcite , which directly modifies the speech features by wrapping or masking them, is a standard approach for data augmentation in speech tasks including st @xcite . wavaugment @xcite is a similar technique that modifies the speech wave by introducing effects such as pitch, tempo and echo @xcite . instead of altering the speech input, our proposed method generates more synthetic data by altering their points of segmentation, and thus is complimentary to techniques such as specaugment. an effective way to address data scarcity in st is by generating synthetic data from external sources. this can be achieved by using an mt model to translate the transcript of an asr dataset or a textto-speech (tts) model to generate speech for the source text of an mt dataset @xcite @xcite . in contrast, segaugment generates synthetic data internally, without relying on external datasets. previous research has established the benefits of generating synthetic examples by cropping or merging the original ones, with sub-sequence sampling for asr @xcite , and concatenation for mt @xcite @xcite , as well as for asr and st 1 github.com/mt-upc/segaugment @xcite . our approach, however, segments documents at arbitrary points, thus providing access to a greater number of synthetic examples. an alternative approach by @xcite involves recombining training data in a linguisticallymotivated way, by sampling pivot tokens, retrieving possible continuations from a suffix memory, combining them to obtain new speech-transcription pairs, and finally using an mt model to generate the translations. our method is similar since it also leverages audio alignments and mt, but instead of mixing speech, it segments at alternative points. context-aware st models have been shown to be robust towards error-prone automatic segmentations of the test set at inference time @xcite . our method bears similarities with @xcite ; @xcite in that it re-segments the train set to create synthetic data. however, unlike their approach, where they split at random words in the transcript, we use a specialized audio segmentation method @xcite to directly split the audio into segments resembling proper sentences. furthermore, instead of using word alignment algorithms to get the target text @xcite , we learn the alignment with an mt model. we thus create high-quality data that can be generally useful, and not only for errorprone test set segmentations. finally, recent work has demonstrated that context-aware st models evaluated on fixed-length automatic segmentations can be competitive compared to the manual segmentation @xcite . here, we find that utilizing data from segaugment yields high translation quality for st models evaluated on automatic segmentations, even surpassing the translation quality of the manual segmentation, and without explicitly making them context-aware. ## proposed methodology the proposed data augmentation method segaug-ment (fig. 3 ) aims to increase the utility of the training data s, by generating synthetic sentencelevel corpora ŝℓ , which are based on alternative segmentations of the speech documents in d (eq. 1). whereas the manual segmentation (fig. 2 ) relies on grammatical features in the text, here we propose to split on acoustic features present in the audio, by utilizing shas ( §3.2). for the i-th speech document x i , segaugment creates alternative segmentation boundaries bi with shas ( §4.1), obtains the corresponding transcriptions ẑi via ctc-based forced alignment ( §4.2), and finally, generates the translations ŷi with an mt model ( §4.3). by repeating this process with different parameterizations ℓ of the segmentation algorithm a, multiple synthetic sentence-level speech corpora can be generated ( §4.4). a synthetic speech corpus ŝℓ with @xmath0: where x i is the original speech for the i-th document (eq. 1), @xmath1,1 , ..., bi,k i ) are its alternative segmentations, ẑi = (ẑ i,1 , ..., ẑi,k i ) are its sentencelevel transcriptions, and ŷi = (ŷ i,1 , ..., ŷi,k i ) are its synthetic sentence-level translations. in total, three different models are utilized for creating a synthetic corpus, a classifier c ( §3.2) for segmentation, a ctc encoder e @xcite for forced alignment, and an mt model m for text alignment. we can use pre-trained models, or optionally learn them from the manually segmented examples of s (fig. 4 ). the classifier c can be learned from ## experimental setup for our experiments we use data from three st datasets, @xcite , mtedx @xcite and covost2 @xcite ( dataset v lang. pair # docs # sents # hours must-c 1.0 en-de 2,093 234k 408 en-es 2,514 266k 496 en-fr 2,460 275k 485 en-it 2,324 254k 457 en-nl 2,219 248k 434 en-pt 2,001 206k 377 en-ro 2,166 236k 424 en-ru 2,448 265k 482 2.0 en-de 2,537 251k 450 mtedx es-es 988 102k 178 pt-pt 812 90k 153 es-en 378 36k 64 pt-en 279 31k 53 es-fr 43 4k 6 covost2 en-de -231k 362 for speech translation we train speech-to-text transformer baselines @xcite . unless stated otherwise, we use the small architecture (s2t_transformer_s) with 12 encoder layers and 6 decoder layers, and dimensionality of 256, with asr pre-training using only the original data. the full details of the models and the training proce- ## conclusions model en-de baseline 24.3 / 51.9 baseline + in-data-kd 25.2 / 52.7 baseline + segaugment 26.2 / 53.7