Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translation (2022)

Abstract: End-to-end speech translation relies on data that pair source-language speech inputs with corresponding translations into a target language. Such data are notoriously scarce, making synthetic data augmentation by back-translation or knowledge distillation a necessary ingredient of end-to-end training. In this paper, we present a novel approach to data augmentation that leverages audio alignments, linguistic properties, and translation. First, we augment a transcription by sampling from a suffix memory that stores text and audio data. Second, we translate the augmented transcript. Finally, we recombine concatenated audio segments and the generated translation. Our method delivers consistent improvements of up to 0.9 and 1.1 BLEU points on top of augmentation with knowledge distillation on five language pairs on CoVoST 2 and on two language pairs on Europarl-ST, respectively.

## introduction end-to-end automatic speech translation (ast) relies on data that consist only of speech inputs and corresponding translations. such data are notoriously limited. data augmentation approaches attempt to compensate the scarcity of such data by generating synthetic data by translating transcripts into foreign languages or by back-translating targetlanguage data via text-to-speech synthesis (tts) @xcite , or by performing knowledge distillation using a translation system trained on gold standard transcripts and reference translations @xcite . in this paper, we present a simple, resource conserving approach that does not require tts and yields improvements complementary to knowledge distillation (kd). for training cascaded systems, monolingual data for automatic speech recognition and textual trans-lation data for machine translation can be used, reducing the problem of scarcity. cascaded systems, however, suffer from error propagation, which has been addressed by using more complex intermediate representations such as n-best machine translation (mt) outputs or lattices @xcite or by modifying training data to incorporate errors from automatic speech recognition (asr) and mt @xcite . end-to-end systems are unaffected by this kind of error propagation and are able to surpass cascaded systems if trained on sufficient amounts of data @xcite . our approach transfers an idea on aligned data augmentation that has been presented for asr @xcite to aligned data augmentation in ast. similar to aligned data augmentation for asr, we utilize forced alignment information to create unseen training pairs in a structured manner. our augmentation procedure consists of the following steps: (1) sampling of a replacement suffix of a transcription and its aligned speech representations, guided by linguistic constraints. (2) translation of the transcription containing the new suffix. (3) recombination of audio data containing the new suffix and the generated translation to distill a new training pair. we thus use the acronym str (sample, translate, recombine) to refer to our method. in comparison to @xcite and @xcite who use tts to generate synthetic speech, we create new examples by recombining real human speech. this reduces the problem of overfitting to synthetic data as for example in skinaugment @xcite where synthetic audio is generated by auto-encoding speaker conversions. the basic idea of our method is comparable to data augmentation techniques for images such as cut-mix @xcite where images are blended together to form new data examples. however, cut-mix selects images randomly, while we recombine phrases in a structured manner. our experimental evaluation is conducted for five language pairs on the covost 2 dataset @xcite and for two language pairs on the europarl-st (iranzo-sánchez et al., 2020) dataset. we find considerable improvements for all language pairs on all datasets for our approach on top of kd. our approach can be seen as an enhancement of @xcite 's kd approach since it requires roughly the same computational resources and consistently improves their gains. ## method our method exploits audio-transcription alignment information to generate previously unseen data pairs for end-to-end ast training. by applying a part-of-speech (pos) tagger on a sentence, we identify potential "pivoting tokens" where the token's prefix or suffix, i.e., the preceding or succeeding tokens, can be exchanged between other sentences containing the same token of the same syntactic function. we then sample possible suffixes for that token from a suffix memory containing text and audio suffixes, and concatenate the prefix, verb, and suffix to generate a new transcription. then, an mt system translates the new transcription, picking up on the idea of knowledge distillation in ast @xcite . the mt system is trained or fine-tuned on the transcription-translation pairs. finally, using the previously sampled audio suffix, we concatenate prefix, verb, and suffix audio together with the mt generated translation to recombine a new audio-translation pair for end-to-end ast training. our augmentation method implements linguistic constraints by making use of the transcription's syntactic structure in combination with alignment information. effectively, we exploit the strict svoscheme of english sentences as we select the verb as our pivoting token. our method is applicable to other languages, however, it will require more effort to identify exchangeable syntactic structures. figure 1 illustrates our approach. we start by identifying the pivoting token in a transcription we want to augment, here "playing" in the sentence "two children are playing on a statue". then, we extract the list of possible suffixes following "playing" from the suffix memory and sample a single audio-text suffix, here "volleyball in a park". together with the original prefix and pivoting token, the textual part of the sampled suffix builds a new augmented transcription. similarly, together with the audio prefix and token, the audio part of the suffix builds a new augmented audio example. the augmented transcription is then translated by an mt model. the new audio example (i.e., the representation of "two children playing volleyball in a park") and the translation (i.e., the text "zwei kinder spielen volleyball in einem park") are then recombined to form a new audio-translation pair. ## experimental setting data preprocessing we evaluate our method on two common ast datasets, covost 2 @xcite and europarl-st (iranzo-sánchez et al., 2020). since europarl-st is too small for mt training from scratch, we use 1.6m en-de sentence pairs from wikipedia following @xcite and 3.2m en-fr sentence pairs from the common crawl corpus foot_0 as additional data. more details on the datasets are in appendix a.1. for speech data preprocessing, we extract log mel-filter banks of 80 dimensions computed every 10ms with a 25ms window. we normalize the speech features per channel using mean and variance per instance. for all textual data, punctuation is normalized using sacremoses. 2 the transcriptions are lowercased with punctuation removed. for the speech-to-text tasks on covost 2, we employ character-level models due to the availability of pre-trained high quality asr models. for the speech-to-text tasks on europarl-st, we learn 5,000 subword units for each target language. for the machine translation tasks in knowledge distillation, we learn a joint subword vocabulary on both source and target for each language pair of size 5,000 for covost 2 and size 40,000 for europarl-st including the additional training data. subword unit creation is always conducted with sentence-piece @xcite . the montreal forced aligner @xcite is applied without any fine-tuning to extract the acoustic alignments. thus, the obtained alignments can be of low quality and we discard such examples from our augmentation procedure. please refer to appendix a.2 for details on our filtering criteria. to extract pos-tags, we use the spacy 3 toolkit. we select the verb as our pivoting token and generate the suffix memory as follows: for each verb, we generate a list of audio-text suffix pairs and store the data in a key-value table . the audio entries contain only references to the original audio segments and our implementation is thus very memory efficient. we only utilize basic offthe-shelf components that are widely available and our suffix memory has a negligible memory footprint. ## results our experiments are focused on the improvements of our proposed method over kd alone on both covost 2 and europarl-st datasets. we evaluate the translation results with both bleu foot_5 @xcite and chrf2 foot_6 @xcite using the implementation of sacrebleu @xcite . each experiment is repeated 3 times and we report mean and standard deviation. we also conduct significance tests using a paired approximate randomization test @xcite with default settings of sacre-bleu. we compute p-values between kd and kd+str for each evaluated language pair of the experiments' datasets and between each run initialized with the same random seed. the individual p-values are reported in appendix a.4. section 4.3 contains a discussion on the connection between str-and mt-performance. we also report additional experiments which show how the amount of str data affects the final performance in section 4.4. an error analysis with examples and a discussion on the limitations of str has been moved to appendix a.5 due to space constraints. ## conclusion we proposed an effective data augmentation method for end-to-end speech translation which leverages audio alignments, linguistic properties, and translation. it creates new audio-translation pairs via sampling from a memory-efficient suffix memory, translating through an mt model and recombining original and sampled audio segments with translations. our method achieves significant improvements over augmentation with kd alone on both large (covost 2) and small scale (europarl-st) datasets. in future work, we would like to investigate the utility of other linguistic properties for ast augmentation and we would like to extend our method to multilingual ast.