Dealing with training and test segmentation mismatch: FBK @ IWSLT 2021 (2021)

Abstract: This paper describes FBK’s system submission to the IWSLT 2021 Offline Speech Translation task. We participated with a direct model, which is a Transformer-based architecture trained to translate English speech audio data into German texts. The training pipeline is characterized by knowledge distillation and a two-step fine-tuning procedure. Both knowledge distillation and the first fine-tuning step are carried out on manually segmented real and synthetic data, the latter being generated with an MT system trained on the available corpora. Differently, the second fine-tuning step is carried out on a random segmentation of the MuST-C v2 En-De dataset. Its main goal is to reduce the performance drops occurring when a speech translation model trained on manually segmented data (i.e. an ideal, sentence-like segmentation) is evaluated on automatically segmented audio (i.e. actual, more realistic testing conditions). For the same purpose, a custom hybrid segmentation procedure that accounts for both audio content (pauses) and for the length of the produced segments is applied to the test data before passing them to the system. At inference time, we compared this procedure with a baseline segmentation method based on Voice Activity Detection (VAD). Our results indicate the effectiveness of the proposed hybrid approach, shown by a reduction of the gap with manual segmentation from 8.3 to 1.4 BLEU points.

## introduction speech translation (st) is the task of translating a speech uttered in one language into its textual representation in a different language. unlike simultaneous st, where the audio is translated as soon as it is produced, in the offline setting the audio is entirely available and translated at once. in continuity with the last two rounds of the iwslt evaluation campaign @xcite , the iwslt2021 offline speech translation task @xcite focused on the translation into german of english audio data extracted from ted talks. participants could approach the task either with a cascade architecture or with a direct end-to-end system. the former represents the traditional pipeline approach @xcite comprising an automatic speech recognition (asr) followed by a machine translation (mt) component. the latter @xcite relies on a single neural network trained to translate the input audio into target language text bypassing any intermediate symbolic representation steps. the two paradigms have advantages and disadvantages. cascade architectures have historically guaranteed higher translation quality @xcite thanks to the large corpora available to train their asr and mt sub-components. however, a well-known drawback of pipelined solutions is represented by error propagation: transcription errors are indeed hard (and sometimes impossible) to recover during the translation step. direct models, although being penalized by the paucity of training data, have two theoretical competitive advantages, namely: i) the absence of error propagation as there are no intermediate processing steps, and ii) a less mediated access to the source utterance, which allows them to better exploit speech information (e.g. prosody) without loss of information. the paucity of parallel (audio, translation) data for direct st has been previously addressed in different ways, ranging from model pre-training to exploit knowledge transfer from asr and/or mt @xcite @xcite , knowledge distillation @xcite , data augmentation @xcite @xcite , and multi-task learning @xcite @xcite . thanks to these studies, the gap between the strong cascade models and the new end-to-end ones has gradually reduced during the last few years. as highlighted by the iwslt 2020 offline speech translation challenge results @xcite , the rapid evolution of the direct approach has eventually led it to performance scores that are similar to those of cascade architectures. in light of this positive trend, we decided to adopt only the direct approach (described in section 3) for our participation in the 2021 round of the offline st task. another interesting finding from last year's campaign concerns the sensitivity of st models to different segmentations of the input audio. the 2020 winning system @xcite shows that, with a custom segmentation of the test data, the same model improved by 3.81 bleu points the score achieved when using the basic segmentation provided by the task organizers. this noticeable difference is due to a well-known problem in mt, st and in machine learning at large: any mismatch between training and test data (in terms of domain, text style or a variety of other aspects) can cause unpredictable, often large, performance drops at test time. in st, this is a critical issue, inherent to the nature of the available resources: while systems are usually trained on corpora that are manually segmented at sentence level, test data come in the form of unsegmented continuous speech. a possible solution to this problem is to automatically segment the test data with a voice activity detection (vad) tool @xcite . this strategy tries to mimic the sentence-based segmentation observed in the training data using pauses as an indirect (hence known to be sub-optimal) cue for sentence boundaries. custom segmentation strategies, which are allowed to iwslt participants, typically go in this direction with the aim to reduce the data mismatch by working on evaluation data. an opposite way to look at the problem is to work on the training data. in this case, the goal is to "robustify" the st model to noisy inputs (i.e. sub-optimal segmentations) at training time, by exposing it to perturbed data where sentence-like boundaries are not guaranteed. our participation in the offline st task exploits both solutions (see section 4): at training time, by fine-tuning the model with a random segmentation of the available in-domain data; at test time, by feeding it with a custom hybrid segmentation of the evaluation data. in a nutshell, our participation can be summarized as follows. after a preliminary model selection phase that was carried out in order to select the best architecture, we adopted a pipeline consisting of: i) asr pre-training, ii) st training with knowledge distillation with an mt teacher, and iii) two-step fine-tuning by varying the type and the amount of data between the two steps. the second fine-tuning step, which was carried out on artificially perturbed data to increase model robustness, represents the main aspect characterizing our participation to this year's round of the offline st task together with our custom automatic segmentation of the test set (see section 4). our experimental results proved the effectiveness of our solutions: compared to a standard st model and a baseline vad-based method, on the must-c v2 english-german test set @xcite , the gap with optimal manual segmentation is reduced from 8.3 to 1.4 bleu. ## training data to build our models, we used most of the training data allowed for participation. foot_0 they include: mt corpora (english-german text pairs), asr corpora (english audios and their corresponding transcripts) and st corpora (english audios with corresponding english transcripts and german translations). mt. among all the available datasets, we selected those allowed for wmt 2019 @xcite and opensubtitles2018 @xcite . some pre-processing was required to isolate and remove different types of potentially harmful noise present in the data. these include non-unicode characters, both on the source and target side of the parallel sentence pairs, which would have led to an increased dictionary size hindering model training, and whole non-german target sentences (mostly in english). the cleaning of this two types of noise, which was respectively performed using a custom script and modern mt @xcite , resulted in the removal of roughly 25% of the data, with a final dataset of ∼49 million sentence pairs. asr. asr corpora, together with the st ones described below, were collected for the asr training. in detail, the allowed native asr datasets are: librispeech @xcite , tedlium v3 @xcite and mozilla common voice. 2 in all of them, english texts were lowercased and punctuation was removed. st. the st benchmarks we used are essentially three: i) europarl-st (obtained from european parliament debates -iranzo-sánchez et al. 2020), ii) must-c v2 (built from ted talks -cattoni et al. 2021), and iii) covost 2 (containing the translations of a portion of the mozilla common voice dataset - @xcite ). to cope with the scarcity of st data, we complemented these native st corpora with synthetic data. to this aim, we used the mt system trained on the available mt data to translate into german the english transcripts of the aforementioned asr datasets. the resulting texts were used as reference material during the st model training. the combination of native and generated data resulted in a total of about 1.26 million samples. the transcription-translation pairs were tokenized using, respectively, source/targetlanguage sentencepiece @xcite unigram models trained on the mt corpora with a vocabulary size of 32k tokens. similar to our last year's iwslt submission @xcite , the entire dataset was used for training in a multidomain fashion, where the two domains were native (original st data) and generated (synthetic data). prior to the extraction of the speech features, the audio was pre-processed with the specaugment @xcite data augmentation technique, which masks consecutive portions of the input both in frequency and in time dimensions. from all the audio files, 80 log mel-filter banks features were extracted using pykaldi @xcite , filtering out those samples containing more than 3,000 frames. finally, we applied utterance level cepstral mean and variance normalization both during asr pre-training and st training phases. the configuration parameters used are the default ones as set in @xcite . ## model and training in order to select the best performing architecture, we trained several transformer-based models @xcite , which consist of 12 encoder layers, 6 decoder layers, 8 attention heads, 512 features for the attention layers and 2,048 hidden units in the feed-forward layers. the asr and st models are based on a custom version of the model by @xcite , which is a transformer whose encoder has two initial 1d convolutional layers with gelu activation functions @xcite . also, the encoder self-attentions were biased using a logarithmic distance penalty in favor of the local context as per @xcite . a connectionist temporal classification (ctc) scoring function was applied as described in @xcite . this was done by adding a linear layer to either the 6th, 8th or 10th encoder layer to map the encoder states to the vocabulary size and compute the ctc loss. the choice of the final architecture, depending on where the ctc loss is applied, was made based on sacrebleu score @xcite after training the models on must-c v1 en-de @xcite . st results computed on the test set are reported on ## coping with training/test data mismatch as mentioned in section 1, the segmentation of audio files is a crucial aspect in st. in fact, mismatches between the manual segmentation of the training data and the automatic one required when processing the unsegmented test set can produce significant performance drops. to mitigate this risk, we worked on two complementary fronts: at training and inference time. at training time, we tried to robustify our model by fine-tuning it on a randomly segmented subset of the training data. at inference time, we applied an automatic segmentation procedure to the test set in order to feed the model with input resembling, as much as possible, the gold manual segmentation. these two solutions, which characterize our final submission, are explained in the following. fine-tuning step #2: using randomly segmented data. for the second fine-tuning step, we re-segmented the must-c v2 en-de training set following the procedure described in @xcite . the method consists in choosing a random word in the transcript of each sample, and using it as sentence boundary instead of the linguistically-motivated (sentence-level) splits provided in the original data. the corresponding audio segments are then obtained by means of audiotext alignments performed with gentle. 3 similarly, the german translation of each re-segmented transcript is extracted with cross-lingual alignments generated by a fast align @xcite model trained on all the mt data available for the task and on must-c v2. in case either of the alignments is model must-c2 must-c2 must-c2 iwslt2015 iwslt2015 manual vad (webrtc) hybrid vad (lium) hybrid 1-ft lsce 27.6 20.8 24.8 16.1 21.9 2-ft lsce -23.4 (+2.6) 26.4 (+1.6) 20.7 (+4.6) 22.7 (+0.8) 1-ft lsce+ctc 27.7 19.9 25.3 14.0 21.7 2-ft lsce+ctc -23.7 (+3.8) 26.3 (+1.0) 20.9 (+6.9) 23.1 (+1.4) automatic segmentation of the test data. at inference time, the test set was segmented with an hybrid approach that considers both the audio content and the length of the resulting segment @xcite . specifically, every segment is ensured to be at least 17s and at most 20s long, but the exact splitting position is determined by the longest pause detected within this interval. pauses are identified with the webrtc vad tool @xcite , using 20ms as frame duration and 2 as aggressivity level. ## experimental settings our implementation is built on top of fairseq pytorch library @xcite . all our models were trained using the adam optimizer (kingma and ba, 2015) with β 1 = 0.9, β 2 = 0.98. during training, the learning rate was set to increase linearly from 0 to 2e-3 for the first 10,000 warm-up steps and then to decay with an inverse square root policy. differently, the learning rate was kept constant for model fine-tuning, with a value of 1e-3 for the first fine-tuning step and 1e-4 for the second one. all the trainings were performed on 2 tesla v100 gpus with 32gb ram. we set the maximum number of tokens to 10k per batch and 8 as update frequency. for generation, the maximum number of tokens was increased to 50k, using a single tesla v100 gpu and by applying a standard 5-beam search strategy. ## results for the evaluation of the fine-tuned models we considered three different test sets: must-c v2 en-de tst-common, iwslt 2015 and 2019 test sets (available on the offline st task evaluation campaign web page foot_3 ). while for must-c v2 we originally had a manual segmentation of the audio files, for the iwslt 2015 and 2019 test sets the organizers provided only automatic segmentations obtained by the lium vad tool @xcite . furthermore, we segmented must-c v2 tst-common using the webrtc vad tool to have a comparable framework. for our primary submission, we chose the twostep fine-tuned model that uses the lsce+ctc losses for the first fine-tuning step (2-ft lsce+ctc) since it achieved the highest bleu on automatically segmented data. in order to measure the contribution of fine-tuning on randomly segmented data also on the official evaluation set, we selected the same model before the second finetuning step (1-ft lsce+ctc) as our contrastive submission. our primary submission scored 30.6 bleu on the tst2021 test set considering both references while our contrastive scored 29.3 bleu, showing the effectiveness of our fine-tuning step. in addition, our primary submission scored 24.7 bleu on the tst2020 test set. ## conclusions we described fbk's participation in the iwslt2021 offline speech translation task @xcite . our work focused on a multi-step training pipeline involving data augmentation (specaugment and mt-based synthetic data), multi-domain transfer learning (kd training first and then fine-tuning on synthetic and native data) and ad-hoc fine-tuning on randomly segmented data. based on the experimental results, our submission was characterized by the use of the ctc loss on transcripts during word-level knowledge distillation training, followed by a two-stage fine-tuning aimed to fill the gap between the performance of models when tested on manual and automatically segmented data. this huge gap was pointed out in our last year submission @xcite , where we highlighted that some strategies should have been adopted in order to mitigate the problem. this paper demonstrates that, following the above-mentioned pipeline, together with some data-driven techniques, we can obtain significant improvements in the performance of end-to-end st systems. research in this direction will help us to build models that are not only competitive with cascaded solutions, but also able to handle different segmentation strategies which are going to be more frequently used in the future.