Efficient yet Competitive Speech Translation: FBK @ IWSLT 2022 (2022)

Abstract: The primary goal of this FBK’s systems submission to the IWSLT 2022 offline and simultaneous speech translation tasks is to reduce model training costs without sacrificing translation quality. As such, we first question the need of ASR pre-training, showing that it is not essential to achieve competitive results. Second, we focus on data filtering, showing that a simple method that looks at the ratio between source and target characters yields a quality improvement of 1 BLEU. Third, we compare different methods to reduce the detrimental effect of the audio segmentation mismatch between training data manually segmented at sentence level and inference data that is automatically segmented. Towards the same goal of training cost reduction, we participate in the simultaneous task with the same model trained for offline ST. The effectiveness of our lightweight training strategy is shown by the high score obtained on the MuST-C en-de corpus (26.7 BLEU) and is confirmed in high-resource data conditions by a 1.6 BLEU improvement on the IWSLT2020 test set over last year’s winning system.

## introduction the yearly iwslt offline speech translation (st) evaluation campaign aims at comparing the models produced by companies, universities, and research institutions on the task of automatically translating speech in one language into text in another language. given a blind test set, participants' submissions are ranked according to the obtained sacre-bleu score @xcite . over the years, the competition to achieve the highest score has driven to bigger and bigger models trained on large datasets: the 2021 winning model @xcite has twice the number of encoder layers (12 vs 6), and a deeper (6 vs 4 layers) and larger (1024 vs 512 features) decoder the authors contributed equally. among the harms inherent to the high computational cost of training st systems, there is also the risk of restricting the participation in competitions like iwslt to few big players from the industry sectors that can afford them. as part of a research institution, with this work we try to answer the question: can we reduce the training cost of st systems without sacrificing final translation quality? specifically, can we train a competitive direct st model from scratch, without expensive pre-training (e.g. asr pre-training or self-supervised learning on huge dataset - @xcite )? to answer these questions, we perform a preliminary study on the english-german (en-de) section of must-c @xcite , one of the most widespread st corpora and then we scale to the high-resource data condition allowed by the task organizers. on must-c, we show that with the aid of a connectionist temporal classification (ctc) auxiliary loss @xcite and compression @xcite in the encoder, our conformerbased @xcite ) model can outperform -to the best of our knowledge -the previous best reported value of 25.3 bleu by @xcite , even avoiding any additional pre-training or transfer learning. moreover, with the addition of a simple data filtering method, we achieve the new state-of-the-art score of 26.7 bleu for a direct st model that does not exploit external (au-dio or textual) resources. scaling to high-resource data conditions, we notice that the gap between an asr pre-trained system and a system trained from scratch is closed only after a fine-tuning on in-domain data. our submission to the offline task consists of an ensemble of three models that scores 32.2 bleu on must-c v2 and 27.6 on iwslt tst2020. in the same vein of reducing the overall training computational costs, we participated also in the simultaneous task using our best offline model and without performing any additional training do adapt it to the simultaneous scenario @xcite . the simultaneous version of our offlinetrained model is realized by applying the wait-k strategy @xcite with adaptive word detection from the audio input @xcite ) that determines the number of words in a speech segment using the greedy prediction of the ctc. our simulst model achieves competitive results on the must-c v2 test set compared to the last year systems, scoring 25 bleu at medium latency (< 2s) and 30 bleu at high latency (< 4s) while keeping low (300 -400ms) the computation overhead and requiring no dedicated training. ## competitive st without pre-training before training systems on huge corpora, we conduct preliminary experiments on the must-c benchmark to find a promising setting aimed at reducing the high computational costs of st. first, we validate on different architectures the finding of previous works @xcite that st models trained with an additional ctc loss do not need an initialization of the encoder with that of an asr model. to this aim, we add a ctc loss @xcite whose targets are the lowercase transcripts without punctuation. 1 second, we explore data selection mechanisms to increase model quality and reduce training time. we always use the same hyper-parameters used in our final trainings for all systems (see section 6) unless otherwise specified. ## simultaneous in light of the recent work that questions the necessity of a dedicated training procedure for simultaneous model @xcite , we participate in the simultaneous task with the same model used for the offline task. their finding is perfectly aligned with the spirit of this submission toward the reduction of training computational costs. we determine when to start generating the output translation adopting the wait-k strategy @xcite that simply prescribes to wait for k words before starting to generate the translation, where k is a hyper-parameter controlled by the user that can be increased or decreased to directly control the latency of the system. the number of words in a given input speech is determined with an adaptive word detection strategy @xcite , because of its superiority over the fixed strategy @xcite in strong models trained in high-resource data conditions @xcite . our adaptive word detection mechanism exploits the predicted output of ctc module in the encoder @xcite to count the number of words in the source speech. the number of words to waitk -is not the only hyper-parameter that controls the wait-k strategy. another important factor is how often we check the number of uttered words that is the length of the speech segment. a short speech segment means that the system decides more frequently whether to wait for more input or to produce a part of output. this can reduce the latency, but it increases the number of forward passes through the encoder and hence the computational cost. in addition, a longer speech segment implies that the system takes decision with more context at disposal, possibly improving the quality. for this reason, we performed preliminary experiments exploring different speech segment dimensions (every 40ms ranging from 120ms to 720ms) and we found 320ms and 640ms to be superior to other values. accordingly, we report the results of our systems for these two speech segment durations varying the value of k to achieve different latency. in particular, we test our model with @xmath0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14} in order to lie in the latency intervals prescribed by the simultaneous shared task. 9 the latency intervals are determined by the average lagging @xcite ) -or al -on must-c v2 tst-common and are: low latency with @xmath1, medium latency with @xmath2, and high latency with @xmath3. we use a standard al-bleu graph to report the system performance, where in the x axis we find the al values ranging from 700ms to 4000ms and in the y axis the corresponding bleu values. moreover, we also report the al ca , the computational aware version of the al metric @xcite accounting also for the computational time spent by the model during inference, in an al ca -bleu graph that will be used to additionally score the performance in the simultaneous task. ## data as training set, we use the asr and st datasets allowed for the offline task, foot_11 which are the same allowed for the simultaneous one. the asr data consist in (speech, transcript) pairs that, in our case, are in english. the st data consist in (speech, transcript, translation) triplets from a source language (here english) to a target language (here german). the asr data we used are: librispeech @xcite , tedlium version 3 @xcite , voxpopuli @xcite , and mozilla common voice. 11 the st data we used are: must-c version 2 @xcite , cov-ost version 2 @xcite , and europarl-st @xcite . the asr-native corpora were included in our st training by applying sequence knowledge distillation @xcite ) -or seqkd -, a popular data augmentation method used in the past iwslt editions @xcite in which a teacher mt model is used to translate the source transcripts into the target language. to avoid additional computational costs, we choose as mt teacher the freely available pre-trained model by @xcite for wmt2021 that was trained on the corresponding wmt2021 dataset @xcite , allowed by the iwslt2022 offline task. the seqkd method was also applied to must-c v2 in order to augment the scarce st available data. as such, our training set comprised the synthetic data built using seqkd and the native st data, both filtered with the method described in section 2.2. the two types of data were distinguished by means of a tag pre-pended to the target text @xcite . ## experimental settings all the models used for our participation were implemented on fairseq-st @xcite . 12 all the architectures (transformer, speechformer, speechformer hybrid, and conformer) consist in 12 encoder layers and 6 decoder layers, 512 features for the attention layers and 2,048 hidden units in the feed-forward layers. we used 0.1 dropout for the feed-forward layer and attention layer. for conformer convolutional layers we also apply 0.1 dropout and we set the kernel size to 31 for the point-and depth-wise convolutions. we trained with the adam optimizer (kingma and ba, 2015) (β 1 = 0.9, β 2 = 0.98). the learning rate was set to increase linearly from 0 to 2e -3 for the first 25,000 warm-up steps and then to decay with an inverse square root policy. differently, it was kept constant for model fine-tuning, with a value of 1e-3. the vocabularies are built via sentencepiece models @xcite . in our preliminary experiments only on must-c, the number of merge operations was set to 8,000 @xcite for the german translations and 5,000 @xcite for the lowercase punctuation-free english transcripts. in the experiments on high-resource data condition, we doubled these values. we normalize the audio features before passing them to our models with cepstral mean and variance normalization. specifically, in offline st the mean and variance are estimated at utterance level, while for simultaneous st inference the normalization is based on the global mean and variance estimated on the must-c version 2 training set. trainings were performed on 4 nvidia a100 gpus with 40gb ram. we set the maximum number of tokens to 40k per mini-batch and 2 as update frequency for the conformer with ctccompression. the other models were trained with 20k tokens per mini-batch and 4 as update frequency. we trained each model for 100,000 updates, corresponding to about 28 hours for the conformer with ctc-compression. for offline generation, the maximum number of tokens was decreased to 25k, since we used a single k80 gpu with 12gb ram and we applied the beam search strategy with @xmath4. for simultaneous generation based on simuleval @xcite , we used a k80 gpu and greedy search. ## results in this section, we report our experiments in highresource data conditions and we discuss our submission to the offline (section 7.1 and simultaneous (section 7.2) tasks. ## conclusions we described the fbk participation in the iwslt 2022 offline and simultaneous tasks @xcite . our focus was to build a system with the least number of training steps but capable of obtaining competitive results with state-ofthe-art models, which typically undergo complex and longer training procedures. to this aim, we i) showed that asr pre-training of the encoder can be avoided without a significant impact on the final system performance, ii) proposed a simple yet effective data filtering technique to enhance translation quality while reducing the training time, and iii) compared different solutions to deal with automatic audio segmentation at inference time. our results on the iwslt2020 test set indicate that a single conformer-based model without pretraining falls behind our best model ensemble by only 1.2 bleu and outperforms the best score reported last year by 0.4 bleu. the same trend occurs on the blind tst2021 and tst2022 sets, with a 0.8-1.1 bleu gap from our best model ensemble, which in turn beats by ∼1 bleu the best reported result last year. these promising results are also confirmed in the simultaneous scenario in which, using the offline-trained model without any adaptation for the simultaneous task, we reach good quality-latency balancing, especially in the more realistic computational aware evaluation setting.