Pretrained Speech Encoders and Efficient Fine-tuning Methods for Speech Translation: UPC at IWSLT 2022 (2022)

Abstract: This paper describes the submissions of the UPC Machine Translation group to the IWSLT 2022 Offline Speech Translation and Speech-to-Speech Translation tracks. The offline task involves translating English speech to German, Japanese and Chinese text. Our Speech Translation systems are trained end-to-end and are based on large pretrained speech and text models. We use an efficient fine-tuning technique that trains only specific layers of our system, and explore the use of adapter modules for the non-trainable layers. We further investigate the suitability of different speech encoders (wav2vec 2.0, HuBERT) for our models and the impact of knowledge distillation from the Machine Translation model that we use for the decoder (mBART). For segmenting the IWSLT test sets we fine-tune a pretrained audio segmentation model and achieve improvements of 5 BLEU compared to the given segmentation. Our best single model uses HuBERT and parallel adapters and achieves 29.42 BLEU at English-German MuST-C tst-COMMON and 26.77 at IWSLT 2020 test. By ensembling many models, we further increase translation quality to 30.83 BLEU and 27.78 accordingly. Furthermore, our submission for English-Japanese achieves 15.85 and English-Chinese obtains 25.63 BLEU on the MuST-C tst-COMMON sets. Finally, we extend our system to perform English-German Speech-to-Speech Translation with a pretrained Text-to-Speech model.

## introduction in the last few years, end-to-end (or direct) speech translation (st) models have gained popularity in the research community. these systems differ from the classical cascade ones in their architecture, where instead of concatenating an automatic speech recognition (asr) model and a machine translation (mt) system, they directly translate * equal contribution speech into the target language without an intermediate transcription. this approach solves some limitations of cascade st systems, like error propagation and slow inference times. but on the other hand, such approaches require more data to be competitive, which are not as abundant as asr and mt data @xcite . however, the performance gap between the two approaches has become very small in the last years @xcite , with end-to-end approaches having the best performances for the iwslt 2020 test set in the last two evaluation campaigns @xcite . following this research trend, we participate in the offline speech translation task of iwslt 2022 @xcite with end-to-end systems that are built on top of our last year's submission @xcite . the approach we follow is to leverage large pretrained speech and text models, in order to reduce the required amount of data usually needed to train competitive end-toend st systems ( §2.1). as a speech encoder, we consider wav2vec 2.0 @xcite and hubert @xcite , both already fine-tuned on english asr data. as a text decoder, we use an mbart50 @xcite fine-tuned on multilingual mt (one-to-many). these two modules are coupled with a length adaptor block, that reduces the length discrepancy. although powerful, combining these modules results in a substantially large system, that is hard to train on normal hardware, given its computational and memory requirements. we thus follow a minimalistic fine-tuning strategy @xcite , which trains only specific modules in the network ( §2.2). in addition, we extend this approach by adding parallel adapters @xcite to the frozen layers ( §2.3). we also explore the use of knowledge distillation @xcite from mt @xcite with mbart as the teacher ( §2.4). finally, we use shas @xcite to approximate the optimal segmentation for the iwslt test sets ( §5). in summary, our contributions with this work are: (1) we perform a comparison of wav2vec 2.0 and hubert for building an st model. (2) we extend the fine-tuning strategy proposed by @xcite with parallel adapters. (3) we study the effect of knowledge distillation for st, in the context of pre-trained models. ## methodology in this section, we describe the main parts of the proposed system 1, along with our approach for knowledge distillation and the text-to-speech model. ## experiments here we describe the experiments we carried out in this work with their implementation details. ## audio segmentation although our training data contain ground truth segmentations derived from strong punctuation of the transcriptions, the iwslt test sets, are unsegmented and thus require an intermediate step of au-dio segmentation, before applying our st models. past evaluation campaigns of iwslt have shown light to the importance of accurate audio segmentation for end-to-end st, where top-performing participants used their own segmentation algorithms to get large improvements in translation quality. for our submission, we are using shas, a segmentation method that can effectively learn the manual segmentation from a labelled speech corpus @xcite . it relies on a segmentation frame classifier and a probabilistic divide-and-conquer (pdac) algorithm to obtain the segmentation for a given audio. the frame classifier is a transformer encoder with a binary classification layer, that predicts the splitting frames in the audio using as inputs contextual representations extracted with a frozen xls-r @xcite . the pdac segmentation algorithm is based on the method of @xcite and progressively splits on the frames of the lowest probability, until all resulting segments are shorter than a prespecified max-segment-length parameter. segmentations created with shas approach the translation quality of the manual segmentation on the en-de tst-common set of must-c v2.0, retaining 95% of the manual bleu. we used the public implementation of shas foot_8 and tested two available pretrained models for the frame classifiers, one trained on english source audio from must-c v2 and a multilingual which is additionally trained on spanish, french, portuguese, and italian data from mtedx @xcite . we obtain the frame probabilities for the audios of the 2019 iwslt test set @xcite with the english and multilingual classifiers, and then used the pdac algorithm with a varying max-segment-length to segment them. to find the best parameters, we maximize the translation quality of the segmentation by the following process: (1) translate the resulting segments with our st model, (2) align the translations with the references using the mwersegmenter tool @xcite and (3) compute the bleu and ter scores. in figure 2 we observe that values of maxsegment-length in the range of 14 and 20 seconds for pdac, result in the best segmentation, with bleu scores of 22.5 and ter scores of 61.5. additionally, in that range, shas with a multilingual classifier performs better than the english one, with small improvements of approximately 0.2 bleu. the highest bleu score overall is obtained with the multilingual classifier and at maxsegment-length of 20 seconds, but given that there is an increase in the ter score, we decided to continue with max-segment-length of 16 seconds, which seems to have more consistent results. thus, for our final results ( §6) for the test sets of 2019 and 2020, as well as for our submissions for 2021 and 2022, we used shas with the multilingual classifier and a max-segment-length of 16 seconds (shas-mult-16). due to the absence of available test sets to fine-tune shas for the japanese and chinese, we also use shas-mult-16 to segment the en-ja and en-zh iwslt 2022 test sets. ## results in this section, we analyze the results of our experiments. we base our experimentation on the en-de language pair, to compare the results with our last year's submission @xcite . hence, first we analyze the results for this language pair ( ## speech-to-speech we have also submitted our system to the speechto-speech (s2s) translation task 12 , by building a cascade system. this is composed of the main endto-end speech-to-text translation model and a textto-speech (tts) system. we used a pretrained foot_12 vits model @xcite for synthesizing the german speech. it is based on normalizing flows @xcite , adversarial training and a stochastic duration predictor. it is capable of generating speech in different pitches and rhythms, resulting in more natural sounding audio utterances. ## conclusions we described the submission of the upc machine translation group for the iwslt 2022 offline st and speech-to-speech tasks. our system is end-toend and leverages asr and mt pretrained models to initialize the encoder and decoder. due to the large size of the system, we employed efficient fine-tuning methods that train only specific layers and provide evidence that the addition of parallel adapters to the non-trainable layers can bring further improvements. we showed that a hubert encoder is more suitable than wav2vec 2.0 for our system and brings improvements in all test sets. we also explored the use of knowledge distillation, which provided only minor improvements to the test sets with ground-truth segmentations, most likely because the mt model was borderline better than our st model. additionally, we show that the shas method provides high-quality segmentations of the iwslt test sets, bringing improvements up to 5 bleu compared to the given segmentation. our best single model, uses a hubert encoder and lna with parallel adapters, and achieved 29.42 bleu on must-c tst-common set, and 24.88 and 26.77 bleu on iwslt 2019 and iwslt 2020 test sets. we ensembled 4 different systems for our final submission, which further increased the bleu in the aforementioned sets by 1 to 1.5 points. we also described our submissions for the english-japanese and english-chinese pairs that scored 15.85 and 25.63 must-c tst-common. finally, we also submitted a speech-to-speech system, by using a pretrained german tts model to the generated translations. for future work, we are planning to explore more pretrained speech encoders and text decoders, and dive deeper into the ways that we can optimally combine them and efficiently fine-tune for end-toend st. we will also investigate how to gain the most from an mt teacher, in such scenarios where there is a small gap between the mt and the st models.