Compact Speech Translation Models via Discrete Speech Units Pretraining (2024)

Abstract: We propose a pretraining method to use Self-Supervised Speech (SSS) model to creating more compact Speech-to-text Translation. In contrast to using the SSS model for initialization, our method is more suitable to memory constrained scenario such as on-device deployment. Our method is based on Discrete Speech Units (DSU) extracted from the SSS model. In the first step, our method pretrains two smaller encoder-decoder models on 1) Filterbank-to-DSU (Fbk-to-DSU) and 2) DSU-to-Translation (DSU-to-Trl) data respectively. The DSU thus become the distillation inputs of the smaller models. Subsequently, the encoder from the Fbk-to-DSU model and the decoder from the DSU-to-Trl model are taken to initialise the compact model. Finally, the compact model is finetuned on the paired Fbk-Trl data. In addition to being compact, our method requires no transcripts, making it applicable to low-resource settings. It also avoids speech discretization in inference and is more robust to the DSU tokenization. Evaluation on CoVoST-2 (X-En) shows that our method has consistent improvement over the baseline in three metrics while being compact i.e., only half the SSS model size.

## introduction in speech-to-text translation (st), using self-supervised speech (sss) models, such as wav2vec 2.0 and hubert @xcite , as model initialization is now common to obtain the sota result @xcite . nevertheless, such model initialisation makes the st model less memory-adaptive and could impose a large memory footprint. these factors hinders ondevice deployment that is crucial for privacy and useful in the absence of internet connection. how can we use the sss model(s) to create a more compact st model? when using the sss model for initialization, the corresponding st model uses the dense representations of the sss model for its task. alternatively, an informative proxy, which requires less memory to obtain, for the dense representation may make the st model more compact. discrete speech units (dsu) extracted from the sss model can be such a good proxy. dsu are k-means clusters of speech representations from selected layers of the sss model. it represents sequence of discrete tokens, which are easier to model within a text processing architecture @xcite . dsu sequences foot_0 are far smaller than the sequences of dense representations. therefore, a straightforward method to distill the sss models is to use dsu as speech inputs, aka the dsu-to-translation (dsu-to-trl) model. although using dsu as inputs allows for transfer learning and a memory-adaptive model, using them at inference still requires storing and calling the quantization modules, i.e, the sss model and the k-means model. we thus propose to use dsu for pretraining (pt) rather than as model input to make st models more compact. our method distils the sss model by pretraining smaller models on the corresponding dsu. more specifically, our method firstly pretrains two smaller encoder-decoder models on 1) filterbankto-dsu (fbk-to-dsu) and 2) dsu-to-trl data respectively. the dsu thus become the distillation inputs of the smaller models. subsequently, the encoder from the fbk-to-dsu model and the decoder from the dsu-to-trl model are taken to initialise the compact model. finally, the compact model is finetuned on the paired fbk-trl data. under this formulation, (1) we can use the sss model to create a st model that is adaptive to the memory footprint. we evaluate our method on covost-2 (wang et al., 2021) x-en language directions (21 in total) using multilingual st. by using a hubert-base model to extract the dsu, our method shows strong and consistent improvements in three evaluation metrics with respect to a st model that is trained from scratch. our main contributions are: ## related work there are a number of related works that use dsu to enhance st. @xcite in contrast, we resort to supervised training on the dsu in acoustic pretraining and require no extra module in inference. more importantly, our goal is to make (multilingual) st more compact, aiming also at low-resource settings where transcripts are not easily available, rather than learning a joint semantic space for both transcripts and audios. our method uses dsu in the form of pretraining to distil knowledge from the sss (dense) representations to creating a more compact st model. in the first step, our method pretrains two smaller encoder-decoder models on 1) fbk-to-dsu and 2) dsu-to-trl data respectively. the fbk-to-dsu model takes the log mel fbk as the encoder input and predicts the dsu sequence. the model is trained by an interpolation of connectionist temporal classification (ctc, @xcite ) loss that is applied to the last encoder layer and label-smoothed cross-entropy (ce) loss: next, we use the encoder of the fbk-to-dsu model and the decoder (and its output layer) of the dsu-to-trl model to initialise the compact model, followed by finetuning on the paired fbk-trl data using both ce and ctc loss @xcite on the translations: ## results and analysis before discussing the results, it is worth noting that (1) hu-transformer is not memory-adaptive, and (2) asr-pretraining requires transcripts, unlike dsu which is self-supervised. both methods are introduced for reference purposes of if such resources are available. ## limitations and future works in the previous sections, we discuss the noticeable benefits of our dsu-pretraining method in creating a more compact st model. in spite of this, there are several factors that are not thoroughly explored and could improve the model performance further: k-means clustering we did not inspect the clustering size (fixed to 1, 000) and the number of training instances (only fixed to 98, 000) used in training the k-means clustering model. apart from tuning its hyper-parameters, using other techniques, such as residual vector quantisation @xcite and multiple codebooks @xcite , might bring better improvement. other acoustic encoders we did not experiment other acoustic encoders, such as conformer @xcite ) and e-branchformer @xcite . this stronger encoders should provide further gains for our method since they also enjoy the benefit of pretraining. a stronger pretrained decoder apart from strengthening the encoder, the dsu-to-trl model and hence its decoder (used in initialisation) could also be improved, e.g. via back-translation, upsampling the textual sequence @xcite and pretraining with more text data, while maintaining the small decoder size. further analyses in addition to improving our pretraining method for better model compactness, there are other related research directions worth further analyzing. one direction would be how, in terms of acoustic pretraining, dsu compared with transcripts (if available in that language) over different data scales. another interesting research direction would be the comparison and analysis of using dsu or dense features in a large pretrained model setting, such as whisper @xcite and large language models. ## conclusion in this paper, we consider a memory-constrained setting for st. our proposed method uses dsu in the form of pretraining to distil the knowledge from the self-supervised speech model to creating more compact speech-to-text translation. our compact model, i.e., the dsu-adapter, shows strong and consistent improvements in three evaluation metrics over the baselines. in contrast to using dsu as model inputs, our method does not require quantization modules in inference and shows stronger robustness to the dsu tokenization. finally, our method requires no transcripts, making it also suitable for low-resource setting.