DUB : Discrete Unit Back-translation for Speech Translation (2023)

Abstract: How can speech-to-text translation (ST) perform as well as machine translation (MT)? The key point is to bridge the modality gap between speech and text so that useful MT techniques can be applied to ST.Recently, the approach of representing speech with unsupervised discrete units yields a new way to ease the modality problem. This motivates us to propose Discrete Unit Back-translation(DUB) to answer two questions (1) Is it better to represent speech with discrete units than with continuous features in direct ST? (2) How much benefit can useful MT techniques bring to ST? With DUB, the back-translation technique can successfully be applied on direct ST and obtains an average boost of 5.5 BLEU on MuST-C En-De/Fr/Es. In the low-resource language scenario, our method achieves comparable performance to existing methods that rely on large-scale external data. Code and models are available at https://anonymous.4open.science/r/DUB/ .

## introduction speech-to-text translation (st) converts the spoken source language into the written target language, which is a closely related task to machine translation (mt). in recent years, direct st that does not rely on intermediate transcription has received considerable attention due to its potential applications in unwritten language scenarios and various domains @xcite @xcite @xcite @xcite . one of the major challenges faced by st is data scarcity, which is similar to the low-resource scenarios encountered in mt. intuitively, techniques developed for low-resource mt @xcite @xcite ; tang † work was done while dong zhang was a research intern at bytedance. * corresponding authors. recently, representing speech with unsupervised discrete units has become popular and successful in the field of speech processing @xcite @xcite . instead of losing relevant information, discretizing continuous speech has been found to have the advantage of filtering out extraneous signals @xcite , leading to significant improvements in the speech tasks, such as automatic speech recognition @xcite ), text-to-speech @xcite , and speech-to-speech translation @xcite @xcite . based on this observation, we are motivated to explore the answers to the following two questions: (1) is it better to represent speech with discrete units and use them as model input than with continuous features for direct st? (2) by narrowing the modality gap with discrete speech units, how much benefit can useful mt techniques bring to direct st? in this paper, we propose discrete unit backtranslation (dub), which migrates the useful backtranslation technique from mt to st by discretizing the speech signals into unit sequences. in our proposed method, we first convert speech into discrete units using the clustering indices on hu-bert @xcite representations. to complete the translation task, we feed the discrete units into the unit-to-text translation (u2tt) model. for the back-translation training strategy, dub employs a text-to-unit translation model that learns to predict the source discrete units from the target text. by leveraging the additional easily accessible text in the target language, we utilize the synthetic parallel data generated by the text-to-unit translation model in conjunction with the original parallel data to update the final unit-to-text model. our contributions include the following. back-translation back-translation (bt) is a widely used method for improving machine translation systems by training a target-to-source model and creating synthetic parallel data from monolin-gual target text. this approach has been shown to be effective in both statistical @xcite and neural machine translation models @xcite @xcite , and is frequently used to improve translation performance in wmt competitions @xcite @xcite . a similar data augmentation idea through synthesizing speech from utterances can be applied to automatic speech recognition (asr) @xcite @xcite . however, applying bt in st is not trivial. @xcite augmented the triplet data by tts generation from transcription, but the experiment showed that such scaling yields minimal improvement to the final st model. discrete speech representation discrete speech representations are often studied in the work on self-supervised speech representation learning @xcite @xcite @xcite . for example, van den oord et al. (2017) proposed vector quantised-variational autoencoder (vq-vae) to map continuous signals, like speech or image, into a discrete sequence space. hsu et al. (2021) proposed hubert, which learns self-supervised speech representation by extracting speech features and clustering them offline, and iteratively training the clustering indexes of features at masked locations. although the clustered discrete representations are only a by-product of hubert, they are used to build the generative spoken language model @xcite , enhance speech representation @xcite @xcite @xcite , and model direct speech-to-speech translation @xcite . in the prior literature, probably the most similar task to ours is the textless speechto-speech translation @xcite , but the difference is that they discretized the target-side speech and convert speechto-speech generation into speech-to-discrete-unit generation, while we discretize the speech at the source side. @xcite leveraged the discrete unit as an interface to align speech and text, and proposed a unified-modal encoder-decoder pretraining model, speechut. speechut aims to improve speech representation via the units, while we use the units to construct a unified framework for ## why does dub work? -analysis on the property of discrete unit why does dub work? to answer this question, we examine the properties of the discretized speech unit. specifically, (1) do the units make sense to replace the original speech input in the forward translation process? (2) do the units generated by back translation also contain semantic information and can they even restore the speech? without extra text, the bleu score is 20.9, which is slightly higher than hubert, but the gap is quite small. ## conclusion in this paper, we propose discrete unit backtranslation (dub), as well as the unit-to-text generation method asr-bleurt beam search 0.60 sampling 0.47