Back Translation for Speech-to-text Translation Without Transcripts (2023)

Abstract: The success of end-to-end speech-to-text translation (ST) is often achieved by utilizing source transcripts, e.g., by pre-training with automatic speech recognition (ASR) and machine translation (MT) tasks, or by introducing additional ASR and MT data. Unfortunately, transcripts are only sometimes available since numerous unwritten languages exist worldwide. In this paper, we aim to utilize large amounts of target-side monolingual data to enhance ST without transcripts. Motivated by the remarkable success of back translation in MT, we develop a back translation algorithm for ST (BT4ST) to synthesize pseudo ST data from monolingual target data. To ease the challenges posed by short-to-long generation and one-to-many mapping, we introduce self-supervised discrete units and achieve back translation by cascading a target-to-unit model and a unit-to-speech model. With our synthetic ST data, we achieve an average boost of 2.3 BLEU on MuST-C En-De, En-Fr, and En-Es datasets. More experiments show that our method is especially effective in low-resource scenarios.

## introduction end-to-end speech-to-text translation (st) means directly translating speech in the source language to target text without generating source transcripts @xcite . different from traditional cascading methods which first transcribe the speech with automatic speech recognition (asr) and then translate the transcripts into the target text with machine translation (mt), end-to-end st has the potential to reduce latency and avoid error propagation. hence, it has drawn much attention and achieved great suc-cess in recent years @xcite . however, it is challenging to train an end-to-end st model with only speech-translation pairs. traditional cascaded models learn cross-modal mapping with asr and cross-lingual mapping with mt. in contrast, end-to-end st requires simultaneous cross-modal and cross-lingual mapping, which is more complicated and usually relies on more training data. however, the amount of st data is usually limited due to the high cost of data collection, so the st model trained with only speech-translation pairs is usually unsatisfactory. to tackle these problems, researchers often utilize source transcripts to assist st training by introducing auxiliary asr and mt tasks. with abundant asr and mt data, the asr task can help the model learn better cross-modal mapping, while the mt task can help learn better cross-lingual mapping, which can significantly improve st as shown in recent st studies @xcite @xcite . unfortunately, source transcripts are not always available. it is estimated that there are around 3000 unwritten languages in the world which have no orthography for transcription 3 . for those languages, we can no longer leverage source transcripts to help with training, so many of the latest techniques fail to benefit them. how to train a stronger st model without transcripts? in this paper, we address this question from the perspective of data augmentation. although asr and mt data are unavailable, a large amount of target-side monolingual data is still easily accessible. motivated by the success of back translation in mt @xcite , we aim to develop a back translation algorithm for st (bt4st) to synthesize pseudo st data from monolingual target data. however, compared to text-to-text back translation, gener-ating source speech from the target text without source transcripts is much more challenging foot_1 . first, the length of text is usually only tens or hundreds, but the length of speech is about tens of thousands 5 . therefore, the model is required to generate an extremely long sequence from a short sequence without the assumption of monotonic alignment, which is a more difficult sequential decision problem. second, the conversion from text to speech is a one-to-many mapping problem due to the variations in speech @xcite . for example, the pronunciation of the same content may differ among speakers. to address these challenges, we introduce selfsupervised discrete units of the source speech as intermediate representations, and achieve back translation by cascading a target-to-unit model and a unit-to-speech model. since the length of unit sequences is similar to the length of target characters foot_3 , and there is a monotonic assumption between the unit sequence and the source speech, the shortto-long generation problem in target-to-unit and unit-to-speech models can be greatly alleviated. besides, discrete units can disentangle content information from other variation information (e.g., pitch and speaker) @xcite , which eases the one-to-many mapping problem in the targetto-unit model foot_4 . we also introduce a speaker encoder to provide speaker information as input to the unit-to-speech model, which allows us to generate diverse source speeches by coupling different speaker representations. following this pipeline, we can synthesize large amounts of pseudo st data from monolingual target data without requiring transcripts. finally, we train our st model with both synthetic and real data. we conduct experiments on must-c @xmath0, @xmath1, and @xmath2. by leveraging about 5m additional monolingual target data for each language pair, we achieve an average improvement of 2.3 bleu compared with the strong baseline. we also observe that our approach is more effective in low-resource scenarios, yielding a boost of 5.6 bleu when only 100 hours of st data are available. in addition, we generate multiple different pseudo datasets with a simple diverse bt4st method, train several models separately with each dataset, and ensemble them together. by ensembling five models, we achieve an average boost of 4.0 bleu in three translation directions. ## background our work focuses on developing a back translation algorithm for st. we first introduce the task definition and model architecture of st in section 2.1, and then introduce the concept of back translation in section 2.2. ## method: bt4st how to acquire a target-to-source model for st given a st parallel corpus? directly generating source speech from the target text is a challenging problem. inspired by recent success in self-supervised discrete representation learning for speech @xcite @xcite , we first transform the source speech into a sequence of discrete units with a speech pre-trained model (section 3.1), which is used as an intermediate representation in the back translation process. with discrete units, we train a target-to-unit model (section 3.2) and a unit-tospeech model (section 3.3) on the parallel corpus, where the former predicts the sequence of discrete units corresponding to the source speech, and the latter converts discrete units into waveform. the model architecture is illustrated in figure 1 . ## conclusion and future work in this paper, we develop a back translation algorithm for speech translation, which can synthesize pseudo st data from monolingual target data without relying on transcripts. we utilize selfsupervised discrete units and achieve back translation by cascading a target-to-unit model and a unit-to-speech model. experimental results on the must-c benchmark demonstrate the superiority of our approach, especially in low-resource settings. this work focuses on enhancing st when the source transcripts are unavailable, which is an essential but under-explored issue. we hope our work will draw more attention to this issue from researchers, which will benefit more real-world unwritten languages. in the future, we are interested in exploring how to combine advanced bt techniques (e.g., iterative bt) with our approach.