End-to-end Speech-to-text Translation ( E2E- ST ), which directly translates source language speech to target language text, is widely useful in practice, but traditional cascaded approaches (ASR+MT) often suffer from error propagation in the pipeline. On the other hand, existing end-to-end solutions heavily depend on the source language transcriptions for pre-training or multi-task training with Automatic Speech Recognition (ASR). We instead propose a simple technique to learn a robust speech encoder in a self-supervised fashion only on the speech side, which can utilize speech data without transcription. This technique , termed Masked Acoustic Modeling (MAM), can also perform pre-training , for the first time, on any acoustic signals (including non-speech ones) without annotation. Compared with current state-of-the-art models on ST , our technique achieves  + 1.4 BLEUimprovement without using transcriptions , and + 1.2 BLEU using transcriptions. The pre-training  of MAM with arbitrary acoustic signals also boosts the downstream speech-related tasks .
End-to-end Speech-to-text Translation ( E2E-ST ), which directly translates source language speech to target language text, is widely useful in practice, but traditional cascaded approaches (ASR+MT) often suffer from error propagation in the pipeline. On the other hand, existing end-to-end solutions heavily depend on the source language transcriptions for pre-training or multi-task training with Automatic Speech Recognition (ASR). We instead propose a simple technique to learn a robust speech encoder in a self-supervised fashion only on the speech side, which can utilize speech data without transcription. This technique termed Masked Acoustic Modeling (MAM), not only provides an alternative solution to improving E2E-ST, but also can perform pre-training on any acoustic signals (including non-speech ones) without annotation. We conduct our experiments over 8 different translation directions. In the setting without using any transcriptions , our technique achieves an average improvement of + 1.1 BLEU , and + 2.3 BLEU with MAM pre-training . Pre-training of MAM with arbitrary acoustic signals also has an average improvement with +1.6 BLEU for those languages. Compared with ASR multi-task learning solution, which replies on transcription during training, our pre-trained MAM model, which does not use transcription, achieves similar accuracy .
R
E2E- ST
E2E-ST
D
,
R
can also
not only provides an alternative solution to improving E2E-ST, but also can
D
, for the first time,
R
Compared with current state-of-the-art models on ST
We conduct our experiments over 8 different translation directions. In the setting without using any transcriptions
A
an average improvement of
R
1.4 BLEUimprovement without using transcriptions
1.1 BLEU
R
1.2 BLEU using transcriptions. The
2.3 BLEU with MAM
A
. Pre-training
R
boosts the downstream speech-related tasks
has an average improvement with +1.6 BLEU for those languages. Compared with ASR multi-task learning solution, which replies on transcription during training, our pre-trained MAM model, which does not use transcription, achieves similar accuracy