  Recently text and speech representation learning has successfully improved
many language related tasks. However, all existing methods only learn from one
input modality, while a unified acoustic and text representation is desired by
many speech-related tasks such as speech translation. We propose a Fused
Acoustic and Text Masked Language Model (FAT-MLM) which jointly learns a
unified representation for both acoustic and text in-put. Within this cross
modal representation learning framework, we further present an end-to-end model
for Fused Acoustic and Text Speech Translation (FAT-ST). Experiments on three
translation directions show that our proposed speech translation models
fine-tuned from FAT-MLM substantially improve translation quality (+5.90 BLEU).
