  In this paper, we propose a weakly supervised multilingual representation
learning framework, called cross-lingual self-training (XLST). XLST is able to
utilize a small amount of annotated data from high-resource languages to
improve the representation learning on multilingual un-annotated data.
Specifically, XLST uses a supervised trained model to produce initial
representations and another model to learn from them, by maximizing the
similarity between output embeddings of these two models. Furthermore, the
moving average mechanism and multi-view data augmentation are employed, which
are experimentally shown to be crucial to XLST. Comprehensive experiments have
been conducted on the CommonVoice corpus to evaluate the effectiveness of XLST.
Results on 5 downstream low-resource ASR tasks shows that our multilingual
pretrained model achieves relatively 18.6% PER reduction over the
state-of-the-art self-supervised method, with leveraging additional 100 hours
of annotated English data.
