  In recent years, pre-trained multilingual language models, such as
multilingual BERT and XLM-R, exhibit good performance on zero-shot
cross-lingual transfer learning. However, since their multilingual contextual
embedding spaces for different languages are not perfectly aligned, the
difference between representations of different languages might cause zero-shot
cross-lingual transfer failed in some cases. In this work, we draw connections
between those failed cases and adversarial examples. We then propose to use
robust training methods to train a robust model that can tolerate some noise in
input embeddings. We study two widely used robust training methods: adversarial
training and randomized smoothing. The experimental results demonstrate that
robust training can improve zero-shot cross-lingual transfer for text
classification. The performance improvements become significant when the
distance between the source language and the target language increases.
