Improved Multilingual Language Model Pretraining for Social Media Text via Translation Pair Prediction (2021)

Abstract: We evaluate a simple approach to improving zero-shot multilingual transfer of mBERT on social media corpus by adding a pretraining task called translation pair prediction (TPP), which predicts whether a pair of cross-lingual texts are a valid translation. Our approach assumes access to translations (exact or approximate) between source-target language pairs, where we fine-tune a model on source language task data and evaluate the model in the target language. In particular, we focus on language pairs where transfer learning is difficult for mBERT: those where source and target languages are different in script, vocabulary, and linguistic typology. We show improvements from TPP pretraining over mBERT alone in zero-shot transfer from English to Hindi, Arabic, and Japanese on two social media tasks: NER (a 37% average relative improvement in F1 across target languages) and sentiment classification (12% relative improvement in F1) on social media text, while also benchmarking on a non-social media task of Universal Dependency POS tagging (6.7% relative improvement in accuracy). Our results are promising given the lack of social media bitext corpus. Our code can be found at: https://github.com/twitter-research/multilingual-alignment-tpp .

## introduction multilingual bert (mbert; @xcite , and other multilingual pretrained language models have been shown to learn surprisingly good cross-lingual representations for a range of nlp tasks @xcite @xcite , despite not having explicit cross-lingual links between their monolingual training corpora @xcite . analysis in @xcite shows that cross-lingual transfer with mbert works best when transferring representations between source mbert en ar mbert + tpp @xcite . these embeddings exhibit distinct language regions, making cross-lingual transfer challenging. the right figure shows the same data after further pretraining using translation pair prediction (figure 2 ), which learns representations that are near cross-lingual translations (change in distance showed in figure 3 of appendix). in section 3, we show this improves cross-lingual transfer performance on several tasks (see in this work, we explore whether we can improve multilingual transfer by providing cross-lingual examples and encouraging mbert to learn aligned representations between source and target texts before task fine-tuning. concretely, we perform a "language pretraining" task called translation pair prediction (tpp) where we assume access to a corpus of source-target text pairs and predict whether the pair is a translation of the same content (see section 2). this pretraining phase is intended to occur after standard mbert pretraining @xcite , but before task fine-tuning (see fig -ure 2 ). the intent behind this task is to leverage translation data that exists between a source language with abundant task training data and a target language with little or no task training data in order to improve transfer learning. as with standard mbert pretraining, one can pretrain mbert using tpp once and fine-tune for multiple tasks (as long as transferring to the same target language set). the translation pair data doesn't need to be related to the downstream task and we can leverage multiple sources of source-target translations of varying degrees of translation quality (see section 3.3). we demonstrate the benefits of adding tpp pretraining in experiments on social media corpus for ner, and sentiment detection, along with a nonsocial media corpus of universal pos tagging (for comparison) on fine-tuned models from english to hindi, japanese, and arabic (see section 3 for evaluation). we show gains from tpp on all tasks (averaged across target languages) using translation data of varying quality (see section 3.3). @xcite argues for the benefits of continued transformer pretraining for better performance on in-domain tasks. our work closely follows this approach however instead of keeping the masked language modelling (mlm) task on in-domain data, we introduce a new pretraining task (tpp). in @xcite the authors train an bilstm encoder-decoder model on a large parallel corpora of multiple languages and demonstrate zero-shot transfer. in @xcite the authors use a domain/task specific fine-tuning to improve zero shot cross lingual performance by inducing labels from the model of @xcite . our work is closely related to the work of @xcite ; @xcite which use a subword averaging and a lstm encoder. our approach uses transformers and also focuses evaluation on individual token level task like ner as well as sentence level task. another closely related work is of @xcite which uses word pair alignment with mbert and evaluates on xnli task. however, their work does not consider typologically different languages like this work. in @xcite the authors use translation language modeling task which concatenate the translation pairs in the bert model and also add a language identifier to each token. huang et al. (2019) include a similar approach as discussed before and get improvement from jointly fine-tuning model with translations of downstream task data. our approach is comparatively simpler to implement as we don't make any assumption about the encoding model or assumptions about word-level alignments; our approach works independently of the choice of encoder (mbert here) or the quality of word-alignment models. ## model a description of our approach is provided in figure 2 . the primary exploration in this work is the impact of adding a translation pair prediction (tpp) pretraining phase between traditional mbert pretraining and downstream task fine-tuning @xcite . the tpp pretraining task assumes a collection of aligned text pairs (s, t) which are "translation" pairs between source and target language text pairs s and t respectively. it is assumed that both languages have word-piece coverage in the pretrained mbert model, but that we primarily have task training data for the source language. after tpp pretraining, we can fine-tune the resulting model for multiple tasks as with standard mbert for transfer to the target language(s). the tpp task is a binary classification problem where given an (s, t) pair we predict whether it is a valid "translation" pair or a random source-target pair using the dot product of mbert encodings for s and t. 1 more formally, if f (•) represents the encoder mapping a document to the mbert [cls] token embedding, we predict if the pair is a translation pair with probability: after tpp pretraining, we fine-tune on task data in the source language (english in our experiments) and evaluate transfer performance on target languages (hindi, japanese, and arabic in our experiments). ## conclusion we evaluated the use of translation pair prediction (tpp) as a pretraining task which can improve the performance of multilingual language models for cross-lingual zero-shot transfer from english. this pretraining task is performed after standard mbert pretraining and the resulting model can be task fine-tuned for cross-lingual transfer similar to mbert. we show significant improvement in zero-shot transfer by adding tpp pretraining on ner and sentiment detection on social media tasks for three languages (hindi, japanese, and arabic) which are dissimilar in vocabulary and syntax from english.