Multi-task pre-finetuning for zero-shot cross lingual transfer (2021)

Abstract: Building machine learning models for low resource languages is extremely challenging due to the lack of available training data (either un-annotated or annotated). To support such scenarios, zero-shot cross lingual transfer is used where the machine learning model is trained on a resource rich language and is directly tested on the resource poor language. In this paper, we present a technique which improves the performance of zero-shot cross lingual transfer. Our method performs multi-task pre-finetuning on a resource rich language using a multilingual pre-trained model. The pre-finetuned model is then tested in a zero-shot manner on the resource poor languages. We test the performance of our method on 8 languages and for two tasks, namely, Intent Classification (IC) & Named Entity Recognition (NER) using the MultiAtis++ dataset. The results showed that our method improves IC performance in 7 out of 8 languages and NER performance in 4 languages. Our method also leads to faster convergence during finetuning. The usage of pre-finetuning demonstrates a data efficient way for supporting new languages and geographies across the world.

## introduction recent advances in natural language processing include the development of language models trained in an unsupervised fashion on large amounts of data @xcite , @xcite . these models were extended to a multilingual context by training on data from a number of languages @xcite , @xcite . these multilingual models were found to perform well in a zero-shot cross lingual transfer tasks -i.e. the multilingual model is finetuned for a task in a resource rich language (such * *equal contribution as english) and is directly tested on a resource poor language (such as swahili) @xcite in this paper, we investigate the usage of multitask pre-finetuning in zero-shot cross lingual tasks. pre-finetuning is a step between pre-training and fine-tuning, where a pre-trained model is trained on additional supervised learning tasks with the aim to improve pre-trained representations @xcite . typically, these additional tasks are unrelated to the tasks that the model will finally be fine-tuned on. past work on multi-task learning showed its usefulness in a monolingual setting. in our work, we extend the pre-finetuning concept to a multilingual setting with zero-shot cross lingual transfer. our method is demonstrated on the xlm-roberta @xcite pre-trained model and uses 8 additional auxiliary tasks from the glue benchmark @xcite for the multitask pre-finetuning on english. the resulting model is then applied for the joint intent-classification & named entity recognition tasks (ic-ner). we show cross lingual transfer by fine-tuning for ic-ner on english and directly test on 8 different languages in a zero-shot manner. we use the mul-tiatis++ dataset @xcite for the ic-ner data in english and 8 other languages for finetuning and testing. our results bring out multiple insights. we find that the multi-task pre-finetuned model is better by 5.12% relative (or 391 absolute basis points (bps)) on average across all 8 languages for the ic task at early stages of fine-tuning. this shows the ability of multitask pre-finetuning to improve the learnt representations of the pretrained model. the results however indicate that such out of the box improvement in pre-trained models is not seen in the ner task, where the performance at early stages of fine-tuning degrades by 10% relative (or 525 bps absolute). we also find that pre-finetuning has improved the performance in monolingual setting where the results for both ic and ner for english have improved by 6.53% relative (or 549 bps absolute) and 11.86% relative (or 700 bps absolute) respectively. improving the ability of cross lingual transfer in a zero shot setting has large implications. through our method, products using language models can improve machine learning support for new languages where sufficient training data is not available. for example, the indian sub-continent has 179 languages and 544 dialects (wikipedia contributors, 2021b) and building machine learning services to cater to all multilingual users is a daunting task since enough data, both in the form of annotated and un-annotated, is not available. this paper is structured as follows. in section 2, we present the related work in zero-shot cross lingual transfer and progress in multi-task pre-finetuning. section 3 presents our method of improving zero-shot cross lingual transfer through the use of multi-task pre-finetuning. we present the results in section 4 and conclude with directions for future work in section 5. ## related work multilingual language models such as mbert @xcite , xlm @xcite and muril @xcite have advanced the state-of-the-art on cross-lingual natural language understanding tasks by training large transformer models @xcite on data from many languages. these multilingual pre-trained models can be used as generic task agnostic neural network architectures and can be applied in different natural language processing tasks by attaching a task specific decoder (such as a linear classifier) and fine-tuning on the task specific training data. the usage of multilingual pre-trained models also enabled zero-shot cross lingual transfer. in zero-shot transfer, the pre-trained model is fine tuned on annotated data from a resource rich language (such as english) and directly tested in a resource poor language (such as swahili). studies @xcite @xcite have shown such multilingual pre-trained models have the ability to learn common representations across languages even though the training methodology was not explicitly designed to build common representations. to improve the performance of pre-trained models in the monolingual setting. prominent work includes the usage of multi-task pre-finetuning @xcite ) @xcite . in prefinetuning, the pre-trained model such as bert is trained in a supervised learning setting on auxiliary tasks. the work in @xcite ) uses 8 different tasks from the glue dataset @xcite and trains for different tasks in random batches. the work in @xcite scales the pre-finetuning concept by training on 50 different tasks. their results show that multi-task pre-finetuning can significantly improve the performance of the pre-trained models. both these works test the improvement in a monolingual setting. in contrast, our work in this paper explores the applicability of pre-finetuning in a multilingual setting. in this section, we present our method to improve the performance of zero-shot transfer of knowledge across languages. current state-of-the-art methods have demonstrated the great ability of a neural network to transfer knowledge of a given task across languages using a pre-trained model that is trained generically @xcite , @xcite , @xcite . in our work, we demonstrate a further intriguing property of deep neural networks. we take a multi-lingual pre-trained neural network and train it over multiple tasks in english using task specific supervised data (we choose english since there is significant amount of supervised training data available for the language). we then fine tune the network for the specific down-stream task in english. we denote the resulting neural network model through our method as mt-dnn-multilingual-finetuned. now, when the mt-dnn-multilingual-finetuned model is tested directly (i.e. zero-shot) on the downstream task in a different language, we observe better performance. this demonstrates the ability of the neural network to not only transfer knowledge of the same task across languages (as demonstrated by existing literature), but also shows its ability to transfer cumulative knowledge of multiple different tasks across languages. figure 1 depicts the novelty of our work in comparison with extant literature. ## conclusion and future directions in this work, we have investigated the effectiveness of multi-task pre-finetuning for cross lingual zero-shot transfer. our method takes a multilingual pre-trained model and further trains it on auxiliary supervised tasks. the pre-finetuned model is then finetuned on a task specific language and tested directly on other languages in a zero-shot setting. we test our method for the tasks of intent classification (ic) and named entity recognition (ner). the results indicate that the method indeed improves the performance for the ic task. this improvement is seen the most in early steps of finetuning and our method allows the training to converge faster. however, we see that the pre-finetuning does not improve results for ner. further, we see that both ic and ner results degrade in turkish. our furture directions include scaling our method to cover large number of auxiliary tasks for pre-finetuning. while our current method used 8 auxuliary tasks, we aim to scale this to beyond 50. large scale multi-task learning has been shown to be effective in a monolingual setting @xcite and we would like to explore this phenomenon in a multilingual setting. we will also explore the role of language families and its interaction with multi-task learning to test the hypothesis of poor performance in agglutinative languages (such as turkish). the pratical application of zero-shot learning provides a data-efficient method to expand the language capability of machine learning based techniques. the results from our technique show that such zero-shot performance can be further improved and also provide impetus for further research.