Por Qué Não Utiliser Alla Språk? Mixed Training with Gradient Optimization in Few-Shot Cross-Lingual Transfer (2022)

Abstract: The current state-of-the-art for few-shot cross-lingual transfer learning first trains on abundant labeled data in the source language and then fine-tunes with a few examples on the target language, termed target-adapting. Though this has been demonstrated to work on a variety of tasks, in this paper we show some deficiencies of this approach and propose a one-step mixed training method that trains on both source and target data with stochastic gradient surgery, a novel gradient-level optimization. Unlike the previous studies that focus on one language at a time when target-adapting, we use one model to handle all target languages simultaneously to avoid excessively language-specific models. Moreover, we discuss the unreality of utilizing large target development sets for model selection in previous literature. We further show that our method is both development-free for target languages, and is also able to escape from overfitting issues. We conduct a large-scale experiment on 4 diverse NLP tasks across up to 48 languages. Our proposed method achieves state-of-the-art performance on all tasks and outperforms target-adapting by a large margin, especially for languages that are linguistically distant from the source language, e.g., 7.36% F1 absolute gain on average for the NER task, up to 17.60% on Punjabi.

## introduction the cost of linguistic data annotation and a plethora of differences across language resources and structures of natural language processing (nlp) tasks result in the problem that sufficient labeled data is only available for a handful of high-resource languages @xcite . the lack of data for low-resource languages leads to the need for effective cross-lingual transfer learning, which aims to leverage abundant labeled high-resource languages to improve model performance on low-resource 1 code is available at: https://github.com/ fe1ixxu/mixed-gradient-few-shot . in this paper, we dissect the potential weaknesses of the ubiquitous target-adapting method and propose a one-step mixed training method that trains on both source and target data with a novel gradientlevel optimization, stochastic gradient surgery. specifically, we highlight 6 benefits (contributions) of our method in this paper as follows: ## conclusion we study the deficiencies of target-adapting in fewshot cross-lingual transfer and propose a mixed training method with gradient-level optimization. our best model achieves state-of-the-art on four diverse nlp tasks with all values of k. moreover, we are the first to use a single model to train all target languages and find that languages can benefit from others that share the same scripts. we also show the effectiveness of our method compared to target-adapting in a single target language case, and the gaps are still significant. finally, we propose only using source dev set in few-shot settings and show that our method is development-free for targets and also able to escape from overfitting issues.