Meta-Learning for Effective Multi-task and Multilingual Modelling (2021)

Abstract: Natural language processing (NLP) tasks (e.g. question-answering in English) benefit from knowledge of other tasks (e.g., named entity recognition in English) and knowledge of other languages (e.g., question-answering in Spanish). Such shared representations are typically learned in isolation, either across tasks or across languages. In this work, we propose a meta-learning approach to learn the interactions between both tasks and languages. We also investigate the role of different sampling strategies used during meta-learning. We present experiments on five different tasks and six different languages from the XTREME multilingual benchmark dataset. Our meta-learned model clearly improves in performance compared to competitive baseline models that also include multi-task baselines. We also present zero-shot evaluations on unseen target languages to demonstrate the utility of our proposed model.

## introduction multi-task and multilingual learning are both problems of long standing interest in natural language processing. leveraging data from multiple tasks and/or additional languages to benefit a target task is of great appeal, especially when the target task has limited resources. when it comes to multiple tasks, it is well-known from prior work on multitask learning @xcite @xcite that jointly learning a model across tasks can benefit the tasks mutually. for multiple languages, the ability of deep learning models to learn effective embeddings has led to their use for joint learning of models across languages @xcite @xcite ; learning cross-lingual embeddings to aid languages in limited resource settings is of growing interest @xcite @xcite . let us say we had access to m tasks across n different languages -c.f. in this work, we adopt a meta-learning approach for efficiently learning parameters in a shared parameter space across multiple tasks and multiple languages. our chosen tasks are question answering (qa), natural language inference (nli), paraphrase identification (pa), part-of-speech tagging (pos) and named entity recognition (ner). the tasks were chosen to enable us to employ a gamut of different types of language representations needed to tackle problems in nlp. in figure 1 , we illustrate the different types of representations by drawing inspiration from the vauquois triangle @xcite , well-known for machine translation, and situating our chosen tasks within such a triangle. here we see that pos and ner are relatively 'shallower' analysis tasks that are token-centric, while qa, nli and pa are 'deeper' analysis tasks that would require deeper semantic representations. this representation suggests a strategy for effective parameter sharing. for the deeper tasks, the same task in different languages could have representations that are closer and hence benefit each other, while for the shallower tasks, keeping the language unchanged and exploring different tasks might be more beneficial. interestingly, this is exactly what we find with our meta-learned model and is borne out in our experimental results. we also find that as the model progressively learns, the meta-learning based models for the tasks requiring deeper semantic analysis benefit more from joint learning compared to the shallower tasks. with access to multiple tasks and languages during training, the question of how to sample effectively from different tasks and languages also becomes important to consider. we investigate different sampling strategies, including a parameterized sampling strategy, to assess the influence of sampling across tasks and languages on our metalearned model. our main contributions in this work are threefold: we also offer rules of thumb for effective meta-learning that could hold in larger settings involving additional tasks and languages. we summarize three threads of related research that look at the transferability in models across different tasks and different languages: multi-task learning, meta-learning and data sampling strategies for both multi-task learning and meta-learning. multitask learning @xcite has proven to be highly effective for transfer learning in a variety of nlp applications such as question answering, neural machine translation, etc. @xcite @xcite . some multi-task learning approaches @xcite have attempted to identify clusters (or groups) of related tasks based on end-to-end convex optimization formulations. meta-learning algorithms @xcite are highly effective for fast adaptation and have recently been shown to be beneficial for several machine learning tasks @xcite . data sampling strategies for multi-task learning and meta-learning form the third thread of related work. a good sampling strategy has to account for the imbalance in dataset sizes across tasks/languages and the similarity between tasks/languages. a simple heuristic-based solution to address the issue of data imbalance is to assign more weight to low-resource tasks or lan-guages @xcite . @xcite define a temperature parameter which controls how often one samples from low-resource tasks/languages. the multidds algorithm, proposed by @xcite , actively learns a different set of parameters for sampling batches given a set of tasks such that the performance on a held-out set is maximized. we use a variant of mul-tidds as a sampling strategy in our meta-learned model. @xcite is most similar in spirit to our work in that they study a crosslingual and cross-task meta-learning architecture but only focus on zero-shot and few-shot transfer for two natural language understanding tasks, nli and qa. in contrast, we study many tasks in many languages, in conjunction with sampling strategies, and offer concrete insights on how best to guide the meta-learning process when multiple tasks are in the picture. ## methodology our setting is pivoted on a grid of tasks and languages (with some missing entries as shown in ## results and analysis task-limited vs lang-limited models. for qa and nli, we observe that the task-limited models are always better than the temperature-based sampling strategy and @xmath0. mdds-task and mdds-lang refer to the use of a development set for multidds that contains all languages for a task and all tasks for a language, respectively. the best result among baseline and three mtl models is highlighted using orange. for each column we present the difference (positive or negative) of the meta models from the best baseline (highlighted in orange) of that column the lang-limited models. this is in line with our intuition that tasks like qa and nli (which require deeper semantic representations) will benefit more by using data from different languages for the same task. we see the opposite seems to hold for pos and ner where the lang-limited models are almost always better than the task-limited models. with pos and ner being relatively shallower tasks, it makes sense that they benefit more from language-specific training that relies on token embeddings shared across tasks. ## conclusion we present effective use of meta-learning for capturing task and language interactions in multi-task, multi-lingual settings. the effective use involves appropriate strategies for sampling tasks and languages as well as rough knowledge of the level of abstraction (deep vs. shallow representation) of that task. we present experiments on the xtreme multilingual benchmark dataset using five tasks and six languages. our meta-learned model shows clear performance improvements over competitive baseline models. we observe that deeper tasks consistently benefit from meta-learning. furthermore, shallower tasks benefit from deeper tasks when meta-learning is restricted to a single language. finally, zero-shot evaluations for several external languages demonstrate the benefit of using meta-learning over two multi-task baselines while also reinforcing the linguistic insight that tasks requiring deeper representations tend to collaborate better. baseline and three mtl models is highlighted using orange. for each column we present the difference (positive or negative) of the meta models from the best baseline (highlighted in orange) of that column