Hyper- X : A Unified Hypernetwork for Multi-Task Multilingual Transfer (2022)

Abstract: Massively multilingual models are promising for transfer learning across tasks and languages. However, existing methods are unable to fully leverage training data when it is available in different task-language combinations. To exploit such heterogeneous supervision, we propose Hyper-X, a single hypernetwork that unifies multi-task and multilingual learning with efficient adaptation. It generates weights for adapter modules conditioned on both tasks and language embeddings. By learning to combine task and language-specific knowledge, our model enables zero-shot transfer for unseen languages and task-language combinations. Our experiments on a diverse set of languages demonstrate that Hyper-X achieves the best or competitive gain when a mixture of multiple resources is available, while on par with strong baseline in the standard scenario. Hyper-X is also considerably more efficient in terms of parameters and resources compared to methods that train separate adapters. Finally, Hyper-X consistently produces strong results in few-shot scenarios for new languages, showing the versatility of our approach beyond zero-shot transfer.

## introduction transfer learning across languages and tasks has long been an important focus in nlp @xcite . recent advances in massively multilingual transformers (mmts; @xcite show great success in this area. a benefit of such models is their ability to transfer task-specific information in a high-resource source language to a low-resource target language (figure 1 , 1 ). alternatively, such models can leverage knowledge from multiple tasks for potentially stronger generalization (figure 1 , 2 ). over time, many research communities have been developing resources for specific languages of focus @xcite @xcite . in practice, it is thus common for data to be available for different tasks in a mixture of different languages. for instance, in addition to english data for both pos tagging and named entity recognition (ner), a treebank with pos annotation may be available for turkish, while ner data may be available for arabic. this example is illustrated in figure 1 , 3 . in contrast to existing cross-lingual transfer paradigms such as single-task zero-shot transfer @xcite or few-shot learning @xcite , multi-task learning on such a mixture of datasets (mixed-language multi-task) poses an opportunity to leverage all available data and model description x-lang. new lang. m-task x-pair (lt) mad-x cross-lingual transfer via language/task adapters ✔ ✔ ✗ ✔ @xcite hyperformer multi-task learning via shared hypernet adapters ✗ ✗ ✔ ✗ @xcite parameter space fact. transfer to unseen task-language pairs via psf ✗ ✗ ✔ ✔ (psf; @xcite hyper-x (this work) multi-language/task transfer via a unified hypernet ✔ ✔ ✔ ✔ to transfer information across both tasks and languages to unseen task-language combinations @xcite . standard fine-tuning strategies, however, are limited in their ability to leverage such heterogeneous task and language data. specifically, mmts are prone to suffer from catastrophic forgetting and interference @xcite when they are finetuned on multiple sources. adapters @xcite , a parameter-efficient fine-tuning alternative are commonly used for transfer either across tasks @xcite or languages @xcite but require training a new adapter for each new language @xcite . in this paper, we propose a unified hypernetwork, hyper-x that is particularly suited to this setting by leveraging multiple sources of information including different languages and tasks within a single model. the core idea consists of taking language and task embeddings as input, and generating adapter parameters via a hypernetwork for the corresponding task-language combination. by parameterizing each task and language separately, hyper-x enables adaptation to unseen combinations at test time while exploiting all available data resources. additionally, hyper-x can make seamless use of masked language modelling (mlm) on unlabelled data, which enables it to perform zero-shot adaptation to languages not covered by the mmt during pre-training. mlm also enables hyper-x to learn a language representation even without available task-specific data. in sum, our work brings together a number of successful transfer 'ingredients' that have been explored in very recent literature (see we evaluate hyper-x for cross-lingual transfer on two sequence labelling tasks, namely part-ofspeech (pos) tagging and named-entity recognition (ner) in 16 languages-7 of which are not covered in pre-training-across the three experimental setups depicted in figure 1 . our experiments demonstrate that hyper-x is on par with strong baselines for cross-lingual transfer from english. in the multi-task and mixed-language settings, hyper-x shows a large improvement compared to the standard baselines and matches the performance of the less efficient adapter-based model due to its ability to leverage heterogeneous sources of supervision. analysis highlights that hyper-x is superior in terms of efficiency-performance tradeoffs. finally, we evaluate our model in a few-shot setting, where hyper-x consistently achieves competitive performance across different languages and tasks, which suggests the usability of our approach in continuous learning scenarios. ## hyper-x we propose, hyper-x, an efficient adaptation of a mmt by exploiting multiple sources of information for transfer to an unseen language or tasklanguage pairs. specifically, hyper-x learns to combine task and language-specific knowledge in the form of embeddings using a hypernetwork. conditioned on the task and language embeddings, the hypernetwork generates composite adapter layers for the corresponding task-language combination (e.g. ner in turkish), thereby enabling transfer to arbitrary task-language pairs at test time. figure 2 provides an overview of our model. by jointly learning from task and language information, hyper-x overcomes some of the limitations of prior work: unlike adapter-based approaches @xcite that transfer cross-lingual information only to the task of the task adapter, our model is capable of leveraging supervision-and positive transferfrom both multiple tasks and languages. moreover, unlike @xcite who require annotated data in one of the target tasks for each language, hyper-x is able to perform zero-shot transfer even when there is no annotated data from any of the target tasks, by using mlm as an auxiliary task for each language. ## experiments dataset and languages we conduct experiments on two downstream tasks: part-of-speech (pos) tagging and named entity recognition (ner). for pos tagging, we use the universal dependencies (ud) 2.7 dataset @xcite and for ner, we use wikiann @xcite with the train, dev and test splits from @xcite . in addition to these two tasks, we also use masked language modelling (mlm) on wikipedia articles as an auxiliary task. we limit the number of sentences from wikipedia to 100k for each language, in order to control the impact of dataset size and to reduce the training time. for the language selection, we consider: (i) typological diversity based on language family, script and morphosyntactic attributes; (ii) a combination of high-resource and low-resource languages based on available data in downstream task; (iii) presence in the pre-training data of mbert; and (iv) presence of a language in the two task-specific datasets. 2 we provide the details of the language and dataset selection in appendix a. ## zero-shot transfer results overall, hyper-x base performs on par with the strongest baseline when transferring from english. in the presence of additional sources, such as a mixture of task-language pairs, hyper-x outperforms both mbert and parameter space factorization (psf). in comparison to mad-x, hyper-x generally performs better on seen languages. we relate this to the unified hypernetwork enabling maximum sharing between languages and higher utilization of the pre-trained capacity in contrast to the isolated adapters. on unseen languages, hyper-x is outperformed by mad-x in most cases. however, we emphasize that mad-x requires training separate language adapters for each new language, english single-task when english is used as the only source language for each task separately, hyper-x (base) performs on par with mad-x for ner (52.7 vs 52.8 f1) but falls behind for pos tagging (63.5 vs 65.4 acc.) on average. both models significantly outperform mbert. looking at the individual language results, hyper-x performs slightly better on 'seen' languages compared to mad-x in ner and pos tagging respectively. for 'unseen' languages, both mad-x and hyper-x benefit from mlm, which results in large improvements with respect to mbert. between the two models, mad-x achieves a higher average score in both ner and pos tagging. english multi-task in a multi-task setting where only english data is available, fine-tuning mbert for both target tasks at the same time gives mixed results compared to single-task training-in line with previous findings noting catastrophic forgetting and interference in mmts @xcite . hyper-x base, on the other hand, shows a small but consistent improvement on the majority of languages, with 0.2 (f1) and 0.1 (acc.) average increase in ner and pos tagging respectively. this confirms that hyper-x is able to mitigate interference while allowing for sharing between tasks when enough capacity is provided. 4 mixed-language multi-task in this setting, a mixture of language data is provided for ner and pos via two separate training partitions while keeping each task-language pair unseen in one of these partitions. all the models including mbert achieve better zero-shot scores compared to the previous settings. among the baselines, parameter space factorization (psf) gives a larger improvement compared to mbert on both tasks, indicating the importance of task-and language-specific parametrization for adapting a mmt. hyper-x base produces the largest performance gain among the models that trains only a single model: it achieves 9.0 (f1) and 4.3 (acc.) average increase for ner and pos. although both psf and hyper-x enable adaptation conditioned on a mixture of task and language combinations, we relate the difference between psf and hyper-x to the contrast in parameter generation. psf only generates parameters of the softmax layer and is thus unable to adapt deeper layers of the model. hyper-x, on the other hand, generates adapter layer parameters inserted throughout the model, which provide a higher degree of adaptation flexibility. hyper-x outperforms psf particularly on unseen languages as it benefits from mlm as an auxiliary task. finally, hyper-x tends to perform slightly better on seen languages compared to the adapted multisource version of mad-x. however, mad-x outperforms hyper-x on unseen languages by 1.2 (f1) and 2.8 (acc.) for ner and pos respectively. besides the expected benefits of independently trained language adapters in mad-x, we relate this to the limited cross-task supervision for unseen languages in hyper-x for this setting. especially, when the target task is pos, most of the unseen languages have only 100 sentences available in ner dataset, which leaves only a little margin for improvements. ## conclusion we have proposed hyper-x, a novel approach for multi-task multilingual transfer learning, based on a unified hypernetwork that leverages heterogeneous sources of information, such as multiple tasks and languages. by learning to generate composite adapters for each task-language combinations that modify the parameters of a pretrained multilingual transformer, hyper-x allows for maximum information sharing and enables zeroshot prediction for arbitrary task-language pairs at test time. through a number of experiments, we demonstrate that hyper-x is competitive with the state-of-the-art when transferring from a source language. when a mixture of tasks and languages is available, hyper-x outperforms several strong baselines on many languages, while being more parameter and time efficient. finally, we show that for few-shot transfer, hyper-x is a strong option with a less computing cost than baselines for the initial task adaptation.