ZGUL : Zero-shot Generalization to Unseen Languages using Multi-source Ensembling of Language Adapters (2023)

Abstract: We tackle the problem of zero-shot cross-lingual transfer in NLP tasks via the use of language adapters (LAs). Most of the earlier works have explored training with adapter of a single source (often English), and testing either using the target LA or LA of another related language. Training target LA requires unlabeled data, which may not be readily available for low resource *unseen* languages: those that are neither seen by the underlying multilingual language model (e.g., mBERT), nor do we have any (labeled or unlabeled) data for them. We posit that for more effective cross-lingual transfer, instead of just one source LA, we need to leverage LAs of multiple (linguistically or geographically related) source languages, both at train and test-time - which we investigate via our novel neural architecture, ZGUL. Extensive experimentation across four language groups, covering 15 unseen target languages, demonstrates improvements of up to 3.2 average F1 points over standard fine-tuning and other strong baselines on POS tagging and NER tasks. We also extend ZGUL to settings where either (1) some unlabeled data or (2) few-shot training examples are available for the target language. We find that ZGUL continues to outperform baselines in these settings too.

## introduction massive multilingual pretrained language models (plms) such as mbert @xcite and xlm-r @xcite support 100+ languages. we are motivated by the vision of extending nlp to the thousands @xcite of unseen languages, i.e., those not present in plms, and for which unlabeled corpus is also not readily available. a natural approach is zero-shot crosslingual transfer -train the model on one or more source languages and test on the target language, zero-shot. two common approaches are (1) standard fine tuning (sft) -fine-tune all parameters of a plm on task-specific training data in source language(s), and (2) language adapters (las)small trainable modules inserted within a plm transformer, and trained on target language's unlabeled data using masked language modeling (mlm). at test time, in sft the fine-tuned plm is applied directly to target language inputs, whereas for the latter, the la of source language is replaced with that of target language for better zero-shot performance. unfortunately, while in the former case, one would expect presence of unlabeled data for a target language to pre-train the plm for good performance, the latter also requires the same unlabeled data for training a target adapter. for the large number of low resource languages, curating a decent-sized unlabeled corpus is a challenge. for instance, there are only 291 languages that have wikipedias with over 1,000 articles. consequently, existing works use english data for training on the task and use the english la @xcite or an ensemble of related language las @xcite at inference time. we posit that this is sub-optimal; for better performance, we should leverage multiple source languages (ideally, related to target language) and their las, both at train and test-time. to this end, we propose zgul, zero-shot generalization to unseen languages, which explores this hypothesis. 1 it has three main components. first, it fuses las from source languages at train time by leveraging adapterfusion @xcite , which was originally developed for fusing multiple task adapters. this allows zgul to locally decide the relevance of each la for each token in each layer. second, zgul leverages the typological properties of languages (encoded in existing language vectors) as addi-tional information for computing global la attention scores. finally, zgul also implements the entropy-minimization (em)-based test-time tuning of la attention weights @xcite . we denote a language group as a set of phylogenetically or demographically close languages, similar to @xcite . we experiment on 15 unseen languages from four language groups: slavic, germanic, african and indo-aryan, on pos tagging and ner tasks. in each group, we train on multiple (3 to 4) source languages (including english), for which task-specific training data and las are available. zgul obtains substantial improvements on unseen languages compared to strong baselines like sft and cpg (conditional parameter generation @xcite ), in a purely zero-shot setting. detailed ablations show the importance of each component in zgul. we perform attention analysis to assess if learned weights to source languages' las are consistent with their relatedness to the target language. further, we study two additional scenarios, where (1) some unlabeled data, and (2) some taskspecific training data are available for the target language. we extend zgul in these settings and find that our extensions continue to outperform our competitive baselines, including ones that use unlabeled data for a target language to either (1) pre-train mbert or (2) train target's la. our contributions can be summarized as: (1) we propose a strong method (zgul) to combine the pretrained language adapters during training itself. to the best of our knowledge, we are the first to systematically attempt this in context of las. zgul further incorporates test-time tuning of la weights using entropy minimization (em). (2) zgul outperforms the competitive multi-source baselines for zero-shot transfer on languages unseen in mbert. ## model for ensembling of adapters our goal is to combine a set of source las for optimal zero-shot cross-lingual transfer to unseen languages, which are neither in mbert, nor have readily available labeled or unlabeled data. similar to previous works @xcite , we focus on languages whose scripts are seen in mbert. handling unseen languages with unseen scripts is a more challenging task, which we leave for future research. our approach can be described using two highlevel components: (1) train-time ensembling and (2) test-time ensembling, explained below for every layer l (for notational simplicity, we skip using l in notations but clarify wherever required). ## experiments we aim to address the following questions. (1) how does zgul perform in a zero-shot setting compared to the other baselines on unseen languages? what is the incremental contribution of zgul's components to the performance on lrls? (2) are la attention weights learnt by zgul interpretable, i.e., whether genetically/syntactically more similar source languages get higher attention scores? (3) how does zgul's performance change after incorporating unlabelled target language data? (4) how does zgul's performance vary in a few-shot setting, where a few training examples of the target language are provided to the model for fine-tuning? 4.1 datasets, tasks and baselines datasets and tasks: we experiment with 4 diverse language groups: germanic, slavic, african and indo-aryan. following previous works @xcite , we choose named entity recognition (ner) and part-of-speech (pos) tagging tasks. we select target languages which are unseen in mbert subject to their availability of test sets for each task. this leads us to a total of 15 target languages spanning germanic and slavic for pos, and african and indo-aryan for ner. for african and indo-aryan ner experiments, we use the masakhaner @xcite and wikiann @xcite datasets respectively. for pos experiments, we use universal treebank 2.5 @xcite . we pick training languages from each group that have pre-trained adapters available. the details of training and test languages, as well as corresponding task for each group are presented in in the second set, we compare against models trained upon multiple source languages (belonging to a group), in addition to english: it is important to note that the em algorithm is not applicable to sft and cpg baselines because sft does not use an adapter, while cpg has only a single (shared) adapter. consequently, there are no ensemble weights that can be tuned during inference for these methods. the em algorithm is a distinctive feature of the ensemble-based methods like zgul, which allows for further optimization and performance improvement. evaluation metric: we report micro-f1 evaluated on each token using seqeval toolkit @xcite . for all experiments, we report the average f1 from three training runs of the models with three different random seeds. the standard deviation is reported in appendix g. ## conclusion and future work we present zgul 7 , a novel neural model for ensembling the pre-trained language adapters (las) for multi-source training. this is performed by fusing the las at train-time to compute local tokenlevel attention scores, along with typological language vectors to compute a second global attention score, which are combined for effective training. entropy minimization (em) is carried out at testtime to further refine those attention scores. our model obtains strong performance for languages unseen by mbert but with the seen scripts. we present various analyses including that the learnt attention weights have significant correlation with linguistic similarity between source and target, and demonstrating scalability of our model in the unlabeled data and few-shot labeled data settings as well. in the future, our approach, being task-agnostic, can be applied to more non-trivial tasks, such as generation @xcite , semantic parsing @xcite , relation extraction @xcite , and knowledge graph completion @xcite . our technique may complement other approaches for morphologically rich languages @xcite and for 7 https://github.com/dair-iitd/zgul those with scripts unseen in mbert @xcite . extending our approach to code-switched data, in which each input token can potentially belong to a different language, is another interesting future direction. efficacy to domain-specific tasks not having sufficient publicly available data in that language and domain as to train a strong adapter (e.g. medical domain for african languages). presently, our technique cannot be tested directly on unseen scripts because our tokenization/embedding layer is same as that of mbert and may become a bottleneck for adapters to directly perform well. our approach is not currently tested on deep semantic tasks and generation-based tasks owing to the lack of suitable large-scale datasets for evaluation.