Zero-shot Cross-lingual Transfer With Learned Projections Using Unlabeled Target-Language Data (2023)

Abstract: Adapters have emerged as a parameter-efficient Transformer-based framework for cross-lingual transfer by inserting lightweight language-specific modules (language adapters) and task-specific modules (task adapters) within pretrained multilingual models. Zero-shot transfer is enabled by pairing the language adapter in the target language with an appropriate task adapter in a source language. If our target languages are known apriori, we explore how zero-shot transfer can be further improved within the adapter framework by utilizing unlabeled text during task-specific finetuning. We construct language-specific subspaces using standard linear algebra constructs and selectively project source-language representations into the target language subspace during task-specific finetuning using two schemes. Our experiments on three cross-lingual tasks, Named Entity Recognition (NER), Question Answering (QA) and Natural Language Inference (NLI) yield consistent benefits compared to adapter baselines over a wide variety of target languages with up to 11% relative improvement in NER, 2% relative improvement in QA and 5% relative improvement in NLI.

## introduction zero-shot cross-lingual transfer refers to the transfer of task-specific knowledge from a (highresource) source language to a (zero-resource) target language that has no labeled task-specific data for training. a popular paradigm for cross-lingual transfer learning is to finetune pretrained multilingual models using labeled task-specific data in the source language and directly evaluate these finetuned models on target language test sets. a parameter-efficient alternative to full finetuning for cross-lingual transfer is mad-x @xcite , an adapter-based framework that * equal contribution scaffolds on multilingual pretrained models to combine task-specific and language-specific modules in a plug-and-play manner. adapters @xcite are feedforward layer blocks inserted within each transformer layer to selectively learn taskspecific and language-specific capabilities via task adapters and language adapters, respectively. language adapters are trained using self-supervised objectives like masked language modeling (mlm) and task adapters are trained using task-specific objectives. to enable task transfer to a target language, the relevant language and task adapters are combined at test-time. in the zero-shot setting, we assume access to unlabeled text in the target languages. in mad-x, this text is only used to train target language adapters and not further used during finetuning. given knowledge of which languages we want to target, can we make effective use of unlabeled text in the target languages even during task-specific finetuning? this is the main question we tackle in this work. we propose a general adapter-based technique to inject target language bias into task-specific finetuning. using the unlabeled text in each target language, we construct an affine subspace from contextualized representations for every transformer layer in the multilingual model. these subspaces are defined using singular value decomposition (svd) and only need to be computed once per target language. during task-specific finetuning using labeled data in the source language, we project the source representations onto the target language subspaces. this projection can be invoked randomly using a projection probability defined as a hyperparameter. projections can also be triggered depending on whether the current source representations are closer to the mean embedding of the source language subspace compared to the mean embedding of the target language subspace. we investigate both these projection policies and find that they both improve performance across multiple tasks in multiple languages compared to state-ofthe-art adapter baselines. we also release code 1 to reproduce our experiments. overview of our technique. we are interested in the setting where we have apriori knowledge of which languages we want to target at test time. we aim to bias cross-lingual transfer towards known target languages during task-specific finetuning. we start with mad-x as our underlying framework and adopt the following 3-step approach: ## experimental setup subspace construction. to construct language specific subspaces, we adopt the settings used by @xcite . text sequences of length 512 are taken from the oscar dataset (ortiz su'arez et al., 2019) and passed through xlmr @xcite to produce layer-wise contextualized embeddings. we pick 262k contextualized representations and subtract the representation mean before computing svd. for a low-dimensional subspace, we select the greatest k singular values such that their sum of squares is greater than or equal to 90% of the total variance. (total variance is given by the sum of the squared singular values produced.) finally, in order to compute the languagespecific subspaces, the corresponding right singular vectors are taken as the basis. datasets. we conduct cross-lingual transfer experiments on three tasks, named entity recognition (ner), question answering (qa) and natural language inference (nli), where the source language is always english. for ner, we use the wikiann dataset @xcite , and show results for nine languages hindi, vietnamese, german, indonesian, icelandic, ilocano, swahili, burmese and javanese with roughly 20k instances in the english train set and between 1k and 10k instances in the target dev and test sets. for qa, we use xquad @xcite , a multilingual extension of squad @xcite and we report results for hindi, vietnamese and german consisting of around 87k examples in the english squad train set and 1190 instances in the three target dev sets. for nli, we use the amer-icasnli dataset @xcite which is an extension of the xnli dataset @xcite with low-resource american languages. we report results on quechua and guarani, consisting of 392k instances in the english train set and 2490 and 5010 instances in the dev and test sets, respectively for each target language. training setup. we use transformer models from the adapter-transformers 2 fork of the huggingface transformers library @xcite . we use pre-trained language adapters from adapterhub @xcite for our transfer experiments. xquad and nli fine-tuning experiments were conducted on a single nvidia a100 80 gb gpu for 15 epochs and 10 epochs, with learning rate 1e-4 and batch size 16. ner experiments were run 2 https://github.com/adapter-hub/ adapter-transformers for 30 epochs on nvidia 1080 ti with 12 gb ram. further details can be found in appendix a. ## results ner, xquad and nli results are shown in ## related work multilingual language models like mbert (devlin, 2018), xlm-r @xcite possess some zero-shot cross-lingual capabilities, even without any explicit finetuning on the languages of interest @xcite . such transfer without any finetuning could lead to degradation in performance across certain language pairs @xcite . nevertheless, multilingual models are a good foundation to bootstrap and further develop cross-lingual generalization. while there is a rapidly growing body of work on cross-lingual transfer, very few approaches utilize language-specific subspaces for this purpose. both @xcite and @xcite construct language-specific subspaces in multilingual models for an exploratory analysis of the model's representations. @xcite use projections on language specific subspaces to remove language specific information from the representations. we note such removal of language bias did not perform well on cross-lingual transfer in our experiments. @xcite train bilingual language adapters using both source and target language text before task adapter training. however, this requires training language adapters using both source and target language unlabelled text, for every language pair, in addition to training task adapters. in contrast, our setup is a simple architectural extension of mad-x, requiring no additional training once the subspaces are computed for each language. to the best of our knowledge, ours is the first work to exploit language-specific subspaces for cross-lingual transfer. ## conclusions in this work, we present a new adapter-based crosslingual transfer technique for an apriori known set of target languages. we construct language subspaces using contextualized representations for source and target languages. representations during task-specific training are projected onto the target subspace if they exceed a probability threshold or if they are closer to a mean source embedding. both schemes consistently improve zero-shot transfer for three natural language understanding tasks across many languages.