Original: BAD - X : Bilingual Adapters Improve Zero-Shot Cross-Lingual Transfer
Revised: BAD - X : Bilingual Adapters Improve Zero-Shot Cross-Lingual Transfer

Original: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Original: Adapter modules enable modular and efficientzero-shot cross-lingual transfer, where currentstate-of-the-art adapter-based approaches learnspecialized language adapters (LAs) for individual languages.
Revised: Adapter modules enable modular and efficient zero-shot cross-lingual transfer, where current state-of-the-art adapter-based approaches learn specialized language adapters (LAs) for individual languages.

Original: In this work, we show thatit is more effective to learn bilingual languagepair adapters (BAs) when the goal is to optimize performance for a particular sourcetarget transfer direction .
Revised: In this work, we show that it is more effective to learn bilingual language pair adapters (BAs) when the goal is to optimize performance for a particular sourcetarget transfer direction .

Original: Our novel BAD - Xadapter framework trades off some modularityof dedicated LAs for improved transfer performance: we demonstrate consistent gains inthree standard downstream tasks, and for themajority of evaluated low-resource languages.
Revised: Our novel BAD - X adapter framework trades off some modularity of dedicated LAs for improved transfer performance: we demonstrate consistent gains in three standard downstream tasks, and for the majority of evaluated low-resource languages.

Original:  Introduction
Revised:  Introduction

Original: Massively multilingual Transformers (MMTs) suchas mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020), and mT5 (Xue et al., 2021)have dominated research in multilingual NLP andcross-lingual transfer recently.
Revised: Massively multilingual Transformers (MMTs) such as mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020), and mT5 (Xue et al., 2021) have dominated research in multilingual NLP and cross-lingual transfer recently.

Original: Pretrained on largeamounts of unlabelled data in 100+ languages,they have been shown to achieve impressive performance for a wide range of languages and tasks, andin zero-shot cross-lingual transfer in particular (Wuand Dredze, 2019; K et al., 2019).
Revised: Pretrained on large amounts of unlabelled data in 100+ languages, they have been shown to achieve impressive performance for a wide range of languages and tasks, and in zero-shot cross-lingual transfer in particular (Wu and Dredze, 2019; K et al., 2019).

Original: However, theirrepresentational capacity is known to be limited bythe curse of multilinguality : a trade-off between thelanguage coverage and model capacity (Conneauet al., 2020), which typically favors high-resourcelanguages.
Revised: However, their representational capacity is known to be limited by the curse of multilinguality : a trade-off between the language coverage and model capacity (Conneau et al., 2020), which typically favors high-resource languages.

Original: Their limitations are thus especiallypronounced in low-resource scenarios, in transferbetween distant languages and towards resourcepoor target languages (Hu et al., 2020; Lauscheret al., 2020; Ansell et al., 2021b, inter alia ).
Revised: Their limitations are thus especially pronounced in low-resource scenarios, in transfer between distant languages and towards resourcepoor target languages (Hu et al., 2020; Lauscher et al., 2020; Ansell et al., 2021, inter alia ).

Original: A standard approach to zero-shot cross-lingualtransfer with MMTs (i) fine-tunes the full MMTon task-specific data in the source language andthen (ii) applies it directly to make predictions inthe target language (Hu et al., 2020).
Revised: A standard approach to zero-shot cross-lingual transfer with MMTs (i) fine-tunes the full MMT on task-specific data in the source language and then (ii) applies it directly to make predictions in the target language (Hu et al., 2020).

Original: On top ofthe expensive fine-tuning of the entire large model,this standard procedure also does not ‘prepare’ the
Revised: On top of the expensive fine-tuning of the entire large model, this standard procedure also does not ‘prepare’ the

Original: MMT to excel at a particular target language or for a particular source-target transfer direction .
Revised: MMT to excel at a particular target language or for a particular source-target transfer direction .

Original: This has been alleviated through modular parameter-efficient adaptations of the MMTs (Bapna and Firat, 2019; Philip et al., 2020;
Revised: This has been alleviated through modular parameter-efficient adaptations of the MMTs (Bapna and Firat, 2019; Philip et al., 2020;

Original: He et al., 2021) which bypass full fine-tuning, most prominently through lightweight adapters (Rebuffi et al., 2017; Houlsby et al., 2019): additional trainable parameters inserted into the MMT’s layers.
Revised: He et al., 2021) which bypass full fine-tuning, most prominently through lightweight adapters (Rebuffi et al., 2017; Houlsby et al., 2019): additional trainable parameters inserted into the MMT’s layers.

Original: They have recently been used for language and task specialization of the MMTs (Pfeiffer et al., 2020b), offering improved and more efficient zeroshot cross-lingual transfer.
Revised: They have recently been used for language and task specialization of the MMTs (Pfeiffer et al., 2020b), offering improved and more efficient zeroshot cross-lingual transfer.

Original: Previous work (Pfeiffer et al., 2020b; Üstün et al., 2020, 2021; Vidoni et al., 2020; Ansell et al., 2021b, inter alia ) focused on creating: 1) dedicated language adapters (LAs) for each individual language, and 2) individual task adapters (TAs).
Revised: Previous work (Pfeiffer et al., 2020b; Üstün et al., 2020, 2021; Vidoni et al., 2020; Ansell et al., 2021, inter alia ) focused on creating: 1) dedicated language adapters (LAs) for each individual language, and 2) individual task adapters (TAs).

Original: Creating single-language LAs enables a very modular approach to cross-lingual transfer, where a source language LA (used in training) can be directly swapped with any target language LA at inference.
Revised: Creating single-language LAs enables a very modular approach to cross-lingual transfer, where a source language LA (used in training) can be directly swapped with any target language LA at inference.

Original: Yet, this procedure still does not prepare nor adapt the MMT for a particular source-target transfer direction .
Revised: Yet, this procedure still does not prepare nor adapt the MMT for a particular source-target transfer direction .

Original: Put simply, if one’s incentive is to optimize the performance of a particular target language L t given annotated data in a particular source language
Revised: Put simply, if one’s incentive is to optimize the performance of a particular target language L t given annotated data in a particular source language

Original: L s , especially under low-data regimes, one might try to capture the interplay between the two languages instead of learning separate LAs.
Revised: L s , especially under low-data regimes, one might try to capture the interplay between the two languages instead of learning separate LAs.

Original: To address this gap, in this work we introduce the BAD - X framework: bilingual adapters (BAs) for zero-shot cross-lingual transfer (see Figure 1), designed towards improving transfer performance for a particular transfer direction, with a focus on low-resource target languages.
Revised: To address this gap, in this work we introduce the BAD - X framework: bilingual adapters (BAs) for zero-shot cross-lingual transfer (see Figure 1), designed towards improving transfer performance for a particular transfer direction, with a focus on low-resource target languages.

Original: The goal of BAD - X is to specialize the MMT for a particular language pair, while preserving all its existing knowledge encoded into the MMT’s parameters.
Revised: The goal of BAD - X is to specialize the MMT for a particular language pair, while preserving all its existing knowledge encoded into the MMT’s parameters.

Original: We experiment with three standard tasks in cross- lingual transfer (Lauscher et al., 2020; Ansell et al. ,2021b): part-of-speech tagging (POS), dependencyparsing (DP) and natural language inference (NLI),and with a total of 20 low-resource target languages.
Revised: We experiment with three standard tasks in cross- lingual transfer (Lauscher et al., 2020; Ansell et al., 2021): part-of-speech tagging (POS), dependency parsing (DP) and natural language inference (NLI), and with a total of 20 low-resource target languages.

Original: Our results demonstrate that trading off modularityof single-language LAs for less modular BAs (tailored for language pairs) indeed yields improvedtransfer performance over the current state-of-theart (SotA) adapter-based transfer framework MADX (Pfeiffer et al., 2020b), in all three tasks and forthe large majority of target languages.
Revised: Our results demonstrate that trading off modularity of single-language LAs for less modular BAs (tailored for language pairs) indeed yields improved transfer performance over the current state-of-theart (SotA) adapter-based transfer framework MADX (Pfeiffer et al., 2020b), in all three tasks and for the large majority of target languages.

Original: Moreover,we show that, under the fixed fine-tuning budgetand resources, further task performance gains canbe achieved by varying the ratio of L s -vs- L t unannotated data when learning BAs.
Revised: We also show that, under the fixed fine-tuning budget and resources, further task performance gains can be achieved by varying the ratio of L s -vs- L t unannotated data when learning BAs.

Revised: Finally, aiming to delve deeper into the trade-off between modularity and training efficiency, we experiment with multilingual adapters that are trained on the source language and all target languages under consideration at once.

Revised: We show that such adapters, despite being more efficient to train, are unable to match the performance of their more specialized counterparts across a diverse set of target languages.

Original: Wewill share ourcode and pretrained BAs online at: [URL] .
Revised: We share our code and pretrained BAs online at: https://github.com/parovicm/BADX .

Original:  BAD - X : Methodology
Revised:  BAD - X : Methodology

Original: Motivation and Overview.
Revised: Motivation and Overview.

Original: The main idea can besummarized into the following: instead of adaptingthe MMT to languages L s and L t separately asdone in the SotA adapter-based MAD - X framework(Pfeiffer et al., 2020b), cross-lingual transfer mightbe more effective by adapting the MMT directlyto the language pair ( L s , L t ) .
Revised: The main idea can be summarized into the following: instead of adapting the MMT to languages L s and L t separately as done in the SotA adapter-based MAD - X framework (Pfeiffer et al., 2020b), cross-lingual transfer might be more effective by adapting the MMT directly to the language pair ( L s , L t ) .

Original: This means that welearn a bilingual language-pair adapter instead oftwo separate monolingual LAs.
Revised: This means that we learn a bilingual language-pair adapter instead of two separate monolingual LAs.

Original: We then learn atask adapter directly on top of the BA: since wefocus on the zero-shot setting, this means usingtask-annotated examples only from L s to fine-tunethe TA.
Revised: We then learn a task adapter directly on top of the BA: since we focus on the zero-shot setting, this means using task-annotated examples only from L s to fine-tune the TA.

Original: This procedure is summarized in Figure 1.[-
-]
Revised: This procedure is summarized in Figure 1.[-
-]

Original: BAD - X Adapters.
Revised: BAD - X Adapters.

Original: BAD - X adapts the MAD - Xadapter framework, where BAs are learnt insteadof single-language LAs.
Revised: BAD - X adapts the MAD - X adapter framework, where BAs are learnt instead of single-language LAs.

Original: The architecture of theadapter in each layer l consists of a down- andup-projection with a residual connection.
Revised: The architecture of the adapter in each layer l consists of a down- and [Figure] up-projection with a residual connection.

Original: Morespecifically, let the down-projection be a matrix
Revised: More specifically, let the down-projection be a matrix

Original: D l ∈ R h × d and the up-projection be a matrix
Revised: D l ∈ R h × d and the up-projection be a matrix

Original: U l ∈ R d × h where h is a hidden size of the MMTand d is the hidden size of the adapter.
Revised: U l ∈ R d × h where h is a hidden size of the MMT and d is the hidden size of the adapter.

Original: Let us denote MMT’s hidden state and the residual at layer las h l and r l , respectively.
Revised: Let us denote MMT’s hidden state and the residual at layer l as h l and r l , respectively.

Original: The adapter computationof layer l is then given by:
Revised: The adapter computation of layer l is then given by:

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: with ReLU as the activation.
Revised: with ReLU as the activation function.

Original: This formulation sub-

Original: [Figure] sumes LAs and TAs in MAD - X , as well as BAs and[Figure] TAs in BAD - X , where LAs/BAs receive the input 
Revised: This formulation subsumes LAs and TAs in MAD - X , as well as BAs and TAs in BAD - X , where LAs/BAs receive the input from the (frozen) Transformer layer, while TAs receive the input from the (frozen)

Original: [Figure] from the (frozen) Transformer layer, while TAs re- [Figure] ceive the input from the (frozen) LA/BA put on top of the frozen Transformer layer (Figure 1).
Revised: LA/BA put on top of the frozen Transformer layer (Figure 1).

Original: MAD - X LAs are trained via masked-language modeling (MLM) objective on the Wikipedia of the corresponding language, while TAs are trained on annotated task data.
Revised: MAD - X LAs are trained via masked-language modeling (MLM) objective on the Wikipedia of the corresponding language, while TAs are trained on annotated task data.

Original: Once LA for L s is available, TA is trained by stacking it on top of the fixed source LA.
Revised: Once LA for L s is available, TA is trained by stacking it on top of the fixed source LA.

Original: Transfer is done by replacing the L s
Revised: Transfer is done by replacing the L s

Original: LA with the L t LA.
Revised: LA with the L t LA.

Original: Unlike MAD - X , BAD - X trains a single bilingual adapter via MLM, alternating between the unlabelled (Wikipedia) data from both
Revised: Unlike MAD - X , BAD - X trains a single bilingual adapter via MLM, alternating between the unlabelled (Wikipedia) data from both

Original: L s and L t .
Revised: L s and L t .

Original: The ‘data alternations’ are done according to a predefined ratio : e.g., the ratio of N : denotes that the model would see N L s sentences followed by 1 L t sentence.
Revised: The ‘data alternations’ are done according to a predefined ratio : e.g., the ratio of N :denotes that the model would see N L s sentences followed by 1 L t sentence.

Original: The motivation for this is twofold: 1) seeing a data mixture from the two languages could produce a BA that is better for transfer than having two independent LAs; 2) LAs for low-resource L t -s might otherwise overfit due to unlabelled data scarcity in L t , and thus could benefit from additional L s data.
Revised: The motivation for this is twofold: 1) seeing a data mixture from the two languages could produce a BA that is better for transfer than having two independent LAs; 2) LAs for low-resource L t -s might otherwise overfit due to unlabelled data scarcity in L t , and thus could benefit from additional L s data.

Original: In BAD - X , TA is then again trained on top of the fixed BA, and the same BA-TA configuration is retained at inference, see Figure 1 again.
Revised: In BAD - X , TA is then again trained on top of the fixed BA, and the same BA-TA configuration is retained at inference, see Figure 1 again.

Original: Advantages and Limitations.
Revised: Advantages and Limitations.

Original: BAD - X allows parameter-efficient transfer to arbitrary tasks and languages by learning modular bilingual and task representations.
Revised: BAD - X allows parameter-efficient transfer to arbitrary tasks and languages by learning modular bilingual and task representations.

Original: It trades-off some modularity of
Revised: It trades-off some modularity of

Original: MAD - X for increased performance and expressiveness when the goal is to perform a transfer for a fixed pair of languages.
Revised: MAD - X for increased performance and expressiveness when the goal is to perform a transfer for a fixed pair of languages.

Original: A disadvantage of BAD - X with respect to modularity is that it no longer offers a zero-cost transfer (once all LAs are learnt) between all language pairs under consideration: it requires training of separate BAs for all pairs of interest.
Revised: A disadvantage of BAD - X with respect to modularity is that it no longer offers a zero-cost transfer (once all LAs are learnt) between all language pairs under consideration: it requires training of separate BAs for all pairs of interest.

Original: However, as we show further in §3, BAD - Xmight be preferable over MAD - X in the cases whenthe goal is to improve a particular source-targetdirection, which is our targeted use-case.
Revised: However, as we show further in §3, BAD - X might be preferable over MAD - X in the cases when the goal is to improve a particular source-target direction, which is our targeted use-case.

Original:  Experiments and Results
Revised:  Experiments and Results

Original: Tasks and Languages.
Revised: Tasks and Languages.

Original: We treat MAD - X as ourprincipal baseline, and conduct all evaluationsand analyses on three standard cross-lingual taskswhich allow for experimentation with low-resourcetarget languages: POS, DP, and NLI.
Revised: We evaluate BAD-X on three standard cross-lingual tasks which allow for experimentation with low-resource target languages: POS, DP, and NLI.

Original: For POS and DP, we sample ten low-resourcelanguages from the Universal Dependencies (UD)2.7 dataset (Zeman et al., 2020), taking into account: 1) the availability and the size of the corresponding Wikipedia; and 2) typological diversityto ensure that different language families are covered.
Revised: For POS and DP, we sample ten low-resource languages from the Universal Dependencies (UD) 2.7 dataset (Zeman et al., 2020), taking into account: 1) the availability and the size of the corresponding Wikipedia; and 2) typological diversity to ensure that different language families are covered.

Original: For NLI, we rely on the recent AmericasNLIdataset (Ebrahimi et al., 2021), spanning ten lowresource languages from the Americas.
Revised: For NLI, we rely on the recent AmericasNLI dataset (Ebrahimi et al., 2022), spanning ten low-resource languages from the Americas.

Original: For AmericasNLI languages, we use Wikipedia if available;otherwise we use the unlabelled data previouslyused by Ansell et al.
Revised: For AmericasNLI languages, we use Wikipedia if available; otherwise we use the unlabelled data previously used by Ansell et al.

Original: (2021a).

Original: English is the sourcelanguage in all experiments for all tasks.
Revised: English is the source language in all experiments for all tasks.

Original:  All languages along with their language codesare listed in Table 2 in the Appendix.
Revised: 4 All languages along with their language codes are listed in Table 3 in the Appendix.

Original: 3. Experimental Setup
Revised: 3. Experimental Setup

Original: MMT.
Revised: MMT.

Original: In all our experiments, we use mBERT, an
Revised: In all our experiments, we use mBERT, an

Original: MMT model pretrained on the Wikipedias of 104languages (Devlin et al., 2019).
Revised: MMT model pretrained on the Wikipedias oflanguages (Devlin et al., 2019).

Original: Training Setup: LAs and BAs.
Revised: Training Setup: LAs, BAs.

Original: To enable a faircomparison between MAD - X and BAD - X under thesame training and inference conditions, we trainour own MAD - X LAs from scratch with the MLMobjective on monolingual Wikipedias: training isrun for 25,000 steps, with a batch size of 64 and alearning rate of 1 e − 4 .
Revised: To enable a fair comparison between MAD - X and BAD - X under the same training and inference conditions, we train our own MAD - X LAs from scratch with the MLM objective on monolingual Wikipedias: training is run for 25,000 steps, with a batch size of 64 and a learning rate of 1 e − 4 .

Original: We evaluate the LAs every500 training steps and finally choose the LA thatyields the lowest perplexity, as evaluated on the 5%of the Wikipedia data that acts as a validation set.
Revised: We evaluate the LAs every 500 training steps and finally choose the LA that yields the lowest perplexity, as evaluated on the 5% of the Wikipedia data that acts as a validation set.

Original: Pfeiffer et al.
Revised: Pfeiffer et al.

Original: (2020b) empirically establishedthat strong task performance of MAD - X on lowresource languages can be achieved already after 20,000 LA training steps, and that longer training offers only modest to negligible performance gains.
Revised: (2020b) empirically established that strong task performance of MAD - X on lowresource languages can be achieved already after 20,000 LA training steps, and that longer training offers only modest to negligible performance gains.

Original: Driven by their findings, we train MAD - X
Revised: Driven by their findings, we train MAD - X

Original: LAs for 25,000 iterations due to computational constraints, a large number of experiments, and the low-resource nature of our target languages.
Revised: LAs for 25,000 iterations due to computational constraints, a large number of experiments, and the low-resource nature of our target languages.

Original: BAD - X BAs are trained on the Wikipedia data of both L s and L t .
Revised: BAD - X BAs are trained on the Wikipedia data of both L s and L t .

Original: The standard BAD - X variant termed Balanced BAD - X (also BAD - X 1:1 ) is trained by alternating one batch of the L s data (i.e.
Revised: The standard BAD - X variant termed Balanced BAD - X (also BAD - X 1:1 ) is trained by alternating one batch of the L s data (i.e.

Original: English) followed by one batch of the L t data, for 50,000 iterations (i.e., this way we match the total number of iterations performed by training MADX L s and L t LAs for 25,000 iterations each), and we adopt all the hyperparameters from MAD - X LA training.
Revised: English) followed by one batch of the L t data, for 50,000 iterations (i.e., this way we match the total number of iterations performed by training MADX L s and L t LAs for 25,000 iterations each), and we adopt all the hyperparameters from MAD - X LA training.

Original: We select as the final BA the one with the lowest L t perplexity.
Revised: We select as the final BA the one with the lowest L t perplexity.

Original: Bilinguality of the BAD - X
Revised: Bilinguality of the BAD - X

Original: BAs allows us to directly train TA on top of it and perform the inference with the same setup.
Revised: BAs allows us to directly train TA on top of it and perform the inference with the same configuration. Multilingual Adapter (MA).

Revised: Given N target languages of interest, one could alternatively train a multilingual adapter on unlabelled text from L s and all N target languages

Original: Training Setup: TAs.
Revised: L t : while this is more computationally efficient than both BAD-X and MAD-X 6 it could, presumably, again lead to the “curse of multilinguality”, as the adapter parameters would be shared across N +1 languages. On the other hand, MA has the chance to benefit from similarities between target languages (especially in the case of AmericasNLI). Concretely, we train two multilingual adapters: one for the set of UD languages and the other for the set of AmericasNLI languages.

Revised: Multilingual UD adapter is trained by alternating one batch of English Wikipedia and one batch from each of 10 UD languages’ Wikipedia for 50,000 iterations.

Revised: Multilingual AmericasNLI adapter is obtained following the same procedure, only using Wikipedias of the NLI target languages.

Original:  For POS and DP, TA is trained by stacking it on top of the source (i.e. English) LA (with MAD - X ) or the English-
Revised: Training Setup: TAs. For POS and DP, TA is trained by stacking it on top of the source (i.e. English) LA (with MAD - X ), the English-

Original: L t BA (with BAD - X )  and performing 15,000 steps with a batch size of 8 and a learning rate of 5 e − 5 .
Revised: L t BA (with BAD - X ) or the multilingual adapter MA and performing 15,000 steps with a batch size of 8 and a learning rate of 5 e − 5 .

Original: We evaluate the TAs every 250 steps on English validation sets, and select as the final TAs the ones with the best accuracy (POS) and LAS scores (DP).
Revised: We evaluate the TAs every 250 steps on English validation set, and select as the final TAs the ones with the best accuracy (POS) and LAS score (DP).

Original: The adapter reduction factor (Pfeiffer et al., 2020a) is  for LAs and 16 for TAs.
Revised: The adapter reduction factor (Pfeiffer et al., 2020a) is 2 for LAs and BAs andfor TAs.

Original: For AmericasNLI, we train its TA on the English MultiNLI dataset (Williams et al., 2018) following the setup of Ebrahimi et al.
Revised: For AmericasNLI, we train its TA using the English MultiNLI data (Williams et al., 2018) following the setup of Ebrahimi et al.

Original: (2021): 5 epochs with a batch size of 32, and a learning rate of 2 e − 5 .
Revised: (2022):epochs with a batch size of 32, and a learning rate of 2 e − 5 .

Original: We evaluate the TA every 625 steps and choose the one with the bestaccuracy on the English validation set.
Revised: We evaluate the TA every 625 steps and choose the one with the best English validation accuracy.

Original: BAD - X : BA Variants.
Revised: BAD - X : BA Variants.

Original: Besides Balanced BADX , we consider other variants of BAD - X BAs that differ in the data ratios between L s and L t ; we denote these variants as BAD - X 1 : N , where 1 batch of L s data is followed by N batches of L t data, and vice versa: BAD - X N : 1 .
Revised: Besides Balanced BADX , we consider other variants of BAD - X BAs that differ in the data ratios between L s and L t ; we denote these variants as BAD - X 1 : N , where 1 batch of L s data is followed by N batches of L t data, and vice versa: BAD - X N : 1 .

Original: With these variants, we aim to answer the following question: given a fixed number of MLM training steps (i.e., a fixed computational budget) for BAs, is it possible to further impact/improve transfer performance?
Revised: With these variants, we aim to answer the following question: given a fixed number of MLM training steps (i.e., a fixed computational budget) for BAs, is it possible to further impact/improve transfer performance?

Original: Is the optimal data sampling ratio task-dependent?
Revised: Is the optimal data sampling ratio task-dependent?

Original: 3. Results and Discussion
Revised: 3. Results and Discussion

Original: The results for all languages and tasks with MAD - X and Balanced BAD - X are summarized in Table 1.
Revised: The results for all languages and tasks with MAD - X and Balanced BAD - X are summarized in Table 1, with additional results in the appendix.

Original: As a general trend, we observe that the proposed 
Revised: As a general trend, we observe that the proposed Balanced

Original: [Table] Balanced BAD - X variant outperforms MAD - X overa majority of languages and across all three tasks:besides offering higher average results, we alsoreport gains on 8/10 (POS; accuracy), 10/10 (DP;
Revised: BAD - X variant outperforms MAD - X and MAs over a majority of languages and across all three tasks: besides offering higher average results, we also report gains on 8/10 (POS; accuracy), 9/10 (DP;

Original: UAS), and 9/10 (NLI; accuracy) target languages.
Revised: UAS), and 8/10 (NLI; accuracy) target languages.

Original: This confirms the positive impact of BA training,which is able to capture additional interactions ofeach language pair, in lieu of LA training.
Revised: This confirms the positive impact of BA training, which is able to capture additional interactions of each language pair, in lieu of LA training.

Original: Performance across Tasks.
Revised: Performance across Tasks.

Original: In particular, BAD - Xgains on average 1.06% in accuracy and 0.66% in
Revised: In particular, BAD - X gains on average 1.06% in accuracy and 0.66% in

Original: F 1 compared to MAD - X on POS task.
Revised: F 1 compared to MAD - X on POS task.

Original:  The gains areeven more pronounced on the more complex DPtask, which shares the target language set with POS: BAD - X outperforms MAD - X on average with aneven larger gap of 2.62% in UAS and 2.38% in LASscores, on average.
Revised: It outscores the multilingual adapter on POS even more: 3.55% in accuracy and 2.87% in F1 on average. The gains over MAD - X are even more pronounced on the more complex DP task, which shares the target languages set with POS: BAD - X outperforms MAD - X on average with a gap of 2.62% in UAS and 2.38% in LAS scores.

Original: The gain is particularly high for
Revised: The gain is particularly high for

Original: Wolof, a West-African language spoken by morethan five million people, with ~9% improvementover MAD - X in both UAS and LAS scores.
Revised: Wolof, a West-African language spoken by more than five million people, with ~9% improvement over MAD - X in both UAS and LAS scores.

Original: Wolofis also a language with one of the highest gains in 
Revised: Wolof is also a language with one of the highest gains in POS.

Original: POS.
Revised: In the DP task, BAD - X achieves similar gains over the multilingual adapter: 2.22% UAS and 2.82% in LAS scores on average. Multilingual adapter achieves high scores on Wolof, which re-establishes Wolof as a language that highly benefits from the involvement of other languages during training.

Original: We also observe the superiority of Balanced BAD - X over MAD - X on NLI, now on another setof low-resource languages, with average accuracygains of 2.4%.
Revised: We also observe the superiority of Balanced BAD - X over MAD - X on NLI, now on another set of low-resource languages, with average accuracy gains of 2.4%.

Original: The highest improvement of 6.67%is observed for Wixarika.
Revised: The highest improvement of 6.67% is observed for Wixarika.

Original: Performance across Languages.
Revised: Performance across Languages.

Original: Importantly, wefind that improvements in all three tasks are metfor target languages coming from diverse languagefamilies (e.g., for Uralic, Indo-European, NigerCongo, Turkic, Aymaran families) and with diversetypological traits.
Revised: Importantly, we find that improvements in all three tasks are met for target languages coming from diverse language families (e.g., for Uralic, Indo-European, NigerCongo, Turkic, Aymaran families) and with diverse typological traits.

Original: We speculate that stacking TAson top of BAs instead of an English-specialised LAforces the model to also take into account information from the target language, which mitigates overfitting to English-only language properties.
Revised: We speculate that stacking TAs on top of BAs instead of an English-specialized LA forces the model to also take into account information from the target language, which mitigates overfitting to English-only language properties.

Original: Furthermore, coupling two languages in the BA trainingmight also allow for some information flow between the languages (e.g., some sharing at lexicallevel).
Revised: Furthermore, coupling two languages in the BA training might also allow for some information flow between the languages (e.g., some sharing at lexical level).

Original: This also might provide a positive impacton transfer performance, while this effect cannotbe achieved with individual LAs as in MAD - X .
Revised: This also might provide a positive impact on transfer performance, while this effect cannot be achieved with individual LAs as in MAD - X .

Original: [Figure]
Revised: Multilingual adapters lag behind MAD - X and BAD - X as they aim to fit too many languages into a small number of adapter parameters, which demonstrates the necessity of language and especially language-pair specialization when performance for a particular source-target transfer direction is paramount.

Original: BAD - X Variants.
Revised: BAD - X Variants.

Original: Figure 2 shows the ‘averageacross-languages’ scores for MAD - X and for all tested BAD - X variants (based on data sampling ratios at BA training; §3.1).
Revised: Figure 2 shows the ‘averageacross-languages’ scores for MAD - X and for all tested BAD - X variants (based on data sampling ratios at BA training; §3.1).

Original: The results indicate[Figure] several findings.
Revised: The results indicate several findings.

Original: First, all BAD - X variants outper- [Figure] form MAD - X on all three tasks on average.
Revised: First, all BAD - X variants outperform MAD - X on all three tasks on average.

Original: Second, there is no single best-performing BAD - X variant for all tasks, that is, the ‘winning’ variant seems to be task-dependent.
Revised: Second, there is no single best-performing BAD - X variant for all tasks, that is, the ‘winning’ variant seems to be task-dependent.

Original: In particular, DP benefits the 
Revised: In particular, DP benefits the most from 5:1 sampling, while for POS and NLI the 1:2 variant outscores the others although DP [Table]

Original: [Figure]
Revised: [Figure]

Original: most from 5:1 sampling, while for POS and NLI the 1:2 variant outscores the others although DP
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: and POS share exactly the same BA training data.
Revised: and POS share exactly the same BA training data.

Original: Note that, due to computational constraints, we did not extensively search for the best sampling ratios of the source and target language during BA training, thus the optimal strategy might not be covered by our experiments.
Revised: Note that, due to computational constraints, we did not extensively search for the best sampling ratios of the source and target language during BA training, thus the optimal strategy might not be covered by our experiments.

Original: However, these findings warrant further investigation in future work. 
Revised: However, these findings warrant further investigation in future work. Multiple Runs.

Revised: To validate that our results hold in the presence of different parameter initializations, we perform a comparison of MAD - X and Balanced

Original: Conclusion
Revised: BAD - X when the scores are averages of 3 runs with the same random seeds both for MAD - X and BADX . Due to computational constraints, we select only a subset of languages for this evaluation. In particular, we choose 4 UD ( MYV , KPV , TE and WO ) and 4 AmericasNLI languages ( CNI , GN , SHP and HCH ) and compare MAD - X and Balanced BADX by taking the average scores obtained fromruns.

Revised: The results are shown in Table 2, and again point to BAD - X ’s superiority over MAD - X in terms of transfer performance in all three tasks.

Original:  We have presented BAD - X , a novel adapter-based framework for zero-shot cross-lingual transfer.
Revised: [Table]  Conclusion We have presented BAD - X , a novel adapter-based framework for zero-shot cross-lingual transfer.

Original: BAD - X targets improving transfer performance for particular fixed source-target transfer directions through the introduction and use of dedicated bilingual language-pair adapters (BAs). 
Revised: BAD - X targets improving transfer performance for particular fixed source-target transfer directions through the introduction and use of dedicated bilingual language-pair adapters (BAs). The effectiveness of the BAs and the BAD - X framework has been demonstrated on three standard transfer tasks, across a plethora of low-resource languages. In future work, we will experiment with more efficient approaches to bilingual adapters, e

Original: The effectiveness of the BAs and  the BAD - X framework  has been demonstrated on three standard transfer tasks, across a plethora of low-resource languages.
Revised: [Table] g., based on contextual parameter generation (Ansell et al., 2021), and port the BAD - X framework to more languages and tasks. Acknowledgements ⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆ This work has been supported by the ERC PoC

Original: B Languages
Revised: Grant MultiConvAI (no. 957356) and a Huawei research donation to the University of Cambridge. Marinela Parovi´c is supported by Trinity College External Research Studentship. We would like to thank Alan Ansell and Jonas Pfeiffer for interesting discussions and suggestions.

Revised: A Details of the Experimental Setup

Revised: Computing Infrastucture.

Revised: All experiments were run on a single NVIDIA GeForce RTX 3090 GPU; training one BAD - X BA or multilingal LA for 50,000 iterations took around 24 hours ( MAD - X

Revised: LA for 25,000 steps took around 12 hours).

Revised: Training of any TA took less than two hours.

Revised: Evaluation is performed within the AdapterHub framework (Pfeiffer et al., 2020a).

Revised: Hyperparameters.

Revised: All hyperparameters were taken from (Pfeiffer et al., 2020b), as discussed in the main paper, and no hyperparameter search was done.

Revised: All reported results except those in table 2 are from a single run.

Revised: B Languages

Original: The list of languages in each task along with theirlanguage codes is provided in Table 2.
Revised: The list of languages in each task along with their language codes is provided in Table 3.

Original: C BAD - X : Full results
Revised: C BAD - X : Full results

Original: Full results on all languages for MAD - X and all
Revised: Full results on all languages for MAD - X and all

Original: BAD - X variants are given in Tables 3, 4 and 5  for
Revised: BAD - X variants are given in Tables 4, 5 and 6 for

Original: POS, DP and NLI, respectively.
Revised: POS, DP and NLI, respectively.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]