Paper: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Merging (ICLR-2024)

Reviewer comment (Experiments):
The paper presents inadequate benchmarks and baselines. Specifically, the argument that there is a lack of enough training data to study standard adapters (Houlsby) merging is unconvincing, as MAD-X serves as an example that is not fundamentally different in terms of training data requirements compared to Houlsby or this work.

Author response:
W1.1: Standard Adapters

Thanks for the recommendation, yes, the training data is sufficient in the context of smaller models instead of LLMs such as Llama2. We have implemented it, please refer to Response 2 in the Novelty part in the general response.

Reviewer comment (Experiments):
The test results for XNLI, XLSum, and XQuAD are sub-sampled (1000, 500, and 1000 data points respectively from the whole test set) without convincing justifications, instead of evaluating on the complete test sets.

Author response:
W1.2 & Q1: Sub-sampled testset

We include the entire test set in our evaluation for MGSM and XCOPA. However, for XNLI, XLSum, and XQuAD, we employ a sampling approach. Specifically, we randomly select 1000, 500, and 1000 data points, respectively, from the complete test set. This sampling strategy is necessary due to the extended inference time of the Large Language Model (LLM) on these datasets. Without sampling, testing one method on the XNLI dataset alone would take approximately 120 hours. Therefore, it has become a common practice, as demonstrated in previous works [1] [2] [3] [4] to evaluate LLM methods on a subset of the test set. It is important to note that the sampled test set is sufficiently large to avoid any potential bias.

Reviewer comment (Experiments):
The claim that differences in results between the same task across two languages approximate 'language ability' was not carefully examined or discussed. Questions remain about whether the domains of the training data are the same for two languages for the same task, and whether any of the cross-lingual data for the reference task was machine-translated.

Author response:
W2.1: 'language ability' is not clearly explained

Regarding the training data, there is no restriction for it to be strictly language-parallel. The only requirement is that the data should pertain to the same task. In the case of the language modeling task, the training data consists of Wikipedia corpora specific to each corresponding language, and the task is conventional language modeling, i.e., next token prediction.

Reviewer comment (Evaluation):
Why do you randomly sub-sample the test set for evaluation for XNLI, XLSum, and XQuAD? What is wrong with evaluating on all test data?

Author response:
W1.2 & Q1: Sub-sampled testset

We include the entire test set in our evaluation for MGSM and XCOPA. However, for XNLI, XLSum, and XQuAD, we employ a sampling approach. Specifically, we randomly select 1000, 500, and 1000 data points, respectively, from the complete test set. This sampling strategy is necessary due to the extended inference time of the Large Language Model (LLM) on these datasets. Without sampling, testing one method on the XNLI dataset alone would take approximately 120 hours. Therefore, it has become a common practice, as demonstrated in previous works [1] [2] [3] [4] to evaluate LLM methods on a subset of the test set. It is important to note that the sampled test set is sufficiently large to avoid any potential bias.

Reviewer comment (Reproducibility):
What hyperparameters do you use for training?

Author response:
Q2: Hyper-paramters for training

Thanks for pointing out. We have listed hyper-paramters for training in the appendix of the new pdf version.

Reviewer comment (Experiments):
What is the reason behind using Llama-2 or T5 as the backbone, when other backbones, especially multilingual backbones, are available?

Author response:
Q3: Multi-Lingual Models

Please refer to Response 3 in the Experiment part in the general response.

Reviewer comment (Experiments):
The experimental setup is unclear and confusing. (1) Training data: Since the proposed methods learn task adapters, I would guess it learns on some training data of the downstream tasks. However, Table 1 only provides the details of test data, and it is unclear why the evaluation is conducted on small subsets of the test sets. (2) Baseline setup: MAD-X[1] adopts a similar idea, and introduces task and language adapters to cross-lingual transfer, which should be the most important baseline. Besides, fine-tune-based cross-lingual transfer methods such as xTune[2], and translate-test[3] should be considered. The setup of the XLT baseline is not clear. If the proposed method uses training data, is XLT evaluated in a few-shot setup as well?

Author response:
W1.1: Training data

For training data of each dataset on the source language (mostly in English), we use default training data of each dataset.

| Dataset   | MGSM | XCOPA | XNLI | XQuAD | XLSum |
| --------- | ---- | ----- | ---- | ----- | ----- |
| #Training | 7.5k | 0.5k  | 10k  | 1.2k  | 10k   |



W1.1: Test data

We include the entire test set in our evaluation for MGSM and XCOPA. However, for XNLI, XLSum, and XQuAD, we employ a sampling approach. Specifically, we randomly select 1000, 500, and 1000 data points, respectively, from the complete test set. This sampling strategy is necessary due to the extended inference time of the Large Language Model (LLM) on these datasets. Without sampling, testing one method on the XNLI dataset alone would take approximately 120 hours. Therefore, it has become a common practice, as demonstrated in previous works [1] [2] [3] [4] to evaluate LLM methods on a subset of the test set. It is important to note that the sampled test set is sufficiently large to avoid any potential bias.



W1.2: Baseline

Please refer to Response 3 in the Experiment part in the general response.

Reviewer comment (Evaluation):
The results are insufficient to support the claim "AdaMergeX consistently outperforms other state-of-the-art methods". First, the XNLI accuracy scores in Table 2 are too low to compete with random guess, which has 33.3% accuracy. For reference, mT5[4] achieves 85.0 accuracy on zero-shot cross-lingual transfer on XNLI. Besides, regarding XLT as a SOTA cross-lingual transfer method is misleading because a lot of cross-lingual transfer methods achieve better performance.

Author response:
W2: Low performance of LLMs on tasks 

The poor performance of Llama-7b can be attributed to its inability to effectively follow instructions. Additionally, in the case of XNLI, we have chosen to use labels such as "entailment", "contradict" and "neutral" instead of numerical values "0", "1", "2". This decision has made achieving an exact match more challenging. However, when we compare our results to those obtained using MAD-X and LF-LFT under the same conditions, our performance significantly improves, as demonstrated in the above first response.

Reviewer comment (Novelty):
What is the relation between AdaMergeX and MAD-X?

Author response:
Q2: relation between $\texttt{AdaMergeX}$ and MAD-X

We have further explained the difference between $\texttt{AdaMergeX}$ and related works. Please refer to Response 1 and Response 2 in the Novelty part in the general response.

Reviewer comment (Evaluation):
The results in absolute terms are quite low — for instance, many XNLI results actually underperform the random baseline (or a majority baseline) in a 3-way classification task such as NLI. The same goes for XCOPA results. There are much higher scores reported on those benchmarks in prior work on cross-lingual transfer learning. Given the reduced novelty and other methods that are very relevant and perform some sort of adapter merging, I fail to see how exactly this approach advances the field.

Author response:
W4 & Q1: Low performance of LLMs on tasks 

The poor performance of Llama-7b can be attributed to its inability to effectively follow instructions. Additionally, in the case of XNLI, we have chosen to use labels such as "entailment", "contradict" and "neutral" instead of numerical values "0", "1", "2". This decision has made achieving an exact match more challenging. However, when we compare our results to those obtained using MAD-X and LF-LFT under the same conditions, our performance significantly improves, as demonstrated in the above first response.

Reviewer comment (Experiments):
Have the authors also considered comparing to other adapter aggregation strategies in the context of XLT (e.g., adapting AdapterFusion for XLT)?

Author response:
Q3: Other adapter aggression baselines

In regards to AdapterFusion [1], AdapterSoup [2], LoRA-Hub [3], and other similar adapter aggregation approaches, they utilize a significantly larger amount of training data to obtain many adapter and encompass training parameters associated with fusion techniques. Hence, it would be unfair to draw comparisons between our method and theirs.



[1] Pfeiffer et al, AdapterFusion: Non-Destructive Task Composition for Transfer Learning, EACL 2021.

[2] Chronopoulou et al, AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models, EACL 2023.

[3] Huang et al, LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition, Arxiv 2023.