Less-Forgetting Multi-Lingual Fine-Tuning
Anonymous Author(s)
Abstract
Multi-lingual fine-tuning (MLF), which fine-tunes a multi-lingual language model(MLLM) with multiple source languages, aims to gain good zero-shot performanceon target languages.
In MLF, the fine-tuned model tends to fit the source languageswhile forgetting its cross-lingual knowledge obtained from the pre-training stage.
This forgetting phenomenon degenerates the zero-shot performance of MLF, whichremains under-explored.
To fill this gap, this paper proposes a multi-lingual finetuning method, dubbed Less-forgetting Multi-lingual Fine-tuning (LF-MLF).
In
LF-MLF, we cast multi-lingual fine-tuning as a constrained optimization problem,where the optimization objective is to minimize forgetting, and constraints arereducing the fine-tuning loss.
The proposed method has superior zero-shot performance; furthermore, it can achieve the Pareto stationarity.
Extensive experiments on
Named Entity Recognition, Question Answering and Natural Language Inferenceback up our theoretical analysis and validate the superiority of our proposals.
Introduction
Multi-lingual Language Models (MLLMs), which are pre-trained on multiple languages, have thecross-lingual generalization ability.
By fine-tuning MLLMs on source languages, we can expectpromising zero-shot performance on target languages [1, 2].
In literature, most of the works focuson mono-lingual fine-tuning, which fine-tunes a multilingual model on only one source language(typically English).
However, in practice, it is common that there are multiple source languagesavailable, e.g., in name entity recognition, it is easy to obtain labeled data for English, German,
French and so on.
In this background, multi-lingual fine-tuning (MLF), which fine-tunes an MLLMon multiple source languages, becomes an emerging research topic.
In the fine-tuning procedure of MLF, each source language has a language-specific gradient descentdirection.
Combining these gradient descent directions together, we can find a set of gradient descentdirections common for all the source languages.
However, in this set, some directions lead a MLLMto fit the source languages too much and forget its cross-lingual knowledge obtained from the pretraining stage.
This forgetting phenomenon can degenerate the zero-shot performance of MLF.
Thus,how to find a common gradient descent direction that benefits both the fine-tuning performance andzero-shot performance is a important issue in MLF, which remains under-explored.
To address this issue, this paper proposes to find a less-forgetting descent direction , which prevents a
MLLM from forgetting the cross-lingual generalization ability and is a common descent direction forthe source languages.
To find this less-forgetting descent direction, we cast MLF as a constrainedoptimization problem.
The optimization objective is to minimize the forgetting of a MLLM’s crosslingual generalization ability, and the constraint is that the direction should be a descent directioncommon to all the source languages.
Correspondingly, we propose a novel algorithm, namely Lessforgetting Multi-lingual Fine-tuning (LF-MLF), to solve this constrained optimization problem andobtain the less-forgetting descent direction.
For LF-MLF, we have conducted both theoretical and experimental analysis.
Theoretical resultsdemonstrate that LF-MLF can effectively reduce the forgetting of multi-lingual models in finetuning; besides, LF-MLF can achieve the Pareto stationarity.
The superiority of LF-MLF has beenexperimentally verified on three kind of tasks (i.e., Named Entity Recognition, Question Answeringand Natural Language Inference ).
Furthermore, we also conduct experimental analysis on severalfactors that impact the performance of multi-lingual fine-tuning (e.g., the impact of the number of thefine-tuning languages), which have the potential to forge new trends in multi-lingual fine-tuning.
Related Works
Multi-lingual Language Models (MLLMs) are pre-trained using tons of unlabeled data frommultiple languages and expected to own the cross-lingual zero-shot transfer ability.
This ability canfacilitate the low resource languages processing with knowledge transferred from the high resourcelanguages [2].
Several MLLMs have been proposed, such as mBERT [3] and XLM-R [4].
Given a
MLLM, most of the existing works analyze its cross-lingual zero-shot transfer ability by observingthe model’s zero-shot performance on the target languages after mono-lingual fine-tuning [5, 6, 7].
Few works [8, 9] have investigated the multi-lingual fine-tuning setting, but they do not consider theforgetting of MLLMs’ cross-lingual generalization ability.
To fill this gap, this paper focuses on theforgetting problem in multi-lingual fine-tuning.
Continual Learning (CL) studies the problem of learning a model on sequential tasks withoutforgetting knowledge obtained from the preceding tasks [10].
In CL, reducing forgetting is the majorchallenge.
Lots of methods have been proposed to reduce forgetting, such as GEM [11], EWC [12]and IMM [13]; however, these methods typically focus on the single-objective optimization setting,where there is only one learning objective.
None of them considers the multi-objective optimizationsetting, where multiple learning objectives involve.
Nevertheless, multi-lingual fine-tuning involves amulti-objective optimization problem, which aims to optimize the fine-tuning performance on all theincluding languages.
Thus, existing CL methods are not suitable for multi-lingual fine-tuning.
Multi-task Learning (MTL) , which simultaneously learns multiple tasks, aims to achieve properperformance on all included tasks.
Recently, various MTL methods [14, 15, 16, 17, 18, 19, 20] havebeen proposed.
Among them, PCGrad [19] and Gradient Vaccine [20] have achieved the state-ofthe-art performance.
Regarding the source languages of MLF as MTL tasks, MTL methods can beused in MLF.
However, MTL methods just focus on the performance of including source languages(tasks), while they do not consider the performance of target languages.
It will bring inferior zero-shotperformance on the target languages.
For example, PCGrad and Gradient Vaccine just focus onfinding a proper common descent direction for the source languages, but the target languages are notconsidered in these methods.
Thus, they cannot expect to achieve promising zero-shot performance.
By contrast, the proposed LF-MLF finds the common descent direction that benefit the zero-shotperformance.
Overall, LF-MLF have more advantages over the MTL methods in MLF.
Less-forgetting Multi-lingual Fine-tuning
The goal of multi-lingual fine-tuning is twofold: firstly, achieving proper fine-tuning performance onthe source languages, and secondly, achieving proper zero-shot performance on the target languages.
For the first goal, it is necessary to avoid the model from forgetting the cross-lingual knowledgeit obtained in the pre-training phase.
For the second goal, we face a multi-objective optimizationproblem, where the losses for the source languages should be jointly optimized.
In this section, wefirstly give a formal definition of the multi-lingual fine-tuning and then propose a method (dubbed
LF-MLF) that can simultaneously achieve the above two goals.
3.1 Problem Definition of Multi-lingual Fine-tuning
Given a multi-lingual language model θ p , which is pre-trained using large amounts of unlabeleddata from language set S p , we aim to fine-tune this model on a downstream task and achieve proper performance.
For the downstream task, we have task-specific labeled training data from a set ofsource languages S f , where S f ⊆ S p .
After fine-tuning, θ p has been updated as θ f .
Assume thereare T languages in source languages set S f .
For each language, the task-specific loss is denoted as
L t ( θ f ) .
Under the gradient descent paradigm of deep learning, in the k th iteration of multi-lingualfine-tuning, the model is updated by
[Equation]
) where η is the learning rate, (cid:80) Tt =1 w kt = 1 and w kt ≥ 0 , t ∈ { 1 , ..., T } .
In Eq.
(1),− (cid:80) Tt
=1 w kt ∇ L t ( θ k − 1 ) is the convex hull of the gradient descent directions of the included languages, which contains feasible gradient descent directions for the multi-lingual fine-tuning.
Furthermore, denote the pre-training loss of θ p and θ f as L p ( θ p ) and L p ( θ f ) respectively.
Accordingto the classic definition of forgetting [21], we define the forgetting between θ p and θ f as F p =
L p ( θ f ) − L p ( θ p ) , which quantifies the performance degeneration of θ f on the multi-lingual pretraining task (e.g., masking tokens).
With the above notation, the two goals for multi-lingualfine-tuning can be formally defined as: (1) minimizing F p , and (2) minimizing { L 1 ( θ f ) , ..., L T ( θ f ) } .3.
Upper Bound the Forgetting in Multi-lingual Fine-tuning
In fine-tuning, we cannot directly compute the forgetting F p , because the pre-training the head (e.g.,masked language modeling head) has already been replaced by downstream-task-specific head (e.g.,a token classification head) and L p ( θ f ) is not accessible.
Alternatively, we propose to minimize theforgetting by minimizing its upper bound.
In Theorem 1, we give a upper bound for the forgetting ofmulti-lingual fine-tuning.
The assumptions in Theorem 1 are widely accepted [21, 22, 23] and canensure proper performance.
Theorem 1.
Assume L p ( θ f ) can be approximated by its second order Taylor expansion and θ p is aminima w.r.t the pre-training loss.
Then, we have
[Equation]
where λ p is the maximum eigenvalue of ∇ 2 L p ( θ p ) , and K is the number of iterations of fine-tuning.
Proof.
The proof is provided in the supplementary materials.
3.3 Less-forgetting Multi-lingual Fine-tuning
In multi-lingual fine-tuning, we not only need to keep the multi-lingual memory of the pretrained model but also need to achieve proper fine-tuning performance, i.e., minimizing F p and{ L 1 ( θ f ) , ..., L T ( θ f ) } together.
Achieving these two goals at same time is challenging and remainsunexplored.
In this section, we propose a novel method, dubbed Less-forgetting Multi-lingual
Fine-tuning (LF-MLF), to jointly achieve these two goals.
Specifically, we minimizing F p by tightening its upper bound introduced in Theorem 1.
Besides,minimizing { L 1 ( θ f ) , ..., L T ( θ f ) } is a multi-objective optimization (MOO) problem.
We solve this
MOO problem by the means of multi-objective gradient descent.
According to Theorem 1, in the k th iteration,
[Equation]
where ¯ ∇ L = (cid:80) k − 1 i =1 (cid:80) Tt =1 w it ∇ L t ( θ i − 1 ) and θ 0 = θ p .
To reduce fogetting, in the k th iteration, we propose to minimize F kp ’s upper bound as in Eq.
[Equation] [Equation]
[Figure]
Except for reducing forgetting, we also need to decrease the task-specific losses of the fine-tunedlanguages, i.e, move toward the objective of minimizing { L 1 ( θ f ) , ..., L T ( θ f ) } .
Thus, in the k thiteration, we need to update the model in a descent direction common to all the source languages,which is the principle of multi-objective gradient descent.
Under this principle, the update directionshould not have a obtuse angle with the gradient direction on each language as in Eq.
[Equation]
To reduce fogetting and minimizing the downstream task losses, we combine Eq.
(4) with Eq.
(5)and propose to formulate the less-forgetting multi-lingual fine-tuning as the following constrainedoptimization problem.
Problem 1.
[Equation]
To illustrate the rationale of this constrained optimization problem, we give a toy example in Figure1.
In this example, we compare LF-MLF with the Uniform-MLF method which adopts the uniformweights across the training procedure (i.e., w kt = 1 for t ∈ { 1 , ..., T } and k ∈ { 1 , ..., K } ).
LF-MLFchooses the common descent direction that minimizes the distance between the original MLLM θ pand the updated MLLM θ k at each update step.
Obviously, LF-MLF has shorter distance than the
Uniform-MLF method, which illustrates that LF-MLF can slow the forgetting of original MLLMdown.
Furthermore, to solve Problem 1, we formulate it as the following quadratic programmingproblem with T variables and 2 T + 1 constraints.
Problem 2.
[Equation]
) where G = ∇ L ( θ k − 1 )
T ∇ L ( θ k − 1 ) , ∇ L ( θ k − 1 ) =
[ ∇ L 1 ( θ k − 1 ) , L 2 ( θ k − 1 ) , ..., L T ( θ k − 1 )] ⊤ .
c =∇ L ( θ k − 1 )
T ¯ ∇ L and b t = ∇ L ( θ k − 1 )
⊤ ∇ L t ( θ k − 1 ) .
Problem 2 can be effectively solved.
This paper adopts a interior-point solver [24].
Let w ∗ k be thesolution of the above quadratic programming problem.
In our proposed LF-MLF, the update rule is
[Equation]
Overall, the challenging multi-lingual fine-tuning problem is transferred to a simple quadraticprogramming problem.
The detailed steps of LF-MLF can be referred to Algorithm 1.
Input: Pre-trained model θ p , Number of iterations K .
for k ← 1 to K do if k =
=
then
Solve Problem.
(2) with c = [0 , 0 , ..., 0] ⊤ and obtain solution w ∗ 1 .
θ 1 = θ p − η (cid:80) Tt =1 ( w ∗ 1 ) t ∇ L t ( θ p ) .
else
Solve Problem.
(2) and obtain solution w ∗ k .
θ
k
= θ k − 1 − η (cid:80) Tt =1 ( w ∗ k ) t ∇ L t ( θ k − 1 ) end if if the feasible set of Problem.
(2) is empty then break.
end if end for return θ K .
Theoretical Analysis
In less-forgetting multi-lingual fine-tuning, we explicitly set the goal, minimizing the upper bound, asthe optimization objective.
Therefore, the less-forgetting property of the proposed LF-MLF methodis straightforward.
By contrast, the target of minimizing { L 1 ( θ f ) , ..., L T ( θ f ) } is achieved by themeans of putting constraints on the update direction, which forces the update direction to be thecommon decreasing direction for all the fine-tuned languages.
Its effectiveness seems not obvious.
Tomake it clear, in this section, we conduct theoretical analysis on LF-MLF’s fine-tuning performance.
In multi-lingual fine-tuning, we aim to jointly fine-tune several languages, namely minimizing theloss vector { L 1 ( θ f ) , ..., L T ( θ f ) } .
It is a multi-objective optimization problem, where the optimalsolutions are Pareto optimal as in Definition 1.
Definition 1 (Pareto Optimality) .
For L ( θ ) =
[ { L 1 ( θ ) , ..., L T ( θ ) } ] ⊤ , the Pareto optimality w.r.t theoptimization objective min θ L ( θ ) is defined as follows: (1) A solution θ dominates a solution θ if
L t ( θ ) ≤ L t ( θ ) for all t ∈ T and
L ( θ ) ̸
= L ( θ ) and (2) A solution θ ∗ is deemed Pareto optimal ifthere exists no solution θ that dominates θ ∗ .
However, in deep learning, the parameter spaces are highly non-convex.
To the best of our knowledge,there is no method that can learn a Pareto optimal deep neural network model.
Settling for less, thebest we can do is to achieve the Pareto stationarity as in Definition 2.
Pareto stationarity is a necessarycondition of achieving Pareto optimality.
Definition 2 (Pareto Stationarity) .
For L ( θ 0 )
=
[ { L 1 ( θ 0 ) , ..., L T ( θ 0 ) } ]
⊤ is said to be Paretostationary at the design-point θ
iff there exists a convex combination of the gradient-vectors, 162 (cid:80)
[Equation]
, that is equal to zero:
[Equation]
In Theorem 2 and 3, we propose that LF-MLF can achieve Pareto stationarity, which theoreticallyverifies that our proposed method can achieve proper performance on the fine-tuning languages.
Theorem 2.
Let H be a Hilbert space of finite dimension N .
Let L t ( θ k ) ( 1 ≤ t ≤ T ≤ N ) be Tsmooth functions of the vector θ k ∈ H , and θ 0 k a particular admissible design-point.
Let w ∗ k be thesolution of Problem 2 and descent direction ∇ L = (cid:80)
Tt =1 ( w ∗ k ) t ∇ L t ( θ k ) .
Then, either ∇ L = ∅ ,and [ L 1 ( θ 0 k ) , ..., L T ( θ 0 k )]
⊤ are pareto stationary at θ 0 k or ∇ L ̸ = ∅ and −∇ L is a descent directioncommon to all { L t ( θ k ) }
Tt =1 .
Proof.
The proof
Theorem 3.
LF-MLF can stop after a finite number of iterations if a Pareto stationary point isreached.
Otherwise, If the sequence of iterates { θ k } Kk =1 of the LF-MLF is infinite, it admits a weaklyconvergent subsequence.
Proof.
The proof is provided in the supplementary materials.
Experiments
In this section, we perform experimental studies on three downstream tasks– Named Entity Recognition (NER), Question Answering (QA) and Natural Language Inference (NLI) respectively to evaluatethe performance of our proposed LF-MLF and verify our theoretical analysis.
The implementationis based on PyTorch [25] and XTREME benchmark [8].
The code is attached in the supplementarymaterials.
5.1 Experimental Setup
Datasets: In our experiments, we adopt the NER [26], TyDiQA-GoldP [27] and XNLI [28] datasetsfor NER, QA and NLI respectively from the XTREME benchmark [8].
The details of these datasetsare introuced in the supplementary material.
Baselines: We compare our proposed LF-MLF with the following baselines: (1) Single-lingual Finetuning (SLF): fine-tuning on each language independently; (2) Uniform-MLF: fine-tuning on theincluded languages simultaneously by means of a uniformly weighted sum of the language-specificgradient descent directions, i.e., w kt = 1 T constantly; (3) PCGrad-MLF: using the Project Conflicting
Gradients (PCGrad) method proposed by [19] on multi-lingual fine-tuning; (4) GradVac-MLF: usingthe Gradient Vaccine (GradVac) method proposed by [20] on multi-lingual fine-tuning.
Experimental Settings: In our experiments, we generally fine-tune XLM-RoBERTa models (basesized model) [29] with a training batch size of 32, and AdamW [30] is used as the optimizer.
As tothe numbers of fine-tuning epochs, we adopt the default setting of the XTREME benchmark, thatare 10, 3, 5 for NER, TyDiQA and XNLI respectively.
Besides, the learning rates are selected from{ 1 e − 5 , 2 e − 5 , 5 e − 5 } , { 1 e − 5 , 2 e − 5 , 5 e − 5 } and { 5 e − 6 , 1 e − 5 , 2 e − 5 } with grid search for
NER, TyDiQA and XNLI respectively.
All the results are averaged over 3 runs.
5.2 Performance of Multi-lingual Fine-tuning
Conducting experiments on NER, TyDiQA and XNLI, we compare LF-MLF with the baselines onthe zero-shot performance and fine-tuning performance respectively.
For each dataset, we choose twoor three source languages for fine-tuning according the following rules: (1) belongs to the Top 10high-resource languages w.r.t Wikipedias; (2) the higher-resource the better.
For NER and XNLI, en,de, fr are chosen, while en, ru are chosen for TyDiQA.
5.2.1 Zero-shot Performance
To evaluate the zero-shot performance of LF-MLF and the baselines, we record the F1 score performance (mean and standard deviation) over the target languages that have not been fine-tuned (i.e,.the other 45 languages except en, de, fr in NER, the other 7 languages except en, ru in TyDiQA andthe other 10 languages except en, de, fr in XNLI).
Furthermore, pairwise t-tests at 0.05 significancelevel are conducted based on the three runs.
The results are reported in Table 1, 2 and 3 for NER,
TyDiQA and XNLI respectively.
Besides, to comprehensively evaluate the superiority of LF-MLF, we further utilized Friedman test as the statistical test to analyze the relative performance among the compared methodsacross the three applications.
At 0.05 significance level, the Friedman statistics is 5.5, and critical is 4.76.
Thus, at 0.05 significance level, the null hypothesis of indistinguishable performance of LF-MTF among all compared methods is clearly rejected.
Subsequently, we employ the
Bonferroni-Dunn test as the post-hoc test by regarding LF-MLF as the control approach.
Figure 2reports the CD diagrams at 0.1 significance level, where the average ranks of the compared approachesis marked along the axis.
From this figure, we can see that LF-MTF achieves highly superior resultsto other baseline methods.
[Table] [Table]
[Table]
5.2.2 Fine-tuning Performance
Except the zero-shot performance, we also care about the fine-tuning performance, namely the F1score on the languages that have been fine-tuned.
The results of fine-tuning performance are reportedin Table 4, 5 and 6 for NER, TyDiQA and XNLI respectively.
From these results, we can see that:(1) the multi-lingual methods do not clearly outperform single-lingual methods on correspondinglanguages; (2) LF-MLF’s performance is comparable to the baselines, and there is no baseline methodcan dominate LF-MLF.
PCGrad-MLF and GradVac-MLF have inferior performance comparing with
Uniform-MLF and LF-MLF.
It because that PCGrad-MLF and GradVac-MLF cannot guarantee tofind a Pareto stationarity, while Uniform-MLF and LF-MLF can find a Pareto stationarity.
It verifiesthe theoretical analysis proposed in Section 4.
[Table]
[Table] [Table]
Overall
[Table]
from section 5.2.1 and 5.2.
[Table]
we can see that LF-MLF has superior zero-shot performanceand comparable fine-tuning performance.
It verifies that LF-MLF has superior cross-lingual transferability.
The superiority comes from LF-MLF’s less-forgetting property
[Table]
because such property enables
LF-MLF to remember more about the target languages.
5.3 Impact of the Number of the Source Languages
In the last section, it is concluded that multi-lingual fine-tuning can significantly improve the zero-shotperformance.
Then, it is natural to ask whether the zero-shot performance can be further improvedwhen more languages have been fine-tuned ?
In this section, we experimentally present that the answeris yes .
Besides, we also investigate the impact of the number of source languages on the fine-tuningperformance.
Our experiments are conducted on the NER dataset, and the source languages areselected from the Top 10 high-resource languages w.r.t Wikipedias, i.e., en, de, fr, ru, es, ja, it, zh,ko, ar .
By contrast, the other thirty eight languages are used as target languages.
In the experiments,we increase the source languages from Top 2 to Top 10 and record the changing of zero-shot andfine-tuning performance.
5.3.1 Impact on the Zero-shot Performance
Figure 3 illustrates the change of MLF’s zero-shot performance along with the increasing numberof source languages.
From this figure, we can conclude that (1) generally speaking, the zero-shotperformance of a multi-lingual model continuously improves with the increasing of the number ofsource languages; (2) LF-MLF consistently outperforms Uniform-MLF when the number of sourcelanguages changes.
5.3.2 Impact on the Fine-tuning Performance
To evaluate the fine-tuning performance over different number of source languages, we present ametric, dubbed average relative F1 score, which measures the average relative improvement of multilingual methods’s fine-tuning performance comparing with the single-lingual fine-tuning.
Specifically,let S st be the F1 score of the single-lingual fine-tuning on language t , and let S mt be the F1 score ofthe multi-lingual fine-tuning on language t .
Assume there are T fine-tuning languages, the averagerelative F1 score is defined as R T
[Table]
[Table] [Table]
[Table]
G e r m a n i c I t a li c
I n d o - I r a n i a n B a l t o - S l a v i c
Germanic
Italic
Indo-Iranian
Balto-Slavic 87.5 77.8 69.4 77.
90 A v e r a g e F 1 S c o r e (
[Figure]
) Cross Language Branches
A v e r a g e F 1 S c
o r e 81.37 80.84 81.08 81.
en de fr it en de bn hi en de bg ru fr it bn hi fr it bg ru bn hi bg ru en de fr bn en de fr ru en de bn ru fr it bn ru en fr bn ru 4 Combinations of 2 4 Combinations of 3 4 Combinations of
[Figure]
Figure 3 demonstrates the change of fine-tuning performance along with the increasing number ofsource languages.
From this figure, we can see that (1) the fine-tuning performance just slightlyfluctuates when the number of source languages changes; (2) LF-MLF and Uniform-MLF do notevidently outperform single-lingual fine-tuning, and they have similar average relative F1 score overdifferent number of source languages.
5.4 Impact of the Diversity of Language Branches Covered by the Source Languages
After investigating the impact of the number of source languages, we further explore whether thediversity of language branches covered by the source languages impacts the zero-shot performance.
In this section, we conduct experiments on the different branches of Indo-European language familybased on the NER dataset.
Specifically, we choose sixteen languages, which belong to four differentlanguage branches (Germanic, Italic, Indo-Iranian, Balto-Slavic) and each branch has four languages,from the NER dataset.
Firstly, we investigate the cross-language-branches transfer performance, namely fine-tuning onthe four languages of a branch and observe the zero-shot performance on another branch, such asfine-tuning on Germanic and test zero-shot performance on Italic.
The experimental results areillustrated in Fig.
5 (a) and Fig.
6 (a) (please refer to the Appendix) for LF-MLF and Uniform-MLFrespectively.
In these figures, the vertical axis represents the source.
These results present that: (1) thecross-language-branches transfer performance of each pair of branches is asymmetric; (2) LF-MLFhas better cross-language-branches transfer ability than Uniform-MLF, which experimentally explainsthe superiority of LF-MLF and further verifies LF-MLF’s less forgetting property.
Next, we explore the impact of language branches’ diversity of the source languages.
Eight languagesare used as candidates for fine-tuning (i.e., source languages), while the other eight languages areused to test the zero-shot performance.
The detailed allocation of these languages are reported in
Table 7 (please refer to the Appendix).
Fixing the number of fine-tuning languages as four, we testthe language branches’ combinations that the source languages cover two branches, three branchesand all four branches respectively, and the results are illustrated in Fig.
5 (b) and Fig.
6 (b) (pleaserefer to the Appendix).
From these figures, we can see that: (1) the language branches’ diversity ofthe fine-tuning languages does not have a clear impact on the zero-shot performance, which meansthat covering more language branches might not lead to better zero-shot performance; (2) LF-MLFgenerally outperforms Uniform-MLF on most of the combinations.
Conclusion
This paper researches on a novel fine-tuning setting for multi-lingual language models, namelymulti-lingual fine-tuning; furthermore, we propose a novel multi-lingual fine-tuning method, dubbed
Less-forgetting Multi-lingual Fine-tuning (LF-MLF).
Comparing with its mono-lingual fine-tuningcounterparts, multi-lingual fine-tuning can achieve superior zero-shot performance; moreover, ourproposed LF-MLF outperforms the baseline multi-lingual fine-tuning methods.
Researching onmono-lingual fine-tuning provides several inspirations and has the potential to forge new trends inmulti-lingual Learning research.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content
N
/A]
If you
used crowdsourcing or conducted research with human subjects
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A
(b) Did you describe any potential participant risks, with links to Institutional Review
Board
(IRB) approvals, if applicable?
[N/A
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A
A AppendixA.1 Limitations
Assumption. In Theorem 1, we assume that L p ( θ f ) can be approximated by its second order Taylorexpansion. This assumption is widely adopted [21, 22, 23] and can ensure proper performance overvarious deep neural network models and learning scenarios.
With this assumption, we obtain anupper bound for the forgetting of multi-lingual fine-tuning, which demonstrates that being close to theoriginal pretraining model can reduce forgetting, in other words, enhance cross-lingual generalization. Thus, if the assumption of second order Taylor approximation failed, being close to the originalpretraining model/objective is not sufficient/desirable for downstream cross-lingual generalization. Note that relaxing this assumption remains an open problem, and if this assumption can be relaxed,our proposed methods can be used in more widely models and scenarios. Theory .
In our theoretical analysis, we prove that our method can achieve a Pareto stationary point.
Pareto stationarity is only a necessary condition for achieving Pareto optimality.
Theoretical analysisof the gap between Pareto stationarity and Pareto optimality has not been studied in this work.
Furthermore, in the setting of deep learning, the gap between Pareto stationarity and Pareto optimalityremains less explored in both the area of multi-objective optimization and machine learning.
In thefuture, we will explore this challenging problem.
A.2 Impact of the diversity on the fine-tuning languages’ branches for Uniform-MLF
Figure 6 illustrates the cross-language-branches transfer performance and the impact of languagebranches’ diversity of the source languages for the Uniform-MLF. [Figure] [Table]
[Figure]
A v e r a g e F 1 S c o r e 80.72 80.38 81.1 80.84 81. en de fr it en de bn hi en de bg ru fr it bn hi fr it bg ru bn hi bg ru en de fr bn en de fr ru en de bn ru fr it bn ru en fr bn ru 4 Combinations of 2 4 Combinations of 3 4 Combinations of
[Figure]
A.3 Data allocation for the language branches
Table 7 illustrates the allocation of candidates for source languages and target languages of the sixteenlanguages. Language Family Germanic
[Table]
Balto
[Table]
Slavic [Table] [Table] hi
[Table]
A.4 Performance of All-lingual Fine-tuning
Table 8, 9 and 10 illustrate the zero-shot performance of LF-MLF and the baselines.
Especially, thesetables report the results of ALL-MLF, which takes all available languages as source languages (i.e., 48 source languages for NER, 9 source languages for TyDiQA, and 13 source languages for XNLI).
The performance of ALL-MLF is the upper bound for all the MLF methods.
Besides, Table 11, 12and 13 illustrate the fine-tuning performance of LF-MLF, ALL-MLF and the baselines..
M e t h od e n - S
L F d e - S
L F fr- S
L F
U n i f
o r mM L F
P C G r a dM L F
G r a d
V acM L F
L FM L F
A LL - M
L F (
U ppp e r
B ound )
Z e r o - S ho t 60 .± 0 .
18 • 58 .± 0 .
19 • 63 .± 0 .
58 • 67 .± 0 .
35 • 67 .± 0 .
32 • 67 .± 0 .
25 • 68 .± 0 .
88 .± 0 .
T a b l e 8 :
Z e r o - s ho t p e rf
o r m a n ce
( F 1 s c o r e
( m e a n ± s t d )) on
N E R .
I n a dd iti on , • i nd
i ca t e s t h a t
L FM L F i s s t
a ti s ti ca ll y s up
e r i
o r t o t
h e c o m p a r
i ng m e t hod ( p a i r
w i s e t - t e s t a t 0 .s
i gn i
fi ca n ce l e v e l ) .
A LL - M
L F i s t
h e a ll - li ngu a l fi n e - t un
i ng m e t hod , w h
i c h fi n e - t un e d t
h e m od e l w it h a ll t
h el a ngu a g e s .
M e t hod e n - S
L F r u - S
L F
U n i f
o r mM L F
P C G r a dM L F
G r a d
V acM L F
L FM L F
A LL - M
L F (
U ppp e r
B ound )
Z e r o - S ho t 55 .± 0 .
6± 0 .
34 • 62 .± 0 .
61 .± 0 .
51 • 61 .± 0 .
36 • 63 .± 0 .
78 .± 0 .
T a b l e 9 : Z e r o - s ho t
p
e rf o r m a n ce
( F 1 s c o r e
( m e a n ± s t d )) on
T y D i QA .
I n a dd iti on , • i nd
i ca t e s t h a t
L FM L F i s s t a t
i s ti ca ll
y s up e r i o r t o t h e c o m p a r
i ng m e t hod ( p a i r
w i s e t - t e s t a t 0 .s
i gn i
fi ca n ce l e v e l ) .
A
LL - M
L F i s t
h e a ll - li ngu a l fi n e - t un
i ng m e t hod , w h
i c h fi n e - t un e d t
h e m od e l w it h a ll t
h e 9 l a ngu
a g e s .
M e t h od e n - S
L F d e - S
L F fr- S
L F
U n i f
o r mM L F
P C G r a dM L F
G r a d
V acM L F
L FM L F
A LL - M
L F (
U ppp e r
B ound )
Z e r o - S ho t 72 .± 0 .
39 • 74 .± 0 .
38 • 74 .± 0 .
14 • 75 .± 0 .
12 • 74 .± 0 .
21 • 74 .± 0 .
14 • 75 .± 0 .
77 .± 0 .
T a b l e 10 :
Z e r o - s ho t p e rf
o r m a n ce
( F 1 s c o r e
( m ea n ± s t d )) on
XN
L I .
I n a dd iti on , • i nd
i ca t e s t h a t
L FM L F i s s t
a ti s ti ca ll y s up
e r i
o r t o t
h e c o m p a r
i ng m e t hod ( p a i r
w i s e t - t e s t a t 0 .s
i gn i
fi ca n ce l e v e l ) .
A LL - M
L F i s t
h e a ll - li ngu a l fi n e - t un
i ng m e t hod , w h
i c h fi n e - t un e d t
h e m od e l w it h a ll t
h el a ngu a g e s .
M e t hod e n - S
L F d e - S
L F fr- S
L F
U n i f
o r mM L F
P C G r a dM L F
G r a d
V acM L F
L FM L F
A LL - M
L F (
U ppp e r
B ound )
e n 82 .± 0 .
[Table]
± 0 .
81 .± 0 .
[Table]
± 0 .
[Table]
[Table] [Table]
[Table] [Table]
d e –
[Table]
± 0 .
[Table]
±
[Table] [Table]
± 0 .
[Table]
± 0 .
[Table]
±
[Table] [Table]
±
[Table]
fr – –
[Table]
± 0 .
89 .± 0 .
[Table]
± 0 .
[Table]
±
[Table]
[Table] [Table]
[Table] [Table]
± 0 .
T a b l e: F i n e - t un i ng p e rf o r m a n ce
( Fs c o r e ( m ea n ± s t d )) on
N E R .
A
LLM L F i s t
h e a ll - li ngu a l fi n e - t un
i ng m e t hod , w h
i c h fi n e - t un e d t
h e m od e l w it h a ll t
h e l a ngu a g e s .
M e t hod e n - S
L F r u - S
L F
U n i f
o r mM L F
P C G r a dM L F
G r a d
V acM L F
L FM L F
A LL - M
L F (
U ppp e r
B ound )
e n 66 .± 0 .
– 68 .± 0 .
69 .± 0 .
67 .± 0 .
69 .± 0 .
73 .± 0 .
r u – 70 .± 0 .
72 .± 0 .
71 .± 0 . 71 .± 0 .
71 .± 0 .
74 .± 0 .
T a b l e 12 :
F i n e - t un i ng p e rf o r m a n ce ( Fs c o r e ( m ea
n ± s t d ))
on
T y D i QA - G o l d P .
A LL - M
L F i s t
h e a ll - li ngu a l fi n e - t un
i ng m e t hod , w h
i c h fi n e - t un e d t
h e m od e l w it h a ll t
h e 9 l a ngu
a g e s .
M e t hod e n - S
L F d e - S
L F fr- S
L F
U n i f
o r mM L F
P C G r a dM L F
G r a d
V acM L F
L FM L F
A LL - M
L F (
U ppp e r
B ound )
e n 84 .± 0 .
– – 84 .± 0 .
84 .± 0 . 84 .± 0 .
81 ± 0 .84 .± 0 .
d e – 78 .± 0 .
– 79 .± 0 .
79 .± 0 . 79 .± 0 .
80 .± 0 . 80 .± 0 .
fr – – 79 .± 0 .
80 .± 0 .
79 .± 0 .
80 .± 0 . 80 .± 0 .
80 .± 0 .
T a b l e: F i n e - t un i ng p e rf o r m a n ce
( Fs c o r e ( m ea n ± s t d )) on
XN
L I .
A LL - M
L F i s t
h e a ll - li ngu a l fi n e - t un
i ng m e t hod , w h
i c h fi n e - t un e d t
h e m od e l w it h a ll t
h e l a ngu a g e s .