Less-Forgetting Multi-Lingual Fine-Tuning
Anonymous Author(s)
Abstract
Multi-lingual Language Models (e.g., mBERT) have the cross-lingual zero-shottransfer ability, i.e., after fine-tuning a multi-lingual model on a source language,the model can achieve promising zero-shot performance on a target language.
Thisability can help to tackle the challenges associated with low-resourced languagesand has gained much attention recently. Existing works typically focus on themono-lingual fine-tuning scenario, where only one source language involves infine-tuning.
However, in practice, there might be multiple high-resource languagescan be used as source languages. It is natural to ask whether fine-tuning a multilingual model on multiple source languages can improve the zero-shot ability?and how to do this kind of multi-lingual fine-tuning effectively? Unfortunately,these issues remain unexplored.
Accordingly, to fill this gap, this paper conductsboth theoretical and experimental analysis on this multi-lingual fine-tuning setting;furthermore, we propose a multi-lingual fine-tuning method, Less-forgetting Multilingual Fine-tuning (LF-MLF), which can achieve superior zero-shot performance.
The results of extensive experiments back up our theoretical analysis and validatethe superiority of our proposals.
Introduction
Multi-lingual Language Models, which are pre-trained using unlabeled data from multiple languages(including both high-resource and low-resource languages), support zero-shot cross-lingual transferlearning; in other words, for a down-stream task, after fine-tuning a multi-lingual language modelusing the labelled task-specific training data of a source language, the model can achieve promisingperformance on the task-specific test data of a target language. It can reduce the need of task-specificlabelled data for low-resource languages and give hope to low-resource language process. Thus,multi-lingual model-based zero-shot cross-lingual transfer learning has drawn large attention.
Existing works focus on the mono-lingual fine-tuning setting, where they fine-tune a multilingualmodel on one source language (typically English) to obtain a model that has proper zero-shotperformance on target languages [1, 2].
By contrast, in practical down-stream tasks, it is common thatthere are multiple high-resource languages have labelled data available for fine-tuning. For example,in Name Entity Recognition (NER), it is easy to obtain labeled data for English, German, French andso on.
It is natural to ask whether fine-tuning a multi-lingual model on multiple source languages canimprove the zero-shot ability? However, this issue remains unexplored. In this paper, our experimental analysis answers yes for the above question. Nevertheless, anotherquestion raises– how to do this kind of multi-lingual fine-tuning effectively?
Because, multi-lingualfine-tuning, which has not been explored before, is quite different from the mono-lingual fine-tuning.
Specifically, when fine-tuning on one language, we just need to update the model toward the gradientdescent direction. By contrast, in the multi-lingual fine-tuning setting, we have to choose an updatedirection in a feasible set, which is the convex hull of the gradient descent directions of the included languages.
The problem becomes which direction should be chosen?
This paper proposes to choosethe less-forgetting direction to avoid the multi-lingual model forgetting the target languages, namelyprevent the model from fitting the source languages too much to lose the generalization abilityon the target languages; moreover, the direction should be a descent direction common to all thesource languages.
To ensure the less-forgetting and common-descent requirements, we formulatemulti-lingual fine-tuning as a constrained optimization problem and propose an algorithm to solve thisproblem. The proposed algorithm is dubbed Less-forgetting Multi-lingual Fine-tuning (LF-MLF).
For LF-MLF, we have conducted both theoretical and experimental analysis.
Theoretical resultsdemonstrate that LF-MLF can effectively reduce the forgetting of multi-lingual models in finetuning; besides, LF-MLF can achieve the Pareto stationarity.
The superiority of LF-MLF has beenexperimentally verified on three kind of tasks (i.e., Named Entity Recognition, Question Answeringand Natural Language Inference ).
Furthermore, we also conduct experimental analysis on severalfactors that impact the performance of multi-lingual fine-tuning (e.g., the impact of the number of thefine-tuning languages), which have the potential to forge new trends in multi-lingual fine-tuning.
Related Works
Multi-lingual Language Models (MLLMs) are pre-trained using tons of unlabeled data frommultiple languages and expected to own the cross-lingual zero-shot transfer ability.
This ability canfacilitate the low resource languages processing with knowledge transferred from the high resourcelanguages [2].
Several MLLMs have been proposed, such as mBERT [3] and XLM-R [4].
Given a
MLLM, existing works commonly analyze its cross-lingual zero-shot transfer ability by the means offine-tuning the model on a source language and observing the model’s zero-shot performance on thetarget languages [5, 6, 7]. The fine-tuning scenario where multiple source languages are involved infine-tuning, namely multi-lingual fine-tuning, remains unexplored.
Continual Learning (CL) studies the problem of learning a model on sequential tasks withoutforgetting knowledge obtained from the preceding tasks [8].
In CL, reducing forgetting is the majorchallenge.
A large number of methods have been proposed to reduce forgetting, such as GEM [9], EWC [10] and IMM [11]; however, these methods typically focus on the single-objective optimizationsetting, where there is only one learning objective in a task.
None of them considers the multiobjective optimization setting, where there are multiple learning objective in a task.
Nevertheless,multi-lingual fine-tuning involves a multi-objective optimization problem, which aims to optimize thefine-tuning performance on all the including languages.
Thus, existing CL methods are not suitablefor multi-lingual fine-tuning.
Multi-task Learning (MTL) simultaneously learns multiple tasks and aims to achieve properperformance on all the including tasks.
It can be formulated as a multi-objective optimizationproblem [12].
Various methods have been proposed to solve this kind of multi-objective optimizationproblem [13, 14, 15, 16]; however, none of them is designed for the pre-training/fine-tuning paradigm,as well as they do not care about the zero-shot performance of a model.
Therefore, MTL methods arenot suitable for multi-lingual fine-tuning, where the zero-shot performance is indispensable.
Less-forgetting Multi-lingual Fine-tuning
Multi-lingual fine-tuning, where a multi-lingual language model is fine-tuned on task-specific datafrom a set of languages (defined as source languages), has two goals: (1) keep the cross-lingualzero-shot transfer ability of the multi-lingual language model and (2) achieving proper performanceon the source languages.
For the first goal, it is necessary to avoid the model from forgetting thecross-lingual knowledge it obtained in the pre-training phase.
For the second goal, we face a multiobjective optimization problem, where the losses for the source languages should be jointly optimized.
In this section, we firstly give a formal definition of the multi-lingual fine-tuning and then propose amethod (dubbed LF-MLF) that can simultaneously achieve the above two goals.
3.1 Problem Definition of Multi-lingual Fine-tuning
Given a multi-lingual language model θ p , which is pre-trained using large amounts of unlabeleddata from language set S p , we aim to fine-tune this model on a downstream task and achieve properperformance.
For the downstream task, we have task-specific labeled training data from a set ofsource languages S f , where S f ⊆ S p .
After fine-tuning, θ p has been updated as θ f .
Assume thereare T languages in source languages set S f .
For each language, the task-specific loss is denoted as
L t ( θ f ) .
Under the gradient descent paradigm of deep learning, in the k th iteration of multi-lingualfine-tuning, the model is updated by
[Equation]
where η is the learning rate, (cid:80) Tt =1 w kt = 1 and w kt ≥ 0 , t ∈ { 1 , ..., T } .
In Eq.
(1),− (cid:80) Tt
=1 w kt ∇ L t ( θ k − 1 ) is the convex hull of the gradient descent directions of the included languages, which contains feasible gradient descent directions for the multi-lingual fine-tuning.
Furthermore, denote the pre-training loss of θ p and θ f as L p ( θ p ) and L p ( θ f ) respectively.
Accordingto the classic definition of forgetting [17], we define the forgetting between θ p and θ f as F p =
L p ( θ f ) − L p ( θ p ) , which quantifies the performance degeneration of θ f on the multi-lingual pretraining task (e.g., masking tokens).
With the above notation, the two goals for multi-lingualfine-tuning can be formally defined as: (1) minimizing F p and (2) minimizing { L 1 ( θ f ) , ..., L T ( θ f ) } [-.3.
-].
3.2 Upper Bound the Forgetting in Multi-lingual Fine
-tuning In fine-tuning, we cannot directly compute the forgetting F p , because the pre-training the head (e.g.,masked language modeling head) has already been replaced by downstream-task-specific head (e.g.,a token classification head) and L p ( θ f ) is not accessible.
Alternatively, we propose to minimize theforgetting by minimizing its upper bound.
In Theorem 1, we give a upper bound for the forgetting ofmulti-lingual fine-tuning.
The assumptions in Theorem 1 are widely accepted [17, 18, 19] and canensure proper performance.
Theorem 1.
Assume L p ( θ f ) can be approximated by its second order Taylor expansion and θ p is aminima w.r.t the pre-training loss.
Then, we have
[Equation]
where λ p is the maximum eigenvalue of ∇ 2 L p ( θ p ) , and K is the number of iterations of fine-tuning.
Proof.
The proof is provided in the supplementary materials.
3.3 Less-forgetting Multi-lingual Fine-tuning
In multi-lingual fine-tuning, we not only need to keep the multi-lingual memory of the pretrained model but also need to achieve proper fine-tuning performance, i.e., minimizing F p and{ L 1 ( θ f ) , ..., L T ( θ f ) } together.
Achieving these two goals at same time is challenging and remainsunexplored.
In this section, we propose a novel method, dubbed Less-forgetting Multi-lingual
Fine-tuning (LF-MLF), to jointly achieve these two goals.
Specifically, we minimizing F p by tightening its upper bound introduced in Theorem 1.
Besides,minimizing { L 1 ( θ f ) , ..., L T ( θ f ) } is a multi-objective optimization (MOO) problem.
We solve this
MOO problem by the means of multi-objective gradient descent.
According to Theorem 1, in the k th iteration,
[Equation]
Input: Pre-trained model θ p , Number of iterations K .
for k ← 1 to K do if k =
{+= 
+}
then
Solve Problem.
(2) with c = [0 , 0 , ..., 0] ⊤ and obtain solution w ∗ 1 . θ [Equation] =
[Equation]
else Solve Problem. (2) and obtain solution w ∗ k . θ
[Equation]
end if if the feasible set of Problem.
(2) is empty then break. end if end for return θ K . where ¯ ∇ L = (cid:80) k − 1 i =1 (cid:80) T t =1 w it ∇ L t ( θ i − 1 ) and θ 0 = θ p . To reduce fogetting, in the k th iteration, we propose to minimize F kp ’s upper bound as in Eq.
[Equation]
Except for reducing forgetting, we also need to decrease the task-specific losses of the fine-tunedlanguages, i.e, move toward the objective of minimizing { L 1 ( θ f ) , ..., L T ( θ f ) } .
Thus, in the k thiteration, we need to update the model in a descent direction common to all the source languages,which is the principle of multi-objective gradient descent.
Under this principle, the update directionshould not have a obtuse angle with the gradient direction on each language as in Eq.
[Equation]
To reduce fogetting and minimizing the downstream task losses, we combine Eq.
(4) with Eq.
(5)and propose to formulate the less-forgetting multi-lingual fine-tuning as the following constrainedoptimization problem.
Problem 1.
[Equation]
Problem 1 can be formulated as the following quadratic programming problem with T variables and2 T + 1 constraints.
Problem 2.
[Equation]
where G = ∇ L ( θ k − 1 )
T ∇ L ( θ k − 1 ) , ∇ L ( θ k − 1 ) =
[ ∇ L 1 ( θ k − 1 ) , L 2 ( θ k − 1 ) , ..., L T ( θ k − 1 )] ⊤ .
c =∇ L ( θ k − 1 )
T ¯ ∇ L and b t = ∇ L ( θ k − 1 )
⊤ ∇ L t ( θ k − 1 ) .
Problem 2 can be effectively solved.
This paper adopts a interior-point solver [20].
Let w ∗ k be thesolution of the above quadratic programming problem.
In our proposed LF-MLF, the update rule is
[Equation]
Overall, the challenging multi-lingual fine-tuning problem is transferred to a simple quadraticprogramming problem.
The detailed steps of LF-MLF can be referred to Algorithm 1.
[-= 
-]
Theoretical Analysis
In less-forgetting multi-lingual fine-tuning, we explicitly set the goal, minimizing the upper bound, asthe optimization objective.
Therefore, the less-forgetting property of the proposed LF-MLF methodis straightforward.
By contrast, the target of minimizing { L 1 ( θ f ) , ..., L T ( θ f ) } is achieved by themeans of putting constraints on the update direction, which forces the update direction to be thecommon decreasing direction for all the fine-tuned languages.
Its effectiveness seems not obvious.
Tomake it clear, in this section, we conduct theoretical analysis on LF-MLF’s fine-tuning performance.
In multi-lingual fine-tuning, we aim to jointly fine-tune several languages, namely minimizing theloss vector { L 1 ( θ f ) , ..., L T ( θ f ) } .
It is a multi-objective optimization problem, where the optimalsolutions are Pareto optimal as in Definition 1.
Definition 1 (Pareto Optimality) .
For L ( θ ) =
[ { L 1 ( θ ) , ..., L T ( θ ) } ] ⊤ , the Pareto optimality w.r.t theoptimization objective min θ L ( θ ) is defined as follows: (1) A solution θ dominates a solution θ if
L t ( θ ) ≤ L t ( θ ) for all t ∈ T and
L ( θ ) ̸
= L ( θ ) and (2) A solution θ ∗ is deemed Pareto optimal ifthere exists no solution θ that dominates θ ∗ .
However, in deep learning, the parameter spaces are highly non-convex.
To the best of our knowledge,there is no method that can learn a Pareto optimal deep neural network model.
Settling for less, thebest we can do is to achieve the Pareto stationarity as in Definition 2.
Pareto stationarity is a necessarycondition of achieving Pareto optimality.
Definition 2 (Pareto Stationarity) .
For L ( θ 0 )
=
[ { L 1 ( θ 0 ) , ..., L T ( θ 0 ) } ]
⊤ is said to be Paretostationary at the design-point θ
iff there exists a convex combination of the gradient-vectors,
[Equation]
Tt
[Equation] 1 w t [Equation] L t ( θ 0 ) , that is equal to zero:
[Equation]
In Theorem 2 and 3, we propose that LF-MLF can achieve Pareto stationarity, which theoreticallyverifies that our proposed method can achieve proper performance on the fine-tuning languages.
Theorem 2.
Let H be a Hilbert space of finite dimension N .
Let L t ( θ k ) ( 1 ≤ t ≤ T ≤ N ) be Tsmooth functions of the vector θ k ∈ H , and θ 0 k a particular admissible design-point.
Let w ∗ k be thesolution of Problem 2 and descent direction ∇ L = (cid:80)
Tt =1 ( w ∗ k ) t ∇ L t ( θ k ) .
Then, either ∇ L = ∅ ,and [ L 1 ( θ 0 k ) , ..., L T ( θ 0 k )]
⊤ are pareto stationary at θ 0 k or ∇ L ̸ = ∅ and −∇ L is a descent directioncommon to all { L t ( θ k ) }
Tt =1 .
Proof.
The proof is provided in the supplementary materials.
Theorem 3.
LF-MLF can stop after a finite number of iterations if a Pareto stationary point isreached.
Otherwise, If the sequence of iterates { θ k } Kk =1 of the LF-MLF is infinite, it admits a weaklyconvergent subsequence.
Proof.
The proof is provided in the supplementary materials.
Experiments
In this section, we perform experimental studies on three downstream tasks– Named Entity Recognition (NER), Question Answering (QA) and Natural Language Inference (NLI) respectively to evaluatethe performance of our proposed LF-MLF and verify our theoretical analysis.
The implementation isbased on PyTorch [21] and XTREME benchmark [22].
The code is attached in the supplementarymaterials.
5.1 Experimental Setup
Datasets: In our experiments, we adopt the NER [23], TyDiQA-GoldP [24] and XNLI [25] datasetsfor NER, QA and NLI respectively from the XTREME benchmark [22].
The details of these datasetsare introuced in the supplementary material.
Baselines: We compare our proposed LF-MLF with the following baselines: (1) Single-lingual Finetuning (SLF): fine-tuning on each language independently; (2) Uniform-MLF: fine-tuning on theincluded languages simultaneously by means of a uniformly weighted sum of the language-specificgradient descent directions, i.e., w kt = 1 T constantly; (3) PCGrad-MLF: using the Project Conflicting
Gradients (PCGrad) method proposed by [26] on multi-lingual fine-tuning; (4) GradVac-MLF: usingthe Gradient Vaccine (GradVac) method proposed by [27] on multi-lingual fine-tuning.
Experimental Settings: In our experiments, we generally fine-tune XLM-RoBERTa models (basesized model) [28] with a training batch size of 32, and AdamW [29] is used as the optimizer.
As tothe numbers of fine-tuning epochs, we adopt the default setting of the XTREME benchmark, thatare 10, 3, 5 for NER, TyDiQA and XNLI respectively.
Besides, the learning rates are selected from{ 1 e − 5 , 2 e − 5 , 5 e − 5 } , { 1 e − 5 , 2 e − 5 , 5 e − 5 } and { 5 e − 6 , 1 e − 5 , 2 e − 5 } with grid search for
NER, TyDiQA and XNLI respectively.
All the results are averaged over 3 runs.
5.2 Performance of Multi-lingual Fine-tuning
Conducting experiments on NER, TyDiQA and XNLI, we compare LF-MLF with the baselines onthe zero-shot performance and fine-tuning performance respectively.
For each dataset, we choose twoor three source languages for fine-tuning according the following rules: (1) belongs to the Top 10high-resource languages w.r.t Wikipedias; (2) the higher-resource the better.
For NER and XNLI, en,de, fr are chosen, while en, ru are chosen for TyDiQA.
5.2.1 Zero-shot Performance
To evaluate the zero-shot performance of LF-MLF and the baselines, we record the average F1 scoreover the target languages that have not been fine-tuned (i.e,. the other 45 languages except en, de, frin NER, the other 7 languages except en, ru in TyDiQA and the other 10 languages except en, de, fr in
XNLI).
The results are reported in Table 1, 2 and 3 for NER, TyDiQA and XNLI respectively.
Theseresults demonstrate: (1) comparing with the single lingual fine-tuning, multi-lingual language finetuning can significantly improve the zero-shot performance; (2) LF-MLF outperforms the baselinemethods on all the three applications.
[Table]
[Table] [Table]
average F1 score over the other 12 languages) on XNLI.
5.2.2 Fine-tuning Performance
Except the zero-shot performance, we also care about the fine-tuning performance, namely the F1score on the languages that have been fine-tuned.
The results of fine-tuning performance are reportedin Table 4, 5 and 6 for NER, TyDiQA and XNLI respectively. From these results, we can see that: [Table]
[Table] [Table] (1) the multi-lingual methods do not clearly outperform single-lingual methods on correspondinglanguages; (2) LF-MLF’s performance is comparable to the baselines, and there is no baseline methodcan dominate LF-MLF.
PCGrad-MLF and GradVac-MLF have inferior performance comparing with
Uniform-MLF and LF-MLF.
It because that PCGrad-MLF and GradVac-MLF cannot guarantee tofind a Pareto stationarity, while Uniform-MLF and LF-MLF can find a Pareto stationarity.
It verifiesthe theoretical analysis proposed in Section 4.
Overall, from section 5.2.1 and 5.2.2, we can see that LF-MLF has superior zero-shot performanceand comparable fine-tuning performance.
It verifies that LF-MLF has superior cross-lingual transferability.
The superiority comes from LF-MLF’s less-forgetting property, because such property enables
LF-MLF to remember more about the target languages.
5.3 Impact of the Number of the Source Languages
In the last section, it is concluded that multi-lingual fine-tuning can significantly improve the zero-shotperformance.
Then, it is natural to ask whether the zero-shot performance can be further improvedwhen more languages have been fine-tuned ?
In this section, we experimentally present that the answeris yes .
Besides, we also investigate the impact of the number of source languages on the fine-tuningperformance.
Our experiments are conducted on the NER dataset, and the source languages areselected from the Top 10 high-resource languages w.r.t Wikipedias, i.e., en, de, fr, ru, es, ja, it, zh,ko, ar .
By contrast, the other thirty eight languages are used as target languages.
In the experiments,we increase the source languages from Top 2 to Top 10 and record the changing of zero-shot andfine-tuning performance.
5.3.1 Impact on the Zero-shot Performance
Figure 1 illustrates the change of MLF’s zero-shot performance along with the increasing numberof source languages.
From this figure, we can conclude that (1) generally speaking, the zero-shotperformance of a multi-lingual model continuously improves with the increasing of the number ofsource languages; (2) LF-MLF consistently outperforms Uniform-MLF when the number of sourcelanguages changes.
5.3.2 Impact on the Fine-tuning Performance
To evaluate the fine-tuning performance over different number of source languages, we present ametric, dubbed average relative F1 score, which measures the average relative improvement of multilingual methods’s fine-tuning performance comparing with the single-lingual fine-tuning.
Specifically,let S st be the F1 score of the single-lingual fine-tuning on language t , and let S mt be the F1 score of the multi-lingual fine-tuning on language t .
Assume there are T fine-tuning languages, the averagerelative F1 score is defined as R T
[Equation]
Figure 1 [-S c o r e 81.37 80.84 81.08 81.
 en de fr it en de bn hi en de bg ru fr it bn hi fr it bg ru bn hi bg ru en de fr bn en de fr ru en de bn ru fr it bn ru en fr bn ru 4 Combinations of 2 4 Combinations of 3 4 Combinations-]demonstrates the change of fine-tuning performance along with the increasing number ofsource languages.
From this figure, we can see that (1) the fine-tuning performance just slightlyfluctuates when the number of source languages changes; (2) LF-MLF and Uniform-MLF do notevidently outperform single-lingual fine-tuning, and they have similar average relative F1 score overdifferent number of source languages.
5.4 Impact of the Diversity of Language Branches Covered by the Source Languages
After investigating the impact of the number of source languages, we further explore whether thediversity of language branches covered by the source languages impacts the zero-shot performance.
In this section, we conduct experiments on the different branches of Indo-European language familybased on the NER dataset.
Specifically, we choose sixteen languages, which belong to four differentlanguage branches (Germanic, Italic, Indo-Iranian, Balto-Slavic) and each branch has four languages,from the NER dataset.
Firstly, we investigate the cross-language-branches transfer performance, namely fine-tuning on thefour languages of a branch and observe the zero-shot performance on another branch, such as finetuning on Germanic and test zero-shot performance on Italic.
The experimental results are illustratedin Fig.
3 (a) and Fig.
4 (a) for LF-MLF and Uniform-MLF respectively.
These results present that:(1)
the cross-language-branches transfer performance of each pair of branches is asymmetric; (2) LFMLF has better cross-language-branches transfer ability than Uniform-MLF, which experimentallyexplains the superiority of LF-MLF and further verifies LF-MLF’s less forgetting property.
Next, we explore the impact of language branches’ diversity of the source languages.
In thisexploration, eight languages are used as candidates for fine-tuning (i.e., source languages), while the [Figure] [Figure]
en de fr it en de bn hi en de bg ru fr it bn hi fr it bg ru bn hi bg ru en de fr bn en de fr ru en de bn ru fr it bn ru en fr bn ru 4 Combinations of 2 4 Combinations of 3 4 Combinations of
[Figure]
other eight languages are used to test the zero-shot performance.
The detailed allocation of theselanguages are reported in Table 7.
Fixing the number of fine-tuning languages as four, we test thelanguage branches’ combinations that the source languages cover two branches, three branches andall four branches respectively, and the results are illustrated in Fig.
3 (b) and Fig.
4 (b).
From thesefigures, we can see that: (1) the language branches’ diversity of the fine-tuning languages does nothave a clear impact on the zero-shot performance, which means that covering more language branchesmight not lead to better zero-shot performance; (2) LF-MLF generally outperforms Uniform-MLF onmost of the combinations. [Figure] [Table] [Figure]
[Table]
[Figure]
Conclusion
This paper researches on a novel fine-tuning setting for multi-lingual language models, namelymulti-lingual fine-tuning; furthermore, we propose a novel multi-lingual fine-tuning method, dubbed
Less-forgetting Multi-lingual Fine-tuning (LF-MLF).
Comparing with its mono-lingual fine-tuningcounterparts, multi-lingual fine-tuning can achieve superior zero-shot performance; moreover, ourproposed LF-MLF outperforms the baseline multi-lingual fine-tuning methods.
Researching onmono-lingual fine-tuning provides several inspirations and has the potential to forge new
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
[Table]
mental results (either in the supplemental material or as a URL)?
[Yes]
[Table]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they
[Table]
were chosen)?
[Yes]
[Table]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[No]
[Table] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
[Table]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
(b) Did you mention the license of the assets?
[Yes]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]
using/curating?
[N/A]
[Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable [Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A] [Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]