Meta-Learning with Self-Improving Momentum Target
Anonymous Author(s)
AffiliationAddress email
Abstract
The idea of using a separately trained target model (or teacher ) to improve theperformance of the student model has been increasingly popular in various machinelearning domains, and meta-learning is no exception; a recent discovery shows thatutilizing task-wise target models can significantly boost the generalization performance.
However, obtaining a target model for each task can be highly expensive,especially when the number of tasks for meta-learning is large.
To tackle this issue,we propose a simple yet effective method, coined Self-improving Momentum Target(SiMT) .
SiMT generates the target model by adapting from the temporal ensembleof the meta-learner, i.e., the momentum network.
This momentum network and itstask-specific adaptations enjoy a favorable generalization performance, enablingself-improving of the meta-learner through knowledge distillation.
Moreover, wefound that perturbing parameters of the meta-learner, e.g., dropout, further stabilizethis self-improving process by preventing fast convergence of the distillation lossduring meta-training.
Our experimental results demonstrate that SiMT brings asignificant performance gain when combined with a wide range of meta-learningmethods under various applications, including few-shot regression, few-shot classification, and meta-reinforcement learning.
Introduction
Meta-learning [38] is the art of extracting and utilizing the knowledge from the distribution of tasksto better solve a relevant task.
This problem is typically approached by training a meta-model that cantransfer its knowledge to a task-specific solver where the performance of the meta-model is evaluatedon the basis of how well each solver performs on the corresponding task.
To learn such meta-model,one should be able to (a) train an appropriate solver for each task utilizing the knowledge transferredfrom the meta-model and (b) accurately evaluate the performance of the solver.
A standard way to dothis is the so-called S / Q (support/query) protocol [41, 25]: for (a), use a set of support set samples totrain the solver; for (b), use another set of samples, called query set samples to evaluate the solver.
Recently, however, an alternative paradigm—called S / T (support/target) protocol—has receivedmuch attention [44, 47, 23].
The approach assumes that the meta-learner has an access to task-specifictarget models , i.e., an expert model for each given task, and uses these models to evaluate task-specificsolvers by measuring the discrepancy of the solvers from the target models.
Intriguingly, it has beenobserved that such knowledge distillation procedure [31, 15] helps to improve the meta-generalizationperformance [47], in a similar way that such teacher-student framework helps to avoid overfittingunder non-meta-learning contexts [21, 16].
Despite such advantage, the S / T protocol is difficult to be used in practice, as training target modelsfor each task usually requires an excessive amount of compute, especially when the number of tasksis large.
Prior works aim to alleviate this issue by proposing methods to generate target models in
[Figure]
a compute-efficient manner.
For instance, Lu et al.
[23] consider the case where the learner has anaccess to a model pre-trained on a global data domain that covers most tasks (to be meta-trainedupon), and propose to generate task-wise target models by simply fine-tuning the model for each task.
However, the method still requires the compute for fine-tuning on a large number of tasks, and moreimportantly, is hard to be used when there is no effective pre-trained model available at hand.
Forexample, a globally pre-trained model is usually not available in reinforcement learning, as collecting“global” data is a nontrivial task [6].
In this paper, we ask whether we can generate the task-specific target models by (somewhat ironically)using meta-learning.
We draw inspiration from recent observations in semi/self-supervised learningliterature [37, 11, 4] that the temporal ensemble of a model, i.e., the momentum network [18], canbe an effective teacher of the original model.
It turns out that a similar phenomenon happens inthe meta-learning scenario: one can construct a momentum network of the meta-model, whosetask-specific adaptation is an effective target model from which the task-specific knowledge can bedistilled to train the original meta-model.
Contribution.
We establish a novel framework, coined Meta-Learning with Self-improving Momentum Target (SiMT) , which brings the benefit of the S / T protocol to the S / Q -like scenario wheretask-specific target models are not available (but have access to query data).
In a nutshell, SiMT iscomprised of two (iterative) steps:
• Momentum target : We generate the target model by adapting from the momentum network, whichshows better adaptation performance than the meta-model itself.
In this regard, generating the
target model becomes highly efficient, e.g., one single forward is required when obtaining
the momentum target for ProtoNet [33].
• Self-improving process : The meta-model enables to improve through the knowledge distillation
from the momentum target, and this recursively improves the momentum network by the temporal
ensemble.
Furthermore, we find that perturbing parameters of the task-specific solver of themeta-model, e
g., dropout [35], further stabilizes this self-improving process by preventing fastconvergence of the distillation loss during meta-training.
We verify the effectiveness of SiMT under various applications of meta-learning, including fewshot regression, few-shot classification, and meta-reinforcement learning (meta-RL).
Overall, ourexperimental results show that incorporating the proposed method can consistently and significantlyimprove the baseline meta-learning methods [7, 22, 27, 33].
In particular, our method improves thefew-shot classification accuracy of Conv4 [41] trained with MAML [7] on mini-ImageNet [41] from47.33% → 51.49% for 1-shot, and from 63.27% → 68.74% for 5-shot, respectively.
Moreover, weshow that our framework could even notably improve on the few-shot regression and meta-RL tasks,which supports that our proposed method is indeed domain-agnostic.
Related work
Learning from target models.
Learning from an expert model, i.e., the target model, has shown itseffectiveness across various domains [21, 26, 50, 39].
As a follow-up, recent papers demonstrate thatmeta-learning can also be the case [44, 47].
However, training independent task-specific target modelsis highly expensive due to the large space of task distribution in meta-learning.
To this end, recentwork suggests pre-training a global encoder on the whole meta-training set and finetune target modelson each task [23]; however, they are limited to specific domains and still require some computations,e.g., they take more than 6.5 GPU hours to pre-train only 10% of target models while ours require
GPU hours for the entire meta-learning process (ProtoNet [33] of ResNet-12 [25]) on the same GPU.
Another recent relevant work is bootstrapped meta-learning [8], which generates the target modelfrom the meta-model by further updating the parameters of the task-specific solver for some numberof steps with the query dataset.
While the bootstrapped target models can be obtained efficiently, theirapproach is specialized in gradient-based meta-learning schemes, e.g., MAML [7].
In this paper, wesuggest an efficient and more generic way to generate the target model during the meta-training.
Learning with momentum networks.
The idea of temporal ensembling, i.e., the momentum network,has become an essential component of the recent semi/self-supervised learning algorithms [2, 4].
Forexample, Mean Teacher [37] first showed that the momentum network improves the performance ofsemi-supervised image classification, and recent advanced approaches [1, 34] adopted this idea forachieving state-of-the-art performances.
Also, in self-supervised learning methods which enforceinvariance to data augmentation, such momentum networks are widely utilized as a target network[14, 11] to prevent collapse by providing smoother changes in the representations.
In this paper, weempirically demonstrate that the momentum network shows better adaptation performance compareto the original meta-model, which motivates us to utilize it for generating the target model in acompute-efficient manner.
Problem setup and evaluation protocols
In this section, we formally describe the meta-learning setup under consideration, and S / Q and
S / T protocols studied in prior works.
Problem setup: Meta-learning.
Let p ( τ ) be a distribution of tasks.
The goal of meta-learning isto train a meta-model f θ , parameterized by the meta-model parameter θ , which can transfer itsknowledge to help to train a solver for a new task.
More formally, we consider some adaptationsubroutine Adapt ( · , · ) which uses both information transferred from θ and the task-specific dataset(which we call support set ) S τ to output a task-specific solver as ϕ τ =
Adapt ( θ, S τ ) . For example,the model-agnostic meta-learning algorithm (MAML; [7]) uses the adaptation subroutine of takinga fixed number of SGD on S τ , starting from the initial parameter θ .
In this paper, we aim to give ageneral meta-learning framework that can be used in conjunction with any adaptation subroutine,instead of designing a method specialized for a specific one.
The objective is to learn a nice meta-model parameter θ from a set of tasks sampled from p ( τ ) (orsometimes the task distribution itself), such that the expected loss of the task-specific adaptationsis small, i.e., min θ E τ ∼ p ( τ ) [ ℓ τ ( Adapt ( θ, S τ ))] , where ℓ τ ( · ) denotes the test loss on task τ .
To trainsuch meta-model, we need a mechanism to evaluate and optimize θ (e.g., via gradient descent).
Forthis purpose, existing approaches take one of two approaches: the S / Q protocol or the S / T protocol.
S / Q protocol.
The majority of the existing meta-learning frameworks (e.g., [41, 25]) splits thetask-specific training data into two, and use them for different purposes.
One is the support set S τwhich is used to perform the adaptation subroutine.
Another is the query set Q τ which is used forevaluating the performance of the adapted parameter and compute the gradient with respect to θ .
Inother words, given the task datasets ( S 1 , Q 1 ) , ( S 2 , Q 2 ) , . . .
, ( S N , Q N ) , 1 the S / Q protocol solves
[Equation]
where L ( ϕ, Q ) denotes the empirical loss of a solver ϕ on the dataset Q .
S / T protocol.
Another line of work considers the scenario where the meta-learner additionally hasan access to a set of target models ϕ target for each training task [44, 23].
In such case, one canuse a teacher-student framework to regularize the adapted solver to behave similarly (or have lowprediction discrepancy , equivalently) to the target model.
Here, a typical practice is to not split eachtask dataset and measure the discrepancy using the support dataset that is used for the adaptation[23].
In other words, given the task datasets S 1 , S 2 , . . .
, S N and the corresponding target modelsϕ τ 1 target , ϕ τ 2 target , . . .
, ϕ τ N target , the S / T protocol updates the meta-model by solving
[Equation]
where L teach ( ϕ, ϕ target , S ) denotes a discrepancy measure between the adapted model ϕ and thetarget model ϕ target , measured using the dataset S .
Meta-learning with self-improving momentum target
In this section, we develop a compute-efficient framework which bring the benefits of S / T protocolto the settings where we do not have access to target-specific tasks or a general pretrained model, asin general S / Q -like setups.
In a nutshell, our framework iteratively generates a meta-target modelwhich generalizes well when adapted to the target tasks, by constructing a momentum network [37]of the meta-model itself.
The meta-model is then trained, using both the knowledge transferred fromthe momentum target and the knowledge freshly learned from the query sets.
We first briefly describeour meta-model update protocol (Section 4.1), and then the core component, coined Self-Improving
Momentum Target (SiMT) , which efficiently generates the target model for each task (Section 4.2).
4.1 Meta-model update with a S / Q - S / T hybrid loss
To update the meta-model, we use a hybrid loss function of the S / Q protocol (1) and the S / T protocol(2).
Formally, let ( S 1 , Q 1 ) , ( S 2 , Q 2 ) , . . .
, ( S N , Q N ) be given task datasets with support-query split,and let ϕ τ 1 target , ϕ τ 2 target , . . .
, ϕ τ N target be task-specific target models generated by our target generationprocedure (which will be explained with more detail in Section 4.2).
We train the meta-model asmin θ
[Equation]
i [Equation] where λ ∈ [0 , 1) is the weight hyperparameter.
We note two things about Eq. 3.
First, while we aretraining using the target model, we also use a S / Q loss term.
This is because our method trains themeta-target model and the meta-model simultaneously from scratch, instead of requiring fully-trainedtarget models.
Second, unlike in the S / T protocol, we measure the discrepancy L teach using thequery set Q τ i instead of the support set.
By doing so, the predictions of target models tend to generate“soft predictions” than on the support set, which is known to improve the generalization of the studentmodel in the knowledge distillation literature [49, 36].
4.2 SiMT: Self-improving momentum target
We now describe the algorithm we propose, SiMT (Algorithm 1), to generate the target model ina compute-efficient manner.
In a nutshell, SiMT is comprised of two iterative steps: momentumtarget and self-improving process .
To efficiently generate a target model, SiMT utilizes the temporalensemble of the network, i.e., the momentum network, then distills the knowledge of the generatedtarget model into the task-specific solver of the meta-model to form a self-improving process.
Momentum target.
For the compute-efficient generation of target models, we utilize the momentumnetwork θ moment of the meta-model.
Specifically, after every meta-model training iteration, we computethe exponential moving average of the meta-model parameter θ as
[Equation]
where η ∈ [0 , 1) is the momentum coefficient.
We find that θ moment can adapt better than the metamodel θ itself and observe that the loss landscape has flatter minima (see Section 5.4), which can be a
Require: Distribution over tasks p ( τ ) , adaptation subroutine Adapt ( · ) , momentum coefficient η , weight hyperparameter λ , dropout probability p , task batch size N , learning rate β . 1: Initialize
3: while not done do 4:
Sample N tasks { τ i } Ni =1 from p ( τ ) 5: for i = 1 to N do 6:
Sample support set S τ i and query set Q τ i from τ i 7: ϕ τ i moment =
Adapt ( θ moment , S τ i ) .
▷ Generate a momentum target.
8: ϕ τ i =
Adapt ( θ, S τ i ) .
▷ Adapt a task-specific solver.
9: ϕ τ i drop =
Dropout ( ϕ τ i , p ) .
▷ Perturb the solver.
L τ i total ( θ ) =
(1 − λ )
· L ( ϕ τ i drop , Q τ i ) + λ · L teach ( ϕ τ i drop , ϕ τ i moment , Q τ i )
▷ Compute loss.
11: end for 12: θ ← θ − βN · ∇ θ (cid:80) N i =1 L τ i total ( θ ) .
▷ Train the meta-model.
13: θ moment ← η · θ moment + (1 − η ) · θ .
▷ Update the momentum network.
14: end while hint for understanding the generalization improvement [20, 9].
Based on this, we propose to generatethe task-specific target model, i.e., the momentum target ϕ moment , by adapting from the momentumnetwork θ moment .
For a given support set S , we generate the target model for each task as
[Equation]
We remark that generating momentum targets does not require an excessive amount of compute (see
Section 5.4), e.g., ProtoNet [33] requires a single forward of a support set, and MAML [7] requiresfew-gradient steps without second-order gradient computation for the adaptation.
Self-improving process via knowledge distillation.
After generating the momentum target, weutilize its knowledge to improve the generalization performance of the meta-model.
To this end, wechoose the knowledge distillation scheme [15], which is simple yet effective across various domains,including meta-learning [23].
Here, our key concept is that the momentum target self-improvesduring the training due to the knowledge transfer.
To be specific, the knowledge distillation fromthe momentum target improves the meta-model itself, which recursively improves the momentumthrough the temporal ensemble.
Formally, for a given query set Q , we distill the knowledge of themomentum target ϕ moment to the task-specific solver of the meta-model ϕ as
[Equation]
where l KD is the distillation loss and | · | is the cardinality of the set.
For regression tasks, we use the
MSE loss, i.e., l KD ( z 1 , z 2 ) :
=
∥ z 1 − z 2 ∥ 22 , and for classification tasks, we use the KL divergence withtemperature scaling [12], i.e., l KD ( z 1 , z 2 ) :
= T 2 · KL ( σ ( z 1 /T )
∥ σ ( z 2 /T )) , where T is the temperaturehyperparameter, σ is the softmax function and z 1 , z 2 are logits of the classifier, respectively.
Wepresent the detailed distillation objective of reinforcement learning tasks in the Appendix.
Also,note that optimizing the distillation loss only propagates gradients to the meta-model θ , not to themomentum network θ moment , i.e., known as the stop-gradient operator [4, 5].
Furthermore, we find that the distillation loss (6) sometimes converges too fast during the metatraining, which can stop the self-improving process.
To prevent this, we suggest perturbing theparameter space of ϕ .
Intuitively, injecting noise to the parameter space of ϕ forces an asymmetricitybetween the momentum target’s prediction, hence, preventing f ϕ and f ϕ moment from reaching a similarprediction.
To this end, we choose the standard dropout regularization [35] due to its simplicity andgenerality across architectures and also have shown its effectiveness under distillation research [46]:ϕ drop :=
Dropout ( ϕ, p ) where p is the probability of dropping activations.
In the end, we use theperturbed task-specific solver ϕ drop and the momentum target ϕ moment for our evaluation protocol (3).
[Table] [Table]
Experiments
In this section, we experimentally validate the effectiveness of the proposed SiMT by measuringits performance on various meta-learning applications, including few-shot regression (Section 5.1),few-shot classification (Section 5.2), and meta-reinforcement learning (meta-RL; Section 5.3).
Common setup.
By following the prior works, we chose the checkpoints and the hyperparameters onthe meta-validation set for the few-shot learning tasks [24, 42].
For RL, we chose it based on the bestaverage return during the training [7].
We find that the hyperparameters, e.g., momentum coefficientη or the weight hyperparameter λ , are not sensitive across datasets and architectures but can vary onthe type of the meta-learning scheme or tasks.
We provide further details in the Appendix.
Moreover,we report the adaptation performance of the momentum network for SiMT.
5.1 Few-shot regression
For regression tasks, we demonstrate our experiments on ShapeNet [10] and Pascal [48] datasets,where they aim to predict the object pose of a gray-scale image relative to the canonical orientation.
To this end, we use the following empirical loss L to train the meta-model: the angular loss for the
ShapeNet ( (cid:80) ( x,y ) ∈Q ∥ cos( f ϕ ( x )) − cos( y ) ∥ 2 + ∥ sin( f ϕ ( x )) − sin( y ) ∥ 2 ) and the MSE loss for the
[Table]
Pascal ( (cid:80) ( x,y ) ∈Q ∥ f ϕ ( x ) − y ∥ 2 ), following the prior works [48, 10].
For the backbone meta-learningschemes we use gradient-based approaches, including MAML [7], ANIL [27], and MetaSGD [22].
For all methods, we train the convolutional neural network with 7 layers [48] and apply dropoutregularization [35] before the max-pooling layer for SiMT.
Table 1 summarizes the results, showingthat SiMT significantly improves the overall meta-learning schemes in all tested cases.
5.2 Few-shot classification
For few-shot classification tasks, we use the cross-entropy loss for the empirical loss term L totrain the meta-model θ .
, i.e., (cid:80) ( x,y ) ∈Q l ce ( f ϕ ( x ) , y ) where l ce is the cross-entropy loss.
We train themeta-model on mini-ImageNet [41] and tiered-ImageNet [28] datasets, following the prior works[23, 42].
Here, we consider the following gradient-based and metric-based meta-learning approachesas our backbone algorithm to show the wide usability of our method: MAML, ANIL, MetaSGD, and
ProtoNet [33].
We train each method on Conv4 [41] and ResNet-12 [25], and apply dropout beforethe max-pooling layer for SiMT.
For the training details, we mainly follow the setups from eachbackbone algorithm paper.
See the Appendix for more details.
In-domain adaptation.
In this setup, we evaluate the adaptation performance on different classesof the same dataset used in meta-training.
As shown in Table 2, incorporating SiMT into existingmeta-learning methods consistently and significantly improves the in-domain adaptation performance.
In particular, SiMT achieves higher accuracy gains on the mini-ImageNet dataset, e.g., 5-shotperformance improves from 63.27% → 68.74% on Conv4.
We find that this is due to the overfittingissue of backbone algorithms on the mini-ImageNet dataset, where SiMT is more robust to suchissues.
For instance, when training mini-ImageNet 5-shot classification on Conv4, MAML starts tooverfit after the first 40% of the training process, while SiMT does not overfit during the training.
Cross-domain adaptation.
We also consider the cross-domain adaptation scenarios.
Here, we adaptthe meta-model on different datasets from the meta-training: we use CUB [43] and Cars [17] datasets.
Such tasks are known to be challenging, as there exists a large distribution shift between trainingand testing domains [13].
Table 3 shows the results.
Somewhat interestingly, SiMT also improvesthe cross-domain adaptation performance of the base meta-learning methods across the considereddatasets.
These results indicate that SiMT successfully learns the ability to generalize to unseen taskseven for the distributions that highly differ from the training.
[Table]
5.3 Reinforcement learning
[Table]
The goal of meta-RL is training an agent to quickly adapt a policy to maximize the expected return forunseen tasks using only a limited number of sample trajectories.
Since the expected return is usuallynot differentiable, we use policy gradient methods to update the policy.
Specifically, we use vanillapolicy gradient [45], and trust-region policy optimization (TRPO; [29]) for the task-specific solverand meta-model, respectively, following MAML [7].
The overall training objective of meta-RL is inthe Appendix, including the empirical loss L , and the knowledge distillation loss L teach .
We evaluate
SiMT on continuous control tasks based on OpenAI Gym [3] environments.
In these experiments, wechoose MAML as our backbone algorithm, and train a multi-layer perceptron policy network withtwo hidden layers of size 100 by following the prior setup [7].
We find that the distillation loss isalready quite effective even without the dropout regularization, and applying it does not improvemore.
We conjecture that dropout on such a small network may not be effective as it is designed toreduce the overfitting of large networks [35].
We provide more experimental details in the Appendix.
[Equation]
[Figure]
2D Navigation.
We first evaluate
Figure 2 shows the adaptation performanceof learned models with up to 3 gradient updates.
These results demonstrate that SiMT(red) could consistently improve the adaptation performance of MAML (blue).
Also,
SiMT makes faster performance improvements than vanilla MAML with additionalgradient updates.
Locomotion.
To further demonstrate ourmethod, we also study high-dimensional,complex locomotion tasks based on the MuJoCo [40] simulator.
We choose a set of goal directiontasks with a planar cheetah (“Half-cheetah”), following previous works [7, 27].
In the goal directiontasks, the reward is the magnitude of the velocity in either the forward or backward direction,randomly chosen for each task.
Figure 2b shows that SiMT significantly improves the adaptationperformance of MAML even with a single gradient step.
5.4 Ablation study
Throughout this section, unless otherwise specified, we perform the experiments in 5-shot in-domainadaptation on mini-ImageNet with Conv4, where MAML is the backbone meta-learning scheme.
Component analysis.
We perform an analysis on each component of our method in both 1-shotand 5-shot classification on mini-ImageNet: namely, the use of (a) the momentum network θ moment ,(b) the distillation loss
L teach (6), and (c) the dropout regularization Dropout ( · ) , by comparing theaccuracies.
The results in Table 4 show each component is indeed important for the improvement.
We find that a naïve combination of the distillation loss and the momentum network does not showsignificant improvements.
But, by additionally applying the dropout, the distillation loss becomesmore effective and further improves the performance.
Note that this improvement does not fully comefrom the dropout itself, as only using dropout slightly degrades the performance in some cases.
[Figure] [Figure]
We visualize the loss landscape of the momentum network θ moment and the meta-model θ , togive insights into the generalization improvement.
To do this, we train MAML with a momentum network (without distillation and dropout)and visualize the loss by perturbing each parameter space [20] (See the Appendix for thedetail of the visualization method).
As shown in
Figure 4, the momentum network forms a flatter loss landscape than the meta-model, whererecent studies demonstrate that such a flat landscape is effective under various domains [9].
Computational efficiency.
Our method may be seemingly compute-inefficient when incorporatingmeta-learning methods (due to the momentum target generation); however, we show that it is not.
Although SiMT increases the total training time of MAML by roughly 1.2 times, we have observedthat it is 3 times faster to achieve the best performance of MAML: in Figure 3a, we compare theaccuracy under the same training wall-clock time with MAML.
Comparison of the momentum network and meta-model.
To understand how the momentumnetwork improves the performance of the meta-model, we compare the adaptation performance of themomentum network and the meta-model during training SiMT.
As shown in Figure 3b, we observethat the performance of the momentum network is consistently better than the meta-model, whichimplies that the proposed momentum target is a nice target model in our self-improving mechanism.
Discussion and conclusion
In this paper, we propose a simple yet effective method, SiMT, for improving meta-learning.
Our keyidea is to efficiently generate target models using a momentum network and utilize its knowledge toself-improve the meta-learner.
Our experiments demonstrate that SiMT significantly improves theperformance of meta-learning methods on various applications.
Limitations and future work.
While SiMT is a compute-efficient way to use target models inmeta-learning, it is still built on top of existing meta-model update techniques.
Since existing metalearning methods have limited scalability (to large-scale scenarios) [32], SiMT is no exception.
Hence,improving the scalability of meta-learning schemes is an intriguing future research direction, wherewe believe incorporating SiMT into such scenarios is worthwhile.
Potential negative impacts.
Meta-learning often requires a large computation due to the numeroustask adaptation during meta-training, therefore raising environmental concerns, e.g., carbon generation[30].
As SiMT is built upon the meta-learning methods, practitioners may need to consider somecomputation for successful training.
To address this issue, efficient meta-learning schemes [42] orlightweight methods for meta-learning [19] would be required for the applications. Checklist For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] The code and
instructions are in the supplementary material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 5 and the Appendix for the training details.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section 5.
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See the Appendix for details.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]