Modeling the Machine Learning Multiverse
Anonymous Author(s)
AffiliationAddress email
Abstract
Amid mounting concern about the reliability and credibility of machine learningresearch, we present a principled framework for making robust and generalizableclaims: the Multiverse Analysis.
Our framework builds upon the Multiverse
Analysis [1] introduced in response to psychology’s own reproducibility crisis.
To efficiently explore high-dimensional and often continuous ML search spaces,we model the multiverse with a Gaussian Process surrogate and apply Bayesianexperimental design.
Our framework is designed to facilitate drawing robustscientific conclusions about model performance, and thus our approach focuses onexploration rather than conventional optimization.
In the first of two case studies,we investigate disputed claims about the relative merit of adaptive optimizers.
Second, we synthesize conflicting research on the effect of learning rate on thelarge batch training generalization gap.
For the machine learning community,the Multiverse Analysis is a simple and effective technique for identifying robustclaims, for increasing transparency, and a step toward improved reproducibility.
Introduction
Machine learning research faces mounting concern about the reliability of our results and thecredibility of our claims [2–20].
The field of psychology has faced a similar crisis [21] andconfrontation with its shortcomings has sparked practical innovations [1, 22–25].
One such innovationis of direct relevance to the machine learning community: the Multiverse Analysis [1].
Throughout any investigation scientists make decisions about how to perform their work.
Inpsychology as in other disciplines, there are a plethora of different ways to conduct and analyzeexperiments.
From just a handful of choices, we reach such a large decision tree that researchers canrepeatedly try different paths until they chance upon a positive result [26].
Even without consciousmanipulation, these decision points pose a fundamental problem: what if the psychologist had chosenan alternate—and perfectly reasonable—route through this garden of forking paths [27]?
Would theirresults and conclusions still stand?
In their replication of a contentious study [28] on the effect of menstrual cycle and relationship statuson women’s political preferences, Steegen et al.
[1] identify both decisions taken and reasonablealternatives.
1 They name the Cartesian product of alternatives a multiverse , a set of parallel universeseach containing a slightly different study.
In their Multiverse Analysis , Steegen et al.
re
-run the studyas if inside each universe, finding that the alleged effect of relationship status and fertility is highlysensitive to different choices, in particular the definition of “single”.
In a machine learning context, we present the Multiverse Analysis as a principled framework foranalyzing robustness and generality.
Consider an example of some modification to a model, say batchnormalization [29].
To verify batch norm’s efficacy, one needs a test bed including model architecture,optimization method, dataset, evaluation metric, and so on.
Regardless of these specific choices, wewould like that batch norm be effective in general.
With a Multiverse Analysis, we can systematicallyexplore the effect of each choice, and understand the circumstances in which a claim holds true.
Our primary contribution is to introduce the Multiverse Analysis to ML, which we use to drawmore robust conclusions about model performance.
To efficiently explore the high-dimensionaland often continuous ML search space, we model the multiverse with a Gaussian Process (GP)surrogate and use Bayesian experimental design (§ 2).
We present motivating evidence that choosingexploration over optimization—the essence of the Multiverse Analysis—is essential when we seekproper understanding of our claims and their generality (§ 3).
In the first of two case studies, we usea Multiverse Analysis over hyperparameters to replicate an experiment on adaptive optimizers [30],finding that claims on the relative merits of optimizers are highly sensitive to learning rate (§ 4).
Second, we perform a more extensive Multiverse Analysis to synthesize divergent research on thelarge batch “generalization gap” [31], finding an interaction effect of batch size and learning rate(§ 5).
We conclude by discussing the limitations of our approach and directions for future work (§ 6).
Efficient multiverse exploration
Our approach to the Multiverse Analysis packages ML researcher decisions into a simple framework,requiring only a choice of search space X and an evaluation function ℓ , which together define ourmultiverse.
This search space defines the set of reasonable choices.
For example, we might consideronly a few directly relevant hyperparameters (§ 4); include purportedly irrelevant choices such asdataset (§ 5); or conduct even more expansive analyses (§ 6).
The evaluation function should pertainto the hypothesis being tested.
Here, we define it as model test accuracy (§ 5) or difference in testaccuracy (§ 4).
A barrier to analyzing any ML multiverse is the size of the search space and the presence of continuousdimensions.
This makes exhaustive search intractable.
To overcome this, we use a GP surrogate—modeling ℓ as a function of X —as a stand-in for the multiverse.
Using the surrogate, we iterativelyexplore the space using Bayesian experimental design [32]:
Sample an initial design, X 0 ∼ X , and evaluate the ℓ at each point, Y 0 =
ℓ ( X 0 ) ;2.
Fit a GP model f to the sampled points X 0 and corresponding results Y 0 ;3.
Use an acquisition function a on f to sample and evaluate a new batch ( X i , Y i );4.
Repeat steps 2–3 until an appropriate stopping criterion.
In step 1, our initial design is drawn from a Sobol sequence, a low discrepancy sequence that achievesimproved coverage over uniform random sampling in higher dimensions [33].
The search space andthe evaluation function specific to the analysis at hand and described later.
In step 2, we model the output of our evaluation function as a noisy function of the inputs,
[Equation]
Placing a GP prior over f with zero mean and positive definite kernel function k ,f ∼ GP (0 , k ) , (2) we obtain the posterior mean µ and variance σ 2 [34, chap. 2]:
[Equation] [Equation]
[Equation]
where K is the kernel matrix and k
=
[ k ( x i +1 , x 0 ) , . . .
, k ( x i +1 , x i )] .
We use a Matérn 5/2 kernelwith automatic relevance determination [35].
In step 3, we use integrated variance reduction (IVR) as our acquisition function [36] (motivatedin § 3).
IVR calculates the change in total variance if we were to acquire a candidate point x i +
[Equation]
where X i , Y i excludes the candidate point and X i +1 ,
Y i +1 includes it.
Using a Monte Carlo approximation for the intractable integral, the most informative point x ∗ tosample next is
[Equation]
A key difference between standard hyperparameter optimization and our framework is that whenoptimizing, all models except the best performing are discarded, whereas we use all availableinformation to draw more robust conclusions.
To do this, we continue to use our surrogate model.
First, we visualize the multiverse using the surrogate’s posterior predictive mean, and visualize theextent to which we have explored through the surrogate’s posterior variance.
Second, we test forinteraction effects by comparing a GP with a shared kernel M shared against an additive kernel M additive(see fig.
S1).
We compare their fit using the Bayes factor K ,
[Equation]
where P ( X, Y | M ) is the marginal likelihood of the observations given a model.
K > 1 indicatesthat M additive better explains the data than M shared , indicating the absence of an interaction.
This alsoprovides an appropriate termination condition for step 4: we continue to sample until we reach aconclusive Bayes factor.
Finally, we perform a Monte Carlo sensitivity analysis [37, 38] to assesshow much a change in one of the parameters would affect the model outputs.
For GP modeling we use GPy [39] with EmuKit [40] for experimental design and sensitivity analysis.
We use TorchVision’s
[41] off-the-shelf deep learning model architectures.
Motivating example: SVM hyperparameters
Here we motivate the Multiverse Analysis and our focus on exploration with an analysis of hyperparameter optimization.
To both aid replication and assess the effect of hyperparameter tuningon final results, there are calls for increased transparency around search space and tuning method(e.g.
Going one step further, we argue that premature optimization—even if transparentlyreported—hinders our understanding.
We illustrate this with a simple example, tuning two hyperparameters of an SVM classifier to show that optimizing leads to a distorted view of hyperparameterspace.
We compare optimization and exploration using the upper confidence bound (UCB) [43] and
IVR acquisition functions.
In this Multiverse Analysis, we define the search space as the SVM’s regularization coefficient Cand the lengthscale γ of its RBF kernel.
Our evaluation function is the test accuracy of the SVM onthe Wisconsin Breast Cancer Dataset [44] of samples of suspected cancer to be classified as benign ormalignant.
Given the same initial sample, we evaluate 23 further configurations for each acquisitionfunction.
If we optimize, we would conclude from fig.
1a that there is a single region of strong performance.
Conversely, exploring with IVR (fig. 1b) shows that this region is in fact a plateau, and we are freeto use any value of C as long as we scale γ accordingly.
While best test accuracy is similar inboth cases, only by exploring do we learn about the full space of our options.
This knowledge isof vital importance if we properly account for additional real-world objectives, such as minimizingdisparity in outcomes across different social groups.
We test this idea by assigning a synthetic
10 1 10 1 10 3 10 5 10 7 10 9 C
G a mm a 0.
T e s t a cc u r a c y (a) UCB 10 1 10 1 10 3 10 5 10 7 10 9 C
G a mm a 0.
T e s t a cc u r a c y (b)
IVR 10 1 10 1 10 3 10 5 10 7 10 9 C
G a mm a 0.
A b s .
g r o u p a cc u r a c y d i ff e r e n c e (c) IVR majority/minority group label, g ∼ Bern(0 . 4) , to each datapoint, and show in fig.
1c how group-leveloutcomes can vary along the plateau revealed by IVR.
Prematurely optimizing could easily result inselecting a model that introduces group disparity.
There is, of course, an appropriate moment for optimization.
As one moves along spectrum fromresearch to deployment, so too should one move from exploration to exploitation.
Our aim here isnot to critique optimization per se , but to highlight that exploration is also of paramount importance.
When conducting scientific research in particular, we argue it is more appropriate to learn andunderstand as much possible, rather than eke out another marginal improvement.
Here we use the
Multiverse Analysis as a framework for systematic exploration.
Case study 1: When are adaptive optimizers helpful?
Adam [45] is a popular adaptive optimizer for training deep neural networks.
Questioning this practice,
Wilson et al.
[30] suggest adaptive optimizers offer limited advantages over vanilla stochastic gradientdescent (SGD) [46].
They present experiments showing that SGD with momentum [47] outperformsall other optimizers across image recognition on CIFAR-10 [48], language modeling, and constituencyparsing.
Under certain hyperparameter conditions, however, adaptive optimizers can be considered equivalentto SGD with momentum, highlighting the crucial role of hyperparameter tuning [49].
In replications of Wilson et al.’s experiment with VGG [50] on CIFAR-10, additionally tuning Adam’s ϵhyperparameter—introduced solely for numerical stability and typically ignored—eliminates SGD’sadvantage [6, 49].
Both replications [6, 49] can be considered proof by existence.
In order to refute Wilson et al.
’sclaim, it suffices to find any point in hyperparameter space where Adam beats SGD with momentum.
However, this approach tells us little about relative optimizer performance in general .
In this shortcase study, we perform a Multiverse Analysis of Wilson et al.
’s
experiment with VGG on CIFAR-10.
We explore a reasonable and relevant search space and analyze how different hyperparameter choiceslead to different optimizer recommendations.
4.1 Multiverse Definition
Because the claim under scrutiny compares optimizers, we define the evaluation function as thedifference between model test accuracy achieved by Adam and SGD with momentum.
Let g t =∇ f
( θ t − 1 ) be the minibatch-estimated gradient of the loss function f w.r.t.
the model parametersθ t − 1 .
For SGD with momentum, we take a step of size α in the direction of a decaying sum of recentgradients,
[Equation]
[Figure]
D i ff e r e n c
e
i n t e s t a cc u r a c y (a) Mean
[Figure]
Learning rate 1010 - 1010 - 1010 - 1010 -
E p s il o n 0.
V a r i a n c e (b) Variance surface 0.
T r a i n acc u r acy 0.
T es t acc u r acy
AdamSGD (c) Final accuracies where momentum parameter µ controls decay.
In contrast, Adam steps along the gradient normalizedby the an unbiased estimate of its first and second moments ˆ m t
and ˆ v t :
[Equation]
Like Wilson et al.
, we use the default of µ = 0 .
9 and use the Polyak implementation.
For Adam,we also use default parameters β 1 = 0 .
9 and β 2 = 0 .
The model is VGG-16 with batchnormalization [29] and dropout [51], trained for 300 epochs on CIFAR-10.
We set our search space as over learning rate α ∈ [10 − 4 , 10 0 ] by ϵ ∈ [10 − 11 , 10 − 4 ] .
ϵ is only appliedto Adam.
We evaluate 3 batches of 32 points.
4.2 Results
[Figure]
[Figure]
EpsilonLR
[Figure]
The contour plot in fig.
2 shows a large region ( 10 − 3 ≤ α ≤ 10 12 ) inwhich SGD outperforms Adam.
However, we also identify regions(approx.
α < 10 − 3 or α > 10 12 ) where the opposite is true, thoughwe note the higher uncertainty (fig. 2b) in the high learning rate region.
See fig.
S2 for raw results.
The change in relative performance is described almost entirely bylearning rate.
The sensitivity analysis (fig. 3) reveals main effects(i.e.
sensitivity to each variable in isolation) for learning rate and ϵ of0 .
990 ± 0 . 014 and − 0 .
004 ± 0 . 0002 , and similarly total effects (i.e.sensitivity to each variable including its interaction with others) of 0 .
996 ± 0 . 014 and 0 .
008 ± 0 .
Testing for interaction effects via Bayes factor (see eq.
, we find K = 2635 , thus rejecting theexistence of an interaction between learning rate and ϵ .
In summary, we find that SGD with momentum often, but not always, outperforms Adam whentraining VGG on CIFAR-10, and that the conclusion we draw is highly sensitive to selected learningrate.
Unlike others [6, 49], we do not find a significant effect of ϵ , instead finding that learning ratedetermines which optimizer performs best.
We do not, however, dispute that ϵ could be tuned tofurther improve Adam’s performance, only that varying ϵ does not lead to different conclusions aboutthe relative merits of Adam versus SGD.
In practice, our results suggest optimizer choice may make little difference to final test error, as longas an appropriate learning rate is used.
If budget for hyperparameter search is limited, ϵ is unlikely tobe the most efficient hyperparameter to include in a search.
Using the Multiverse Analysis, we havesystematically explored how decisions about learning rate and epsilon impact conclusions about theoptimal optimizer.
In doing so, we have shown that reported results may vary according to researcherchoices.
Case study 2: Is there a large-batch generalization gap?
Larger batch sizes are a prerequisite for distributed neural network training.
However, many researchers have reported a “generalization gap”, where generalization performance declines as batchsize increases [31, 52, 53], a phenomenon potentially [54–57] but not conclusively [58] linked to thesharpness of the resulting minima.
Subsequent work has sought ways to mitigate the generalization gap by, in broad strokes, eitherscaling up the batch size throughout training [59, 60, pp. 262-265], or by scaling learning rateproportional to batch size [61–64].
To make sense of these results, one must cut through myriadresearcher choices about datasets; model architectures; termination criteria [65]; linear [63] orsublinear (e.g. square root) [61] scaling rules; high initial learning rate [61]; learning rate decay orwarmup [63]; layer-specific learning rates [64]; batch norm variants [61]; regularization techniquese.g.
label smoothing [65]; and so on.
In our second case study, we use a Multiverse Analysis tosynthesize existing research on the relationship between batch size, learning rate and generalizationerror.
5.1 Multiverse Definition
Our evaluation function is the test accuracy of a model trained according to sampled configuration.
The search space includes learning rate α ∈ [10 − 4 , 10 − 12 ] , batch size ∈ { 2 4 , . .
. , 2 13 } ,model ∈ { AlexNet [66], VGG [50], ResNet [67] } , and dataset ∈ { CIFAR-10, CIFAR-100 [48], Tiny
ImageNet[68] } .
Specifically, we use VGG-16 with batch norm and ResNet-18.
Tiny ImageNet wasselected as a substitute for ImageNet [69] to limit compute expenditure.
For categorical parameters, we treat each pair of model and dataset as its own function with anintrinsic coregionalization model [70, 71].
Given base kernel k , our multi-output kernel matrix is:
[Equation]
where B d and B m represent dataset and model respectively.
In interpreting these parameters, theouter product of w defines how related each output is, whereas κ allows them to vary independently.
We evaluate 6 batches of 32 points.
Before analysis, we discard all model runs with trainingaccuracy < 0 .
99 and re-fit the model.
In addition to previous methods we also analyze the coregionalization parameters to investigate how much dataset and model impact our results.
5.2 Results
Of 224 trials, 9 failed to converge and a further 115 did not reach training accuracy ≥ 0 . 99 .
Apreliminary analysis of the raw results (see fig.
S3) shows test accuracy is barely positively correlatedwith learning rate ( ρ = 0 . 08 ) and slightly negatively correlated with batch size ( ρ = − 0 . 2 ).
In fig.
4,however, a consistent plateau emerges across all models on both CIFAR-10 and CIFAR-100: testaccuracy is maintained as long as batch size scales up with learning rate.
On both sides of this plateau,if either learning rate or batch size are too large, generalization error will increase.
[Figure]
black crosses were excluded.
Overlayed translucent regions indicate high training error.
For Tiny ImageNet see fig.
S4; for variance see fig.
S5.
The discrepancy between contours and data points in (a) is due to the coregionalized model sharing information across functions.
We were unable to successfully train any model on Tiny ImageNet.
Of the 42 models we trained on
Tiny ImageNet, 22 reached the 0 .
99 training accuracy threshold, but none of these obtained a testaccuracy higher than 0.01, indicating significant overfitting.
See fig.
S4 for Tiny ImageNet contours.
Testing for interaction effects via Bayes factor (see eq.
, we find K = 0 .
58 , suggesting thatan additive kernel is not sufficient to explain the data and indicating an interaction effect.
Thisimplies that batch size alone does not explain generalization, but batch size and learning rate together .
Our modeling approach gracefully handles the failures on Tiny ImageNet, such that even with Tiny
ImageNet trials removed, the interaction effect remains present ( K = 0 . 34 ).
The coregionalization kernels explain how model and dataset affect our results.
In fig.
5a all modeloutputs are highly correlated, indicating that choice of model does not impact the relationshipbetween batch size, learning rate, and generalization error.
In contrast fig.
5b shows highly correlatedoutputs across CIFAR-10 and CIFAR-100, and moderate negative correlation with Tiny ImageNet.
Astrength of our chosen coregionalized model is that it flexibly captures the batch size and learning raterelationship in CIFAR-10 and 100, despite the failures on Tiny ImageNet.
Our results on CIFAR-10and CIFAR-100 indicate that the relationship may hold across datasets where training was successful ,including across different dataset complexities, in contrast to previous work [52].
Finally, our sensitivity analysis in fig.
6 reveals next to no main effect for either learning rate( 0 .
03 ± 0 .
06 ) or batch size ( 0 .
02 ± 0 . 06 ).
However, the larger total effects for both learning rate( 0 .
19 ± 0 . 11 ) and batch size ( 0 .
15 ± 0 .
11 ) again support the existence of an interaction effect.
Bothlearning rate and batch size are dominated by the main effect of dataset ( 0 . 83 ± 0 . 03 ), though this islikely due to Tiny ImageNet.
In conclusion, our framework reveals a complex interaction between learning rate, batch size, andgeneralization error that is consistent across all models and two out of three datasets.
These results support scaling learning rate linearly with batch size [56, 62, 63], but do not support a thresholdbatch size (up to 2 13 ) after which learning rate is no longer corrective [52, 65].
After accountingfor learning rate, we find no consistent evidence of the proposed [31, 53] large-batch generalizationgap.
Here, the Multiverse Analysis has proven a useful tool to pull together disparate research on thegeneralization gap and to surface underlying trends.
Discussion
C
I F
A R
[Figure]
C
I F
A R
[Figure]
T i n y
[Figure]
m a g e N e
[Figure] [Figure]
In the three applications of the Multiverse Analysis presentedhere, we have demonstrated that decisions taken by the researcher can have a significant effect on the final conclusionsdrawn.
In our SVM hyperparameter example, we highlight theimportant role of systematic exploration and show that prematureoptimization may both limit our understanding and present onlya partial truth to downstream practitioners.
Our first case studyon optimization demonstrated how varying the learning rate results in fundamentally different conclusions about whether touse SGD with momentum or Adam for optimization.
Conversely,our second case study showed a simple relationship betweenbatch size, learning rate and generalization error irrespective ofmodel and dataset.
6.1 Choice of search space
Across these examples we have tried to illustrate differentlycomplex analyses by way of differently sized search spaces.
Incase study 1 we inflated a rather small multiverse focusing onlyon directly relevant hyperparameters, though in case study 2we built a larger multiverse including parameters (model anddataset) that vary freely between existing research.
An expectedcritique of the Multiverse Analysis is that choice of search spaceis itself a choice.
However, choosing dimensions and bounds fora search space is more principled than choosing specific points,and that declaring each dimension makes explicit assumptionsof relevance or irrelevance.
That said, it is always possible forsubsequent work to critique the choice of search space and toadd new dimensions.
Indeed, our first case study should beexpanded to include each of Wilson et al.
’s
experiments with different model architectures, datasets, deep learning frameworksand additional adaptive optimizer variants, though we reinforcethat we chose a limited analysis to provide a simple case study.
Our second case study also presents a number of interesting avenues for expansion in subsequent analyses, including additionaldatasets and model architectures, learning rate schedules andoptimizers.
Most interesting would be inclusion of terminationcriterion and evaluation metric, both previously highlighted askey drivers of divergent findings [65].
6.2 Compute cost
Our main contribution is to present the Multiverse Analysis framework and show how it can be used todraw robust conclusions about model performance, so we allocated our compute budget to showcaseillustrative examples.
Our framework is equally appropriate for research pushing state-of-the-artwith a larger budget, though such experiments aren’t necessary to demonstrate our framework’s value.
In this project, we used 1138 hours of GPU time, at a rough cost of $775 with 140 kg of
CO 2 .
We expect costs to scale with multiverse size, requiring a pause for consideration in light ofrecent critiques of the environmental impacts of ML [72–74].
In response, we first note that at leastsome of the required compute is already taking place as part of existing trial-and-error workflows.
Second, through the introduction of a surrogate model for the multiverse, one can substantially reducethe amount of exploration required.
Finally, we suggest that all resources consumed in producingnon-robust results, and those of subsequent work that builds upon them, are wasted by definition.
6.3 Future work
It is common practice when reporting model performance to train a number of runs with differentrandom seeds and report the mean and standard deviation.
Number of runs varies according toresearcher budget and time.
Appropriate sample size to enable robust inference in the face of noise israrely considered.
Our approach to the Multiverse Analysis presents an elegant potential solutionto this issue.
In our examples thus far we train only a single model for each point and assume forsimplicity homoscedasticity.
However, research in experimental design suggests that accounting forheteroscedasticity by separately modeling the variance in the surrogate [75, 76] could allow for aprincipled trade-off between conducting another run and a sampling a new configuration [77].
While we introduce the Multiverse Analysis to ML, it has previously been applied to a handfulof studies in human computer interaction [78].
In an exciting area for development, the authorsalso develop interactive visualizations to help the reader explore the effects each choice, up to andincluding re-rendering written conclusions in an online version of the paper.
Given that the largestmultiverse in our work is 4-dimensional, our faceted contour plots are sufficient to communicate keytrends.
However, in more expansive multiverses novel visualization techniques will become essential.
Recent experiments with pre-registration in ML (e.g. [79]) promise to enforce a distinction betweenexploratory and confirmatory research [23, 80].
Committing to an experimentation and analysisplan in advance can be a helpful foil for questionable practices such as tweaking the parametersuntil one reaches a positive result.
However, this commitment is limited to a single analysis—oneparticular instantiation of a decision set—while the other possible analyses, the remainder of themultiverse, remain unexplored.
We see the Multiverse Analysis as complementary to pre-registration,in that pre-registering a Multiverse Analysis both codifies auxiliary hypotheses and allows room forexploration.
6.4 Conclusion
For continued progress in ML, we depend on reproducible results and conclusions that generalize tonew settings.
We have introduced the Multiverse Analysis as a principled framework for exploringthe impact of researcher decisions and facilitate the drawing of more robust conclusions.
Ourfirst case study unifies conflicting work on the benefit of adaptive optimizers and reinforces thatoptimizer merit is driven primarily by learning rate.
In our second case study we identified a complexinteraction between batch size, learning rate and generalization error, and we dispute the existence ofa generalization gap.
We have also presented evidence supporting the practice of scaling learning ratelinearly with batch size.
By using the Multiverse Analysis, both researchers and practitioners gain more robust claims,better understanding of how decisions impact results, and helpful insight into the generality andreproducibility of conclusions.
Checklist
For all authors...
(a
main claims made abstract and accurately reflect the paper’scontributions and
(b) Did
describe
the
limitations of work
Yes
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See § 6.
for compute cost/climate impact discussion.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] To be included
in supplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See § 6.2.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] All datasets andmodels cited throughout.
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No]
If you used crowdsourcing or conducted research with
human
subjects
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Appendix
3 X       k ( X , [[ 0 , 0 ]])      3 X       k
( X , [[ 0 , 0 ]]) (a) Shared      3 X       k
( X , [[ 0 , 0 ]])      3 X       k
( X , [[ 0 , 0 ]])
[Figure] [Figure]
[Table]
− 4 − 3 − 2 − 1 Learning rate (log ₁₀ ) 0.
[Table]
T es t acc u r acy
[Figure] [Figure]
₂ ,) 0.
[Figure]
T es t acc u r acy
[Figure]
− 4 − 3 − 2 − 1 Learning rate (log ₁₀ ) 0.
T es t acc u r acy
[Figure]
T es t acc u r acy
[Figure] [Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure]