Maximum Class Separation as Inductive Bias in One Matrix
Abstract
Maximizing the separation between classes constitutes a well-known inductive biasin machine learning and a pillar of many traditional algorithms.
By default, deepnetworks are not equipped with this inductive bias and therefore many alternativesolutions have been proposed through differential optimization.
Current approachestend to optimize classification and separation jointly: aligning inputs with classvectors and separating class vectors angularly.
This paper proposes a simplealternative: encoding maximum separation as an inductive bias in the network byadding one fixed matrix multiplication before computing the softmax activations.
The main observation behind our approach is that separation does not requireoptimization but can be solved in closed-form prior to training and plugged intoa network.
We outline a recursive approach to obtain the matrix consisting ofmaximally separable vectors for any number of classes, which can be added withnegligible engineering effort and computational overhead.
Despite its simplenature, this one matrix multiplication provides real impact.
We show that ourproposal directly boosts classification, long-tailed recognition, out-of-distributiondetection, and open-set recognition, from CIFAR to ImageNet.
We find empiricallythat maximum separation works best as a fixed bias; making the matrix learnableadds nothing to the performance.
The closed-form implementation and code toreproduce the experiments are provided in the supplementary materials.
Introduction
An inductive bias of a learning algorithm describes a set of assumptions about the target functionindependent of training data [36].
Inductive biases play a vital role in the design of machine learningalgorithms, consider for example inductive biases for image structures ( e.g., the convolution operator),symmetries ( e.g., rotational equivariance) or relational structures ( e.g., graph layers).
Specifically forcategorization, an important and long-standing inductive bias is optimal class separation.
Perhapsthe most well-known example of the use of this bias is Support Vector Machines, which maximizethe margin of the hyperplane between samples of two classes [6].
Given many possible hyperplanes,the rationale is to select the one that represents the largest margin, i.e., separation, known as amaximum-margin classifier.
Similarly, boosting algorithms give weights to samples depending ontheir margin, such as AdaBoost which focuses on low margin samples [13].
In such algorithms andin many more, separation has been incorporated in their design to pull apart the classes, therebyimproving generalization in classification settings to unseen data.
In deep learning-based classification, neural networks are by default not equipped with a maximumseparation inductive bias.
The cross-entropy loss with a softmax function is a common choice to traina classifier, aiming for discriminative power.
Yet this framework does not explicitly drive maximumseparation between the classes.
Liu et al.
[30] observed that networks by themselves try to decoupleinter-class differences based on angles and intra-class variations based on norms.
This decoupling is however limited out-of-the-box and a wide range of works have investigated optimization strategiesto explicitly force classes away from each other.
A common approach is to introduce additional losseswhich incorporate a notion of class margins, see e.g., [24, 28, 32, 33] or by including orthogonalityconstraints [24].
Specifically, hyperspherical uniformity has shown to be an effective way to separateclasses, whether it be as additional objectives in a softmax cross-entropy optimization [25, 28, 29, 67]or through hyperspherical prototypes [35].
This paper provides an alternative view to embed separation in deep networks with a straightforwardsolution: encoding maximum separation as an inductive bias in the network architecture itself withone fixed matrix.
Rather than treating separation as an optimization problem, we find that the problemcan be optimally solved in closed-form.
The closed-form solution is given by a recursive algorithmwhich embeds all class vectors uniformly, such that all are equiangular and with zero mean.
Thesolution holds for any finite number of classes and results in a single matrix that can be added apriori as a matrix multiplication on top of any network.
The outputs of the proposed embedding arethe inner product scores that feed into the softmax function, allowing us to still leverage powerfulobjectives such as cross-entropy on softmax activations.
We demonstrate that remarkably, adding only one fixed matrix in the neural network providescompelling improvements across various classification tasks.
We show that enriching ResNet witha maximum separation inductive bias improves standard classification and especially long-tailedrecognition.
We also show that our proposal improves methods specifically designed for long-tailedrecognition, highlighting its general utility.
The results generalize to ImageNet and improves bothshallow and deep networks.
The added inductive bias helps to balance classes by design, avoiding theunder-representation of rare classes.
Beyond classification, we illustrate that out-of-distribution andopen-set recognition benefit from an embedded maximum separation.
Our general-purpose solutioncomes with no measurable difference in runtime and can be added in one line of code, enabling itto be adopted generally.
To highlight the broad applicability of our proposal, we also show howtask-specific state-of-the-art approaches for open-set recognition benefit directly when adding theproposed maximum separation.
All code is provided in the supplementary materials.
Closed-Form Maximum Separation between Classes
Classification constitutes a supervised setting, where we are given a training set of N examples{ ( x i , y i ) } Ni =1 , where x i denotes input data at index i and y i ∈ { 1 , . . .
, C } denotes the correspondinglabel for one of C classes.
Conventionally, this problem can be addressed by feeding inputs through anetwork, i.e., o i =
Φ( x i ) ∈ R C .
The outputs denote class logits, which serve as input for a softmaxactivation followed by a cross-entropy loss.
Our goal is to embed maximum separation as an inductive bias into any network, so that the networkno longer needs to learn this behaviour.
We do so by adding a single matrix multiplication ontop of the output of network Φ ′ ( · ) .
The matrix itself should contain all necessary informationabout maximum separation between classes.
While the general problem of optimally separatingarbitrary numbers of classes on arbitrary numbers of dimensions is an open problem [28, 35], wenote that we only need maximum separation of C classes in a particular output space.
To that end, letˆ x i =
Φ ′ ( x i ) ∈
R C − 1 now denote the output of a network to a ( C − 1) -dimensional output space.
Weadd the matrix multiplication to obtain C -dimensional class logits: o i =
P T
ˆ x i with P ∈ R
( C − 1) × C .
The matrix P denotes a set of C vectors, one for each class, each of ( C − 1) dimensions.
This matrixis fixed such that the class vectors are separated both uniformly and maximally with respect to theirpair-wise angles, which we in turn embed on top of a network.
Below, we outline the definition andimplementation of class vectors on a hypersphere with maximum separation.
In what follows, thenumber of classes is denoted by k + 1 = C for notational convenience.
Moreover, this definition isrestricted to matrices of the form R k × ( k +1) .
Definition 1.
(Maximally separated matrix)
For k + 1 classes, let P k
=
[ p 0 , . .
, p k ] denote amatrix of k + 1 column vectors { p i } k i =
⊂ S k − 1 , such that ∀ i,j,k,i ̸
= j,j ̸ = k ⟨ p i , p j ⟩ = ⟨ p i , p k ⟩ and 86 (cid:80) [-k i =
-]ki =0 p i = 0 .
Here, ⟨· , ·⟩ denotes the cosine similarity.
Definition 1 ensures that all class vectors are as far awayfrom each other as possible based on two requirements.
First, the angle between any two class vectorsshould be the same.
Equal pair-wise similarity entails uniformity as any deviation from this setup [Figure] leads to classes that move closer to each other [47].
Second, the mean over all class vectors shouldbe the origin.
This constraint prevents trivial solutions such as an identical embedding of all classesas well as suboptimal embeddings in terms of separation such as one-hot encodings.
In short, thedefinition prescribes that k + 1 classes can be optimally separated specifically when using k outputdimensions.
To be able to construct such an embedding, we first need to know what the pair-wiseangular similarity is for any fixed number of classes:
Lemma 1.
For maximally separated matrix P k , ∀ i,j,i ̸
= j ⟨ p i , p j ⟩ = − 1 k .
Proof.
With P k maximally separated, we have (cid:80) ki =
p i = {+
+}
and hence:0 = ⟨ p 0 , k (cid:88) i
= p i ⟩ = 1 + k (cid:88) i = ⟨ p 0 , p i ⟩ = 1 + k ⟨ p 0 , p 1 ⟩ ,
[Equation]
since all class vectors are at equal angle to each other.
We have ⟨ p 0 , p 1 ⟩ = − 1 k and again by extensionof equidistance: ∀ i,j,i ̸
= j ⟨ p i , p j ⟩ = − 1 k .
The implication of the lemma above is that an optimal separation of k + 1 classes on the manifold S k − 1 results in classes being separated beyond orthogonality, a direct result of using the fullspace and different from classical one-hot encodings, which only span the positive sub-space andtherefore provide only orthogonal separation.
The outcome of the lemma allows us to construct P krecursively [1].
For any finite number of classes k + 1 , the closed-form solution for P k is given as:
[Equation]
The closed-form solution for two classes ( P 1 ) is trivially given as +1 and -1 on the line.
For morethan 2 classes, k ≥ 2 , we start from the solution given for k − 1 on S k − 2 .
We add the vector forthe new class as p 1 = (1 , 0 , . . . , 0) .
We place the solution P k − 1 at ( − 1 k , 0 , . . . , 0) , in the hyperplaneperpendicular to p k , where P k − 1 is shrunk so that its radius is (cid:113) 1 − 1 k 2 .
In this manner, all points in
P k are of radius 1 and uniformly distributed, as proven below.
Figure 1 shows the construction of ourembeddings for 3 and 4 classes.
Theorem 1.
For any k ≥ 1 , P k is a maximally separated matrix.
Proof.
We proceed by induction.
Note that, for k = 1 , the angular similarity of the two vectors isclearly given by − 1 k = − 1 .
Now, assume the theorem holds for k − 1 ∈ N and let ( P n ) ∗ i denote thei -th column vector of P n for arbitrary n ∈ N .
Then, for k and i ̸
= j , we obtain the case where eitheri or j is 1 and the case where neither is 1 .
For the first case, without loss of generality assume that thei -th element is the first, so that:
[Equation]
For the second case, we derive:
[Equation]
since (cid:10) ( P k − 1 ) ∗ i , ( P k − 1 ) ∗ j (cid:11) =
− 1 k − 1 by assumption.
Therefore, the vectors of P k have a pair-wiseangular similarity of − 1 k .
To prove that P k has zero mean, we again proceed by induction.
Note thatfor P 1 , the statement obviously holds.
Now, assume the statement holds for k − 1 ∈ N , then
[Equation]
since (cid:80) k − 1 i =
( P k − 1 ) ∗ i = 0 by assumption.
Input
Backbone
Output x
P
For C = k + 1 classes, the solution P k consists of
C vectors of unit length.
The main idea of this workis to utilize P k to obtain class logits under a fixedmaximum separation, see Figure 2.
For sample x i ,we want its network output ˆ x i =
Φ ′ ( x i ) to align withthe vector of its corresponding class.
Given P k , thealignment is maximized if the network output pointsin the same direction as the class vector.
This can beobtained by computing the dot product between both.
Since we need to compute the logits for all classvectors in the final softmax activation, we simplyaugment the network output with a single matrix multiplication: o i =
P T
ˆ x i .
By enforcing maximumseparation from the start, a network no longer needs to learn this behaviour, essentially simplifyingthe learning objective to maximizing the alignment between samples and their fixed class vectors.
We note that the norm of the class vectors is not taken into account during their construction as weoperate on the hypersphere.
We can optionally include a radius hyperparameter which controls thenorm of the class vectors, i.e., the network output is given as: o i = ρP T
ˆ x i with ρ as hyperparameter.
In practice however, the radius of the class vectors is of little influence to the task performance andwe set it to 1 unless specified otherwise.
Experiments
Implementation details.
Across our experiments, we train a range of network architectures including
AlexNet [21], multiple ResNet architectures [16, 64], and VGG32 [44], as provided by PyTorch [37].
CIFARCIFAR- - 0.
ConvNet 56.
+ This paper 57.
+0. +0.
+0. +0.
+1.
+0. +0.
+1.
+3.
+4.
ResNet75.
+ This paper 76.
+0. +0.
+1.
+2.
+3.
+0.
+3.
+5.
+8.
+12.
All networks are optimized with SGD with a cosine annealing learning rate scheduler, initial learningrate of 0.1, momentum 0.9, and weight decay 5e-4.
For AlexNet we use a step learning rate decayscheduler.
All hyperparameters and experimental details are also available in the provided code.
3.1 Classification and long-tailed recognition
Classification with maximum separation.
In the first set of experiments, we evaluate the potentialof maximum separation as inductive bias in classification and long-tailed recognition settings usingthe CIFAR-100 and CIFAR-10 datasets.
Our maximum separation is expected to improve learningespecially when dealing with under-represented classes which will be separated by design with ourproposal.
We evaluate on a standard ConvNet and a ResNet-32 architecture with four imbalancefactors: 0.2, 0.1, 0.02, and 0.01.
We set ρ = 0 .
1 for ResNet-32 as this provides a minor improvementover ρ = 1 .
The results are shown in Table 1.
For an AlexNet-style ConvNet, classification accuracyimproves in all settings, especially when imbalances is highest.
With a more expressive network suchas a ResNet, we find that embedding maximum separation provides a bigger boost in performance.
The higher the imbalance, the bigger the improvements; on CIFAR-100 the accuracy improves from35.02 to 38.85, on CIFAR-10 from 56.98 to 69.70, an improvement of 12.72 percent point.
Forfurther analysis, we have investigated the relation between accuracy improvement and train samplefrequency.
In Figure 3 we show the results over the imbalance factors, highlighting that our proposalimproves all classes as imbalance increases, especially classes with lower sample frequencies.
Should maximum separation be a fixed inductive bias?
Separation in this work takes the formof a fixed matrix multiplication that can be plugged on top of any network.
But is the informationin this matrix as inductive bias optimal for classification?
To test this hypothesis, we compare theprevious results on CIFAR-100 and CIFAR-10 and their long-tailed variants to two baselines.
Thefirst baseline makes the matrix P learnable and our proposal acts as an initialization of the learnablematrix.
Intuitively, if the results improve by making the matrix learnable instead of fixed, there is moreinformation that is not captured in the closed-form solution.
The second baseline adds a learnablefully-connected layer with standard random initialization instead of a fixed matrix multiplication, toinvestigate whether our obtained improvements are not simply a result of extra computational efforts.
The results are shown in Figure 4.
On both datasets across all imbalance factors, embedding maximumseparation as a fixed constraint in a network works best.
Making the matrix learnable actually
[Figure]
None 0.
Imbalance factor
A cc u r a c y (a) CIFAR-100.
None 0.
Imbalance factor
A cc u r a c y
Fixed (This paper)
Learnable (This paper)
Learnable (Random) (b) CIFAR-10.
decreases the performance, indicating that further optimization discards important information andhighlighting that separation is an inductive bias we should not steer away from.
Simply adding anextra learnable layer with standard initialization is not effective, which indicates that improvementsare not guaranteed when adding more fully-connected layers.
We conclude that maximum separationis best added in a rigid manner as strict constraint for networks to adapt to.
Enriching long-tailed recognition approaches.
So far, we have shown that conventional networkarchitectures are better off with maximum separation as inductive bias for classification and longtailed recognition.
In the third experiment, we investigate the potential of adding our proposal on topof methods that are specifically designed for long-tailed recognition.
We perform experiments ontwo methods, LDAM [8] and MiSLAS [66].
LDAM addresses learning on imbalanced data througha label-distribution-aware margin loss which replaces the conventional cross-entropy loss.
We caneasily augment LDAM by placing the matrix on top of the used backbone.
Their hyperparameterµ in LDAM is set to 0.5 when using their method as is and set to 0.4 when augmented with ourproposal as this maximizes the scores for both settings.
MiSLAS is a recent state-of-the-art approachfor long-tailed recognition that improves two-stage methods by means of label-aware smoothing andshifted batch normalization.
We again plug in our approach in the provided code simply as a fixedmatrix multiplication on top of the used backbone.
In Table 2, we report the results on CIFAR-100 using a ResNet-32 backbone across three imbalancefactors.
For LDAM, we find that both for the SGD and the DRW variants, adding maximum separationon top improves the accuracy.
For imbalance factor 0.01 for LDAM, we improve the accuracy from39.87 to 42.02 with the SGD variant and from 42.37 to 43.19 for the DRW variant.
The recent
MiSLAS approach to long-tailed recognition also benefits from maximum separation after both stages.
The most competitive results after their stage 2 are improved by 1.59, 0.83, and 0.42 percent point forrespectively imbalance factors 0.1, 0.02, and 0.01.
The long-tailed baselines are specifically designedto address class imbalance but additionally benefit from maximum separation across the board.
Weconclude that maximum separation as a fixed inductive bias complements task-specific approaches,highlighting its general applicability and usefulness.
ImageNet experiments.
To highlight that our proposal is also beneficial with many classes anddeeper networks, we provide results on ImageNet for two ResNet backbones in Table 3.
With a
ResNet-50 backbone, we obtain improvements of 1.6 percent point on ImageNet and 3.5 percentpoint on its long-tailed variant in top 1 accuracy.
With a deeper ResNet-152, the improvements are0.6 and 1.4 percent points for the standard and long-tailed benchmarks.
We also find that the top 5accuracy improves from 92.4 to 94.9 with ResNet-50 and from 94.3 to 95.1 with ResNet-152.
Weconclude that the proposed inductive bias in matrix form is also useful for larger-scale classification.
Imbalance factor 0.
LDAM-SGD 55.
+ This paper 57.
+2.
+1.
+2.
LDAM-DRW 57.
+ This paper 58.
+0. +0.
+0.
Imbalance factor 0.
MiSLAS (stage 1) 58.
+ This paper 59.
[Table] [Table]
+ [Table]
MiSLAS (stage 2) 61.
+ This paper 63.
+1.
[Table]
[Table] [Table]
Imagenet
[Table]
-tailed ImageNet variant.
3.2 Out-of-distribution detection and open-set recognition
We also investigate the potential of our proposal outside the closed set of known classes.
To that end,we perform additional experiments on out-of-distribution detection and open-set recognition.
Out-of-distribution with maximum separation.
Intuitively, maximum separation and uniformitybetween classes allows for more opportunities for samples outside the distribution of known classesto fall in between the spaces of class vectors, especially since all classes are positioned beyondorthogonal to each other as per Lemma 1.
We evaluate this intuition first on out-of-distributiondetection.
Following Liu et.al.
[27], we use CIFAR-100 as in-distribution set and experiment with
SVHN and Placed 365 as out-of-distribution sets.
Using the same experiment settings reported in theirpaper, we train on a WideResNet and compare the out-of-distribution performance on the commonmetrics FPR95, AUROC, and AUPR.
We determine whether a sample is in- or out-of-distributionusing three scoring functions.
The first directly uses the maximum softmax probability to distinguishin- and out-of-distribution samples [18].
The second is the energy score as introduced in Liu etal.
[27], which classifies an input as out-of-distribution if the negative energy score is smaller than athreshold value,
The third is the confidence score based on Mahalanobis distance from Lee et al.
The results are shown in Table 4.
With SVHN as out-of-distribution set, embedding a maximumseparation inductive bias improves all metrics across all out-of-distribution detection methods.
Especially the energy score benefits from maximum separation with an improvement of 6.90 in FPR95,11.48 in AUROC, and 3.01 in AUPR.
On Places 365, the results are overall closer.
Maximumseparation improves on the AUROC and AUPR metrics but not the FPR95 metric.
We observe thatadding maximum separation performs best for near out-of-distribution data.
In Figure 5 we providefurther analyses on the effect of maximum separation for out-of-distribution detection with energyscores, highlighting that separation increases the gap in energy scores between both sets, making iteasier to discriminate both.
We conclude that a maximum separation inductive bias is valuable forout-of-distribution detection, especially for near out-of-distribution data.
Open-set recognition with maximum separation. We also investigate the potential of maximumseparation for open-set recognition, which can be viewed as a generalization of out-of-distribution
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
as out-of-distribution dataset for CIFAR-100 as in-distribution dataset, we find that embedding maximum separation increases the gap in energy scores between both datasets, making it easier to differentiate between both.
With Places 365 as out-of-distribution dataset, the energy scores are affected less, resulting in smaller improvements with maximum separation.
detection [34].
To take our proposal to the ultimate test, we start from the recent state-of-the-art workof Vaze et al.
We experiment with adding our proposal to two approaches outlined in their work.
The first applies various training tricks and hyperparameter tuning to improve the closed-set classifier,followed by open-set recognition using the maximum softmax probability (MSP+) score.
The secondperforms the same training with the Maximum Logits Score (MLS) before the softmax activations todifferentiate closed- from open-set examples.
All experiments are run on author-provided code onfour benchmarks: SVHN, CIFAR10, CIFAR + 10, and CIFAR + 50, where we use the 5 splits andreport the average scores as outlined by Vaze et al.
The results of the open-set experiments are shown in Table 5 with the AUROC metric.
For both the
MSP+ and the state-of-the-art MLS scores, maximum separation is a positive addition.
This resultfollows the intuition of the original approach that improving the closed-set classifier – here by meansof the extra inductive bias – is also beneficial for open-set recognition.
Related work
In deep learning literature, the principle of separation has been investigated from several perspectives.
Separation can be achieved by enforcing or regularizing orthogonality.
For example, Li et al.
[24]propose to regularise class vectors orthogonal to each other.
Orthogonality is also often used asregularisation for intermediate network layers, see e.g., [3, 39, 40, 56, 61, 62].
Orthogonal constraintsare fixed and easy to obtain by design.
For classification specifically however, orthogonality does notresult in a maximal separation between classes, as only the positive sub-space is utilized. We strivefor a solution that makes use of the entire output space.
SVHN
[Table]
+ This paper
[Table]
+ This paper
[Table]
Beyond orthogonality, multiple works improve class separation by promoting angular diversitybetween classes.
A common approach to do so is by incorporating class margin losses [11, 31, 33, 55,65].
For example, the generalized large-margin softmax loss enforces both intra-class compactnessand inter-class separation [32].
These works are in line with the observations of Liu et al.
[30] thatdeep networks have a natural tendency to break classification down into maximizing class separationand minimizing intra-class variance.
In particular, separation has in recent years been successfullytackled by adopting the perspective of hyperspherical uniformity.
Such uniformity can for examplebe optimized in deep networks by minimizing the hyperspherical energy between class vectors [25,28, 29, 15].
Recently, Zhou et al.
[67] extend this line of work by learning towards the largestmargin with a zero-centroid regularization.
Hyperspherical uniformity can also be approximated byminimizing the maximum cosine similarity between pairs of class vectors [14, 35, 58].
In this work,we also start from hyperspherical uniformity and deviate from current literature on one importantaxis: hyperspherical uniformity does not require optimization.
We show and prove that maximumseparation has a closed-form solution and can be added as a fixed matrix to any network.
Havingmaximum separation as a fixed inductive bias does not require additional hyperparameters and comeswith minimal engineering effort, lowering the barrier towards broad integration in the field.
Our work fits into a broader tendency of incorporating inductive biases into deep learning, for exampleby incorporating rotational symmetries [10, 12, 23, 43, 46, 48, 51, 54, 60], scale equivariance [45, 59],gauge equivariance [9, 17] graph structure [4, 7, 19, 53], physical inductive biases [2, 20, 38, 41],symmetries between reinforcement learning agents [5, 26, 50, 57], and visual inductive biases [42,49, 63].
Embedding maximum separation as inductive bias is complementary to the listed works.
Conclusions
This paper strives to embed a well-known inductive bias into deep networks: separate classesmaximally.
Separation is not an optimization problem and can be solved optimally in closed-form.
We outline a recursive algorithm to compute maximum class separation and add it as a fixed matrixmultiplication on top of any network.
In this manner, maximum separation can be embedded withnegligible effort and computational complexity.
To showcase that our proposal is an effective buildingblock for deep networks, we perform various experiments on classification, long-tailed recognition,out-of-distribution detection, and open-set recognition.
We find that our solution to maximumseparation as inductive bias (i) improves classification, especially when classes are imbalanced, (ii)is best treated as a fixed matrix, (iii) improves standard networks and task-specific state-of-the-artalgorithms, and (iv) helps to detect samples from outside the training distribution.
With separationdecoupled and solved prior to training, a network only needs to optimize the alignment betweensamples and their fixed class vectors.
We conclude that our one fixed matrix is an effective, broadlyapplicable, and easy-to-use addition to classification in networks.
Our approach is currently focusedon multi-class settings only, where exactly one label needs to be assigned to each example. Maximumseparation does also not naturally generalize to zero-shot settings, as such settings require that classesare represented by semantic vectors that point in similar directions based on semantic similarities. While we focus on the classification supervised setting, maximum separation has potential forunsupervised learning as well as self-supervised learning paradigm. As the matrix imposes geometricstructure in the output space it could potentially help in improving the uniformity of classes.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)? [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
hourly
participant compensation