SelecMix: Debiased Learning by Contradicting-pair Sampling
Abstract
Neural networks trained with ERM (empirical risk minimization) sometimes learnunintended decision rules, in particular when their training data is biased, i.e.,when training labels are correlated with undesirable features.
Techniques havebeen proposed to prevent a network from learning such features, using the heuristicthat spurious correlations are “simple” and learned preferentially during training. Recent methods augment training data such that samples displaying spuriouscorrelations (i.e., bias-aligned samples) become a minority, whereas the other,bias-conﬂicting samples become prevalent.
However, they require sophisticatedtechniques such as disentanglement with careful tuning, making them challengingto train the model and scale to complex real-world datasets.
In this work, we usethe mixup, which builds convex combinations of input images and their labels,to augment the bias-conﬂicting samples.
Mainly, we propose a selective mixupscheme, SelecMix, where the mixup is applied to the pairs having (i) the same labelbut dissimilar biased features, and (ii) different label but similar biased features. To compare samples with respect to the biased features, we propose the biasampliﬁed contrastive model relying on the heuristic that biased features are learnedpreferentially during training.
Experimental results demonstrate the effectivenessof the proposed method on both the synthetic and real-world datasets.
Furthermore,we validate the robustness of our method under noisy labels, which is more realistic,and challenging to identify bias-conﬂicting samples.
Introduction
Deep neural networks (DNNs) have shown remarkable performance in the applications of computervision [18] and natural language processing [4].
However, the inductive biases contributing to thesuccess of DNNs also limit their capabilities for out-of-distribution (OOD) generalization.
Theyare vulnerable to learning undesirable decision rules based on the spurious features that have strongcorrelations with the labels, which we refer to as the dataset bias [30].
For example, if birds andblue skies frequently appear together in a dataset (i.e., a selection bias in the data), a DNN mightassociate bird labels with blue pixels rather than with the shape of the animal.
In the case of test-timedistribution shift and unpredictable shortcuts, the performance of the model degrades dramatically [7].
Biased datasets typically contain a majority of so-called bias-aligned examples and a minority ofbias-conﬂicting ones.
In bias-aligned examples, ground truth labels are correlated with both robustand biased features.. 1 In bias-conﬂicting examples, labels are correlated only with robust features. Clearly, the issues of models trained on biased datasets stem from the prevalence of bias-aligned samples.
Various approaches for debiased learning encourage models trained on biased datasets toignore biased features.
However, the identiﬁcation of biased features from i.i.d.
data is ill-deﬁned, andrequires additional assumptions or supervision with heterogeneous (non-i.i.d.) training samples [26].
In this work, we approach debiased learning with the popular heuristic that biased features are “easierto learn” than robust ones, meaning that they are incorporated in the model earlier during training [28].
Existing works that use this heuristic typically train two separate models: (i) an auxiliary modelthat purposefully relies on biased features, and (ii) the desired debiased model. The auxiliary modelserves to guide the training of the debiased one [23, 25]. For example, Nam et al.
[23] trains theauxiliary model with the generalized cross-entropy (GCE) loss [36] that strengthens its reliance onbiased, easy-to-learn features.
The simultaneous or subsequent training of the debiased model eitheraugments the data with novel bias-conﬂicting examples [16, 20] or upweights existing ones [23].
Augmentation can be more effective than upweighting but it requires careful tuning and disentangledrepresentations that makes the application to real-world data difﬁcult.
This work proposes a simple and effective approach based on the mixup [34], which implicitlyupweights the bias-conﬂicting samples in the dataset and generates new ones, but does not requireany complicated techniques such as disentanglement.
Mixup is a data augmentation method thatbuilds convex combinations of input images and their labels.
In a biased dataset, applying mixupon randomly-chosen pairs of examples is likely to involve two bias-aligned examples, since theyconstitute the majority of the data.
We argue that this could aggravate a propensity for the predictionbased on the biased features.
Therefore, we propose a selective mixup strategy coined SelecMix,where the mixup pair is selected so as to contradict the biased prediction of the model.
More precisely,
SelecMix selects contradicting pairs , i.e., pairs having (i) the same label but dissimilar biased features,and (ii) different labels but similar biased features.
In the absence of the explicit bias label, we utilizean auxiliary biased model to measure the similarity of the biased features (i.e., bias similarity ).
Toproperly measure the bias similarity, we propose to train the bias-ampliﬁed contrastive model with themodiﬁed supervised contrastive loss [14] to further facilitate the reliance on easy-to-learn features.
We evaluate the method on standard benchmarks for debiasing methods.
Experimental results suggestthat SelecMix with the bias-ampliﬁed contrastive model consistently outperforms prior methods.
Ablation studies show independent advantages of both (i) SelecMix, which is shown to be effectivecompared to the other mixup variants, and (ii) the bias-ampliﬁed contrastive model, which improvesthe biased prediction of the biased model and provides a better measure of bias similarity comparedto previous biased models.
A distinguishing capability of our method is its applicability to cases oflabel noise.
This setting is challenging due to the increased difﬁculty in identifying bias-conﬂictingexamples.
It better mimics real-world conditions in which debiasing methods could be useful.
Wedemonstrate this capability with novel versions of standard benchmarks that include label noise.
Related work
Debiasing when a bias label or a speciﬁc form of bias is known.
Prior works on debiasingassumed certain information about the biased features and leverage this to train a debiased modelfrom a biased dataset.
These approaches are categorized by the type of assumptions about the bias.
One category assumes that the explicit bias label is accessible [11, 15, 21, 24, 29].
Since the biaslabel is not always available, another line of work instead assumes a speciﬁc form of the bias (e.g.,color or texture) and devises an auxiliary bias-capturing model which is designed to intentionallylearn the biased features [1, 5, 32].
For example, ReBias [1] utilized BagNet [3], which is designed tobe intentionally biased toward the texture information, to learn the representation independent of it.
Debiasing biases that are learned more quickly than robust features.
A number of recent worksassume that biased features are learned more quickly than robust ones [20, 23].
A popular approachis to train an auxiliary biased model with the Generalized Cross Entropy (GCE) loss [36] deliberatelylearning the biased features.
The biased model is used to guide the training of the debiased model,e.g., to identify and upweight the loss of bias-conﬂicting samples [23] or to augment them [16, 20].
LfF [23] proposed to learn biased and debiased models simultaneously.
Based on the relative losson the biased and debiased model, each sample is reweighted having greater weights to the biasconﬂicting samples.
On top of their framework, DFA [20] emphasized the importance of diversifyingbias-conﬂicting samples and proposed a feature augmentation scheme, which disentangles the biasedand robust features and then randomly swaps the biased feature between the samples to generate
[Figure]
bias-conﬂicting samples.
Since augmentation-based methods [16, 20] require the disentanglement, itis challenging to scale to real-world datasets.
We resolve this issue by utilizing a devised mixup toefﬁciently generate the bias-conﬂicting samples.
Method
We ﬁrst brieﬂy describe the possible use of mixup as a debiasing strategy, assuming that each trainingexample is provided with a bias label that indicates a precise value for its biased features (Sec. 3.1).
Then, we move to a more realistic scenario where such labels are unavailable.
We introduce thebias-ampliﬁed contrastive model which compares examples with respect to biased features (Sec.
3.2)and describe SelecMix, the selective mixup strategy combined with our biased model (Sec. 3.3).
3.1 Mixup for diversifying bias-conﬂicting samples
Learning from scarce bias-conﬂicting samples is crucial since they allow us to distinguish betweenrobust features and biased features.
In this regard, the recent approaches identify and then upweighttheir loss [23], and generate bias-conﬂicting samples [16, 20].
For example, DFA [20] proposesto disentangle the biased and robust feature and randomly swaps the biased features among thesamples to generate diverse bias-conﬂicting samples.
The quality of disentanglement is crucial inthis approach, especially for the complex real-world images (e.g., to disentangle gender and agefeatures from the images of the human faces).
On the contrary, we utilize a simple augmentationstrategy, mixup [34].
Mixup interpolates a randomly-chosen pair of two samples and their labelsfor a generated sample and label, respectively.
It is well known to improve the robustness andgeneralization of the model [34, 35].
A generated sample of the vanilla mixup is deﬁned as follows:
[Equation]
where ( x i , y i ) is the input image and its label, ( x k , y k ) is the other randomly-chosen sample in abatch, and  2 [0 , 1] is a mixup coefﬁcient.
However, if the ratio of the bias-conﬂicting samples ↵ inthe biased dataset is very low (i.e. 1  ↵ ' 1 ), then two randomly-chosen samples are highly likelybias-aligned (i.e. (1  ↵ ) 2 ' 1 ).
Since the bias-aligned samples do not contradict the predictionbased on the biased features, the samples generated by mixing the pair of bias-aligned samples onlystrengthen the biased prediction of the model.
Therefore, applying random mixup on a highly biaseddataset is likely to be ineffective.
[Figure]
[Figure]
the reliance on biased features, thus the clustering in embedding space remains a good indicator of the similarity of biased features during the whole training process.
Our goal is to diversify bias-conﬂicting samples and prevent the model from learning a decisionrule based on the biased feature.
When the bias label is available, we can generate bias-conﬂictingsamples by applying mixup on the pairs having (i) the same label but different bias labels, and (ii)different labels but the same bias label.
However, since the bias label is not always accessible, weutilize the biased model to select the pair.
We proceed to elaborate on how to train and leverage thebiased model to select the desired mixup [-pair.3.
-]pair.
3.2 Replacing the bias labels with the bias-ampliﬁed contrastive learning
In the absence of the explicit bias label, we utilize a biased model to quantify the bias similarity ofthe pair of two samples and selectively choose the mixup pair based on this measure.
Exploitingthe heuristic that biased features are easier to learn than the robust features, the prior works [20, 23]utilized the Generalized Cross Entropy (GCE) loss [36], GCE ✓ ( p, y )
=
1  p qy q , to train the biasedmodel.
p and p y are the softmax output of the model parameterized by ✓ and its probability assignedto label y , respectively.
q 2 (0 , 1] is a hyperparameter.
Note that GCE is a generalization of thecross-entropy (CE) loss in the sense that GCE ✓ ( p, y ) !
CE ✓ ( p, y ) for every p and y as q ! 0 .
The GCE loss modiﬁes the gradient of cross-entropy (CE) loss as follows:
[Equation]
Since the loss of the bias-conﬂicting samples is relatively high, the GCE loss discourages the modelfrom learning the bias-conﬂicting samples.
It is known that the contrastive loss [6, 10, 14] more efﬁciently clusters the samples in the embeddingspace than the CE loss.
To this end, we propose to utilize a contrastive learning framework, whichlearns hyperspherical embeddings by measuring the similarity of the pairs.
Speciﬁcally, we utilizethe supervised contrastive loss (SupCon) [14] as follows:
[Equation]
1: Input : batch B = { ( x i , y i ) } Bi =1 , biased model g  2: Output : batch e
B = { (˜ x i , ˜ x i ) } Bi = 3:
Sample p
⇠ Uniform [0 , 1] , e
B = { ? } 4: for i = 1 , · · · , B do 5:
Sample  ⇠ Beta [ ↵ , ↵ ] 6:  min(  , 1   ) 7: if p > 0 .
5 : k = argmin
j 2 P i cos ( g  ( x i ) , g  ( x j )) 8: else: k = argmax j 2 N i cos ( g  ( x i ) , g  ( x j )) 9: ( e x i , e y i )
(  x i + (1   ) x k , y k ) 10: e
B e
B [ { ( e x i , e y i ) } 11: end for
1: Input : a dataset D = { ( x i , y i ) } , a model f ✓ , a biased model g  , the number of iterations T 2: Output : a debiased model f ✓ 3: Initialize ✓ and  4: for t = 1 , · · · , T do 5:
Draw a batch B = { ( x i , y i ) }
Bi
=1 from D 6: e
B = S ELEC M IX ( B , g  ) 7:
Update ✓ with L CE ( e
B ) 8:
Update  with L G-SupCon ( B ) {Eq.
(4)} 9: end for where B = { 1 , 2 , · · · , B } are the sample indices in a batch,
z i is the normalized embedding of 136 image x
i , P i = { k 2 B\{ i } | y i =
y k } is the set of positive samples for the sample i , and ⌧ is the 137 temperature of the softmax function.
We train on the biased datasets with the SupCon loss and compute the cosine similarity of the pair tomeasure the relative similarity.
Fig.
2 show that samples are clustered according to biased featuresearly in the training.
In other words, predictive rules involving biased features are learned faster thanthose involving robust features, as desired.
Our observation is aligned with the prior works that neuralnetworks tend to prefer easy-to-learn cues when trained with the CE loss [23, 27].
To further amplify the reliance of the auxiliary model on biased features, we modify the SupCon lossinspired by the GCE loss [20, 23].
The resulting generalized SupCon (G-SupCon) loss is written asfollows:
[Equation]
where ˆ p qi,k indicates the variable detached from p qi,k to be excluded in a gradient-based optimization.
Here, the term ˆ p qi,k impedes the model from learning from bias-conﬂicting samples and thus stablizelearning.
Notice that the G-SupCon loss more explicitly facilitates the feature clustering than the
GCE loss does by contrastive learning.
3.3 Selective mixup with the biased embedding space
Equipped with the bias-ampliﬁed contrastive model in Sec. 3.2 for quantifying bias similarity, wepropose to apply the mixup on the pairs having (i) the same label but dissimilar biased features, and(ii) different label but similar biased features.
Fig.
1 shows the overview of our proposed method.
Contradicting positives.
For each query x i , we pick the sample with the lowest cosine similarity(i.e.
, the maximum l 2 -distance in the biased embedding space) among the positives as follows:
[Equation]
where g  is the biased model.
Since we select the pair among the positives, the training loss ofthe mixed sample is l (
[Equation]
Since most of the samples are bias-aligned in the training set, the query ( x i , y i ) is also likely to bebias-aligned.
To effectively contradict the biased prediction of the model, we sample
⇠ Beta ( ↵ , ↵ )and assign the smaller value between  and 1   to the query x i and the larger value to x k .
Contradicting negatives.
We propose to select the negative sample that has the highest cosinesimilarity (i.e., the minimum l 2 -distance) for each query x
i as follows:
[Equation]
Similarly, we sample
⇠ Beta ( ↵ , ↵ ) and let  min(  , 1   ) .
In the vanilla mixup, the trainingloss of the mixed sample is: l ( e
[Equation]
i
[Equation]
[Equation]
[Equation]
+ (
[Equation]
[Table]
) x k , y i ) + (1   ) · l (  x i + (1   ) x k , y k ) .
However, considering the fact that (i) the query ( x i , y i )is likely to be bias-aligned, and (ii) the pair ( x i , x k ) shares a similar biased features, the ﬁrst term · l (  x i +(1   ) x k , y i ) does not contradict the prediction based on the biased features.
Thus, ratherthan interpolating the label, we simply assign e y i y k .
The pseudo-code of the proposed algorithmis presented in Alg. 1 and Alg.
Interpretation.
As shown in Fig.
1, our method can be viewed as the augmentation of scarce biasconﬂicting samples, by injecting bias-aligned samples as noise to existing bias-conﬂicting samplessince the label of the generated sample is assigned to the label of the selected sample, which islikely to be bias-conﬂicting.
Rather than simply suppressing the weight of the bias-aligned samples,the proposed method effectively leverages abundant bias-aligned samples in the training set.
Sincebias-conﬂicting samples are frequently chosen for the mixup pair, our method can be also interpretedas implicitly upweighting the bias-conﬂicting samples.
Experiments
In this section, we validate the effectiveness of the proposed method on standard datasets (Sec. 4.1).
In addition, we evaluate our method under the presence of label noise, which is more challengingand realistic, yet less explored (Sec. 4.1).
We also provide a detailed analysis of each componentof the proposed method, (i) bias-ampliﬁed contrastive model (Sec. 4.2) and (ii) selective mixup(Sec. 4.3).
We ﬁrst brieﬂy describe the experimental setup.
We provide the experimental details in
Appendix A.4.
Datasets.
The Colored MNIST is a modiﬁed MNIST [19] that consists of the colored images of tendigits where each digit is correlated with the color (e.g., most of the images of "1" is colored withred).
Here, the target is a digit (i.e. 0 ⇠ 9 ) and the bias is color.
Corrupted CIFAR10 is constructedby applying different types of corruptions to the corresponding objects in the original CIFAR-10 [17]dataset (e.g., most of the images of dog is corrupted with G AUSSIAN BLUR noise).
Biased FFHQ(BFFHQ)
[20] is constructed based on the real-world dataset FFHQ [13], where the (target, bias) is(age, gender).
The ratio of bias-conﬂicting samples in the training set is ↵ 2 { 0 .
5% , 1% , 2% , 5% }for {Colored MNIST, Corrupted CIFAR10} and ↵ = 0 .
5% for BFFHQ.
All datasets are available inthe ofﬁcial repository of DFA [20].
We defer the detailed description of the datasets in Appendix A.2.
Baselines.
To evaluate the effectiveness of our method in debiasing, we compare it with the priormethods LfF [23] and DFA [20] which also aim to mitigate the bias which is more easy-to-learn.
LfF [23] trains the biased model and debiased model with GCE loss and CE loss, respectively, wherethe samples are reweighted based on the relative difﬁculty of the two models when training thedebiased model.
DFA [20] disentangles the biased and robust feature based on the framework of LfF and swaps the biased feature between samples i.e. feature augmentation.
We also compare withthe EnD [29] which leverages explicit bias labels, including ReBias [1] and HEX [32], bias-tailoredmethods exploiting the pre-deﬁned form of bias (i.e., color and texture).
VanillaV+mixLfFDisEntV +Ours(AB)L+Ours(AB)
A cc u r a cy
Noisy label (%)     (a)
Noisy label (%)     (b) Noisy label (%)     (c)
Noisy label (%)     (d)
[Table]
4.1 Main results
We apply our method to the vanilla ResNet18 (V+Ours) and the LfF (L+Ours) and evaluate these onthe Colored MNIST, Corrupted CIFAR10, and BFFHQ.
As shown in Table 1, our method consistentlyoutperforms baselines, except for ReBias [1] on the Colored MNIST.
Note that their bias-capturingmodel BagNet [3]) directly accesses the local patches (e.g., color and texture), circumventinglearning biased features.
While both DFA and ours diversify scarce bias-conﬂicting samples, DFA’sdisentanglement seems to struggle on complex real-world datasets.
In contrast, our method is lessaffected by the complexity of the dataset, thanks to the simplicity and effectiveness of the mixup.
Speciﬁcally, our method outperforms DFA by a large margin on BFFHQ while the gap is relativelysmall on Colored MNIST, where the disentanglement is much easier.
Debiasing under the presence of label noise.
The methods that implicitly or explicitly detect thescarce bias-conﬂicting samples for debiasing are especially vulnerable to the label noises since theyidentify noisy samples and regard these samples as the bias-conﬂicting samples to amplify.
As shownin Fig.
3, our proposed method is relatively robust to label noise compared to the baseline methods.
Notably, our method improves the generalization capability of the base model even when there aretwice as many noisy label samples as the bias-conﬂicting samples in the training dataset.
We providethe details of the label noise experiments in Appendix A.4.
4.2 Detailed analyses of the bias-ampliﬁed contrastive model
Comparison of the biased embedding spaces quantifying bias similarity.
We compare the biasampliﬁed contrastive model g with the biased model of LfF, g GCE , trained with the GCE loss [20, 23]by the various metrics for quantifying the bias similarity.
For the metrics, we used l 2 distance, cosinesimilarity, and the KL-divergence of the softmax outputs of the biased model of LfF. Notice that ourmethod uses cosine similarity.
As shown in Table 2, the proposed g achieves the best performance.
Especially, the performance gap is signiﬁcant for the contradicting negatives.
This supports ourclaim in Sec. 3.2 that the contrastive loss effectively clusters the features with the same bias label.
Considering that (i) the contrastive loss clusters the samples with the same label and (ii) most of thepositive pairs have the same bias label in the biased dataset, it effectively attracts the samples with thesame bias label in the embedding space.
In contrast, the GCE loss learns the decision boundary w.r.t.
[Table]
[Figure]
the bias labels and does not explicitly cluster the features, thus selecting the contradicting negatives,i.e., the pairs with different labels but a similar biased features, seems to underperform ours.
Bias prediction of the biased model.
Table 3 shows the unbiased test accuracy of the bias labelprediction of the biased model for each method.
The bias-ampliﬁed contrastive model does not havea classiﬁer since it is trained with a contrastive loss.
We attach a linear classiﬁer on top of the modeland ﬁne-tune it.
Note that the bias label is used only for the evaluation.
As shown in Table 3, thebias-ampliﬁed contrastive model shows the best performance.
As the ratio of the bias-conﬂictingsamples increases, they degrade the performance.
Notice that bias-ampliﬁed contrastive model isrobust to this effect compared with the prior methods relying on the GCE loss.
Visualization.
Fig.
4 illustrates t-SNE [31] plot of the features extracted from the correspondingmodels trained on the Corrupted CIFAR10.
As shown in Fig. 4(d), which corresponds to the proposedbias-ampliﬁed contrastive model, we observe that the samples are well clustered on the embeddingspace of the bias-ampliﬁed contrastive model.
Similar to the results in Table 3, the GCE-based biasedmodels, i.e., LfF and DFA, suffer from the increasing number of bias-conﬂicting samples.
4.3 Detailed analysis of the SelecMix
Ablation study.
Table 4 shows that the vanilla mixup degrades the generalization capability.
Whileboth the contradicting positives (A) and the contradicting negatives (B) signiﬁcantly contribute to theperformance across the benchmarks for both the base models, Vanilla and LfF. Note that we reportthe accuracy of SelecMix with both of its components, i.e., Ours (AB), for all experiments in themain results (Table 1).
Comparison with a mixup variant when bias label is available.
We compare SelecMix with therecently proposed LISA [33], the mixup strategy addressing domain generalization and subpopulationshift problems.
LISA similarly applies the vanilla mixup on the pair of (i) the samples with the samelabel but different domains, and (ii) the samples with different labels but the same domain.
Here, thedomain label corresponds to the bias label in our experiment.
The key differences between SelecMixand LISA are (i) ours does not need the bias label (Sec. 3.2), (ii) we do not use the coefﬁcient to interpolate the labels of the generated samples for the reason described in Sec. 3.3, and (iii)we sample  and let  min(  , 1   ) , i.e., we always assign the higher weight to the selected
[Table] [Table]
sample and the lower weight to the query sample (Sec. 3.3).
Table 5 empirically conﬁrms the betterperformance of SelecMix, using the explicit bias label for a fair comparison, over LISA, implyingthat augmenting the samples close to the bias-conﬂicting samples is crucial on the biased datasets.
More discussions on the comparison between the SelecMix using the explicit bias label and LISA isprovided in Appendix A.7.
[Table]
When equipped with other types of bias capturing model.
To validate the compatibility of theselective mixup with other types of bias capturing models, we utilize the BagNet [3] capturing thetexture bias.
Speciﬁcally, we pretrain BagNet for 120 epochs and use this model to measure thebias similarity for our SelecMix.
As shown in Table 6, it signiﬁcantly improves the generalizationcapability of the vanilla network.
We defer the experimental details to Appendix A.5.
Discussion and conclusion
In this work, we present the SelecMix for debiasing, where the selective mixup is applied to the pairsof samples having (i) the same label but dissimilar biased features, and (ii) different label but similarbiased features.
For measuring the bias similarity, we propose the bias-ampliﬁed contrastive learningmodel.
Extensive experiments on the extensive benchmarks suggest that our method signiﬁcantlyimproves the performance of debiasing.
Furthermore, the evaluation in the noisy label environmentshows the robustness of our method.
Limitation.
First, our method exploits the easy-to-learn propertyof bias for training the auxiliary biased model.
All biased features may not be learned faster thanrobust features, although it has not negative effect since the easy-to-learn biases are problematic.
Second, the SelecMix requires an auxiliary biased model to quantify the bias similarity, while wecan exploit a pretrained biased model to reduce the training cost, as we have shown in Appendix.
Societal impact.
If we fail to capture the biases in the auxiliary biased model, it potentially generatesthe biased output.
We may need a supplementary means for critical systems. Checklist For all
authors..
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix A
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...(
a)
If your work uses existing assets, did you cite the creators?
Yes
b) Did
you mention the license of the assets
[Yes]
c) Did you include any new assets either in the supplemental material or as a URL?
[No
d)
Did
you
discuss
whether and
how consent was
obtained from
people whose data you’reusing
/curating?
[No
](e
Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[No]5.
If you used crowdsourcing or conducted research with human subjects...(
a)
Did you include the full text of instructions given to participants and screenshots, ifapplicable?
N/A
b)
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable
N
A
c)
Did you include the estimated hourly wage paid to participants
and
the total amountspent on participant
compensation?
[N/A