Alleviating the Sample Selection Bias in Few-shot Learning by Removing Projection to the Centroid
Abstract
Few-shot learning (FSL) aims to achieve good generalization without sufficient annotations in the novel classes.
Despite the successes of a number of few-shot learning methods , the sensitivity to the limited amount and the discriminative power of support data is not well understood, which is also called the sample selection bias problem.
This paper reveals one such phenomenon —- degenerated and unstable classification results are usually observed if the position of support samples are in the vicinity of the data centroid for a given task, which we call the task centroid .
To alleviate this sample bias, aggravated by the bad effect of the task centroid, we propose a simple feature transformation, named Task Centroid Projection Removing (TCPR).
TCPR aims to remove the component of features along the direction of approximated task centroid which is estimated through similar examples from the base dataset.
This effectively prevents features from being too close to the task centroid.
Extensive experiments over ten datasets from different domains show that TCPR can reliably improve classification accuracy across various feature extractors, training algorithms and datasets.
The code can be found in the https://github.com/KikimorMay/FSL-TCBR .
Introduction
Deep networks, especially Convolutional Neural Networks (CNNs) [15, 36, 13, 47, 30], have achieved great successes in image classification, relying on massive labeled image data.
However, large-scale annotated datasets are costly and even infeasible to obtain in many real-world applications.
As a new learning paradigm, Few-shot learning (FSL) [9, 41, 37], has been recently proposed aiming to recognize novel classes of data (known as the query set) given only a few labeled images (known as the support set) to achieve good generalization ability without sufficient annotations.
A simple yet efficient scenario tackling few-shot classification is to first learn a good feature extractor from an large additional dataset composed of base classes (which are different from the classes to recognize) and then a linear classifier upon it on the novel classes [40, 7].
Despite the successes of this scenario, the classifier constructed from the support data is not reliable enough, since the small number of samples is hard to accurately estimate the whole class distribution.
Thus, the performance on the query set is strongly correlated with the sampling of the support data leading to an ambiguity effect in classification, which is known as the sample selection bias.
To measure the ambiguity, we first define a new concept called the task centroid which is defined as the ground-truth center of all the novel data in each classification task.
Intuitively speaking, the direction of the task centroid
[Figure]
The correlation between the classification accuracy and the average distance of all the support data to the task centroid. The accuracy gets lower with a large variance when the support samples are closer to the task centroid. After the using proposed transformation, the bias problem is alleviated. can be regarded as the commonality (or the shared pattern) among the novel classes.
If the support data of one class is similar with other classes, the model lose the disciminative ability of the class.
As illustrated in Figure 1(a), considering the case that only one support sample (also called a shot) per class is observed, when these two samples are close to the task centroid, the classifier learnt from these samples will be largely misleading; while when both samples are far away from the task centroid, the corresponding classifier could be relatively accurate.
In this way, the classification ambiguity in FSL highly correlates with the distance between the class prototypes (obtained by the support samples in each class) and the task centroid.
To further understand the correlation in the typical FSL setting, we conduct a binary classification of two novel classes ( “Lion” and “Dalmatian”) in mini Imagenet by sampling 10 , 000 tasks in each of which we sample one image per class as the class prototype (i.e., support data).
Then we report the average and variance of the classification accuracy for tasks with the different distance (rounded to the scale of 0 . 1 ) to the estimated task centroids.
As shown in Figure 1(b), the tasks with support data closer to the centroid generally result in a lower average accuracy value (plotted in blue) while with a high variance (plotted in red).
This suggests that the vicinity of the task centroid often leads to poor classification performance due to strong sample selection bias.
To alleviate the sample bias, we propose a simple transformation on novel features by removing the projection along the direction of an approximated task centroid, named Task Centroid Projection
Removing(TCPR).
Since the ground-truth task centroid is not available and a small amount of support data is inadequate for a good approximation, we seek the assistance of the base set based on the observation: the distribution of novel classes tends to cluster in some primary direction in the feature space [39], which means only limited images in base set are correlated to the novel samples.
Inspired by that, we estimate the direction using the most similar top k base samples to the support data.
Since our proposed transformation removes the projection to the centroid, the model become more discriminative while mitigates the impact of the commonality of different classes. After transformation, the sampling bias problem can be relieved shown in Figure 1(b).
The transformation helps the classifier capture the object of interest from the misleading shared patterns(see Figure 6).
Our method is generic, flexible, and agnostic to the pre-training based FSL methods.
Empirical experiments show that the transformation function can consistently and largely boost the classification performance on ten datasets with various gaps between base classes and novel classes, demonstrating the importance of reducing the widely existing sampling bias.
Our contributions are summarized as follows: 1) To the best of our knowledge, we are the first to unravel the connection of the sample selection bias with the task centroid, which is tied with FSL and has been ignored so far; 2) To reduce the sample bias, we propose a simple transformation to remove projection to an approximated task centroid with the assistance of related neighbors in the base set, and 3) Comprehensive experiments show the effectiveness of the proposed transformation.
Related Work
Most recent few-shot learning approaches can be categorized into three groups: optimization-based, generation-based and metric-based.
The optimization-based methods aim to learn models with good initialization so that the model can quickly adapt to novel set with a limited number of gradient descent on a few labeled examples, including MAML [9, 10], Reptile[29], LEO[35],Meta-B[5], MeTAL[2] and S / T [21].
The generation-based methods deal with data deficiency problem by generating pseudo support samples based on extra prior knowledge [45].
Some work utilizes the transferable intra-class deformations in base set like SGM [11], VFD [48]and DC [49].
Other methods use high level semantic embedding of classes as extra prior information to generate novel samples, i.e., class-level attributes [50], word2vec [27, 52] or unlabeled query data [20].
Metric-based methods attempt to encode images into a well-shaped feature embedding space and perform classification with distancebased classifiers, from a metric learning perspective, e.g., MatchingNet[41], ProtoNet [37] and
Relation Network[38].
Some studies reveal that the standard transfer learning paradigm can achieve surprisingly competitive performance in few-shot learning [4, 7, 6, 43].
In the paradigm, the models are trained on base set to learn high quality feature extractors and only the classifiers are learned with novel samples.
Based on that observation, several work considers to improve the generalization ability of learned image representations on the base set with some auxiliary pretext tasks.
Inspired by recently proposed self-supervised learning methods [12, 3], some FSL methods propose to use the contrastive learning with some regularization techniques to learn high-quality feature representation, e.g., S2M2 [26], Inv-Equ[33], VLCL [22], PAL [25], ArL [52] and POODLE [16].
Those methods demonstrate the significance of powerful feature representations in FSL.
However, the generalization ability will be damaged when there exists a large distribution shift between base and novel classes, leading biases in novel-class feature distributions [39].
COSCO [23] manifests that image background is one harmful source knowledge for FSL when category gap exists between training and testing.
Luo et al.
[24] reveal that channels may have different importance in different tasks, while CNNs are likely insensitive to such a shift.
Some simple transformation functions reshape the skewed distribution of novel features to improve the classification performance.
The l 2 normalization can remove the negative effect caused by the magnitude of features [4, 19, 40].
Based on that, SEN [28] with l 2 normalization transforms features onto the hyper-sphere while keeping features repel from all other prototypes.
SimpleShot [44] subtracts the mean of whole training set features before l 2 normalization.
However, when there exists a large distribution shift between base and novel set, the performance of SimpleShot drops quickly since the mean of whole training set features is not good static information for the various distribution of novel set.
ZN [8] discovers the existence of the hubness problem in FSL and uses ZN normalization to address it.
For every component of an individual novel feature, the ZN transformation subtracts the mean of all components and then divides the standard deviation of all components.
DCM [39] calibrates the distribution of novel samples to approach zero centered mean and unit standard deviation using support data to improve the evaluation performance.
Compared with ZN and DCM, the motivation of our TCPR is different.
We focus on reducing the classification bias when the support samples in the vicinity of local centroid.
Besides, both ZN and DCM transform novel features without the help of base set, which may lose some useful information.
On the other hand, using a few samples in novel classes also results in a bias in task sampling.
Some studies [1, 49, 39, 46] show that modern few-shot learning algorithms are extremely sensitive to the data used for adaptation.
However, none of them analyzes the connection of the sampling bias to the effect of task centroid.
Methods 3.1 Preliminaries
In few-shot classification, we are given a base set D b and a novel set D n with disjiont classes C b and
C n , i.e., C b ∩ C n = ∅ .
Training takes place on D b , and few-shot evaluation takes place on D n .
Each
FSL evaluation episode consists of a N -way K -shot task, which evaluates model’s ability to learn a new classifier discriminating N novel classes of D n with only K labeled data available in each class.
The N × K labeled samples comprise the support set S
= { ( x i , y i ) }
N × K i =0 .
Based on the labeled support set S and the base dataset D b , our goal is to predict the category of all the unlabeled samples in the query set Q = { ( x i , y i ) } N × K + N × q i = N
× K +1 correctly, where q denotes the number of images for each class in the query set.
The query set is sampled from the same N novel classes as S in each task.
Our work is based on the popular pre-training pipeline.
The sufficient labeled base samples are used for pre-training.
For novel classes, we fix the parameters θ in feature extractor f θ unchanged. We
[Figure] learn a new classifier f W using the limited samples in the support set S to predict labels for query samples Q .
Our approach is of feature-level and is independent of the pre-trained backbones.
Pre-training Stage: Recently, some studies [40, 7] have demonstrated that the significance of powerful feature representations with a simple feature embedding network f θ learned on base classes through a image classification proxy task has good transferability to novel samples.
Furthermore, to strengthen the generalization of f θ , the self-supervised learning with other proxy tasks, e.g., colorization and rotation, is used as auxiliary loss in FSL.
To show our method is agnostic to the feature extractor, we implement the proposed transformation on different algorithms, including
Baseline++ [4]( which trains f θ with vanilla cross-entropy loss); S2M2 [26](which uses mixup regularization and an auxiliary task to predict the rotation angles);
Inv-Equ [33] (which uses sixteen geometric transformations for self-supervised learning and knowledge distillation techniques).
Evaluation stage: For each N -way K -shot task, we train a new linear classifier f W using the support set S with N × K limited labeled samples.
Let W =
[ w 1 , w 2 , ..., w N ] ∈ R d × N denotes the learn-able parameters of the classifier, where d is the dimension of the extracted features.
We normalize the weight vector for each class c , where || w c ||
= 1 , c ∈ [1 , 2 , ..., N ] , to eliminate the effect of the magnitude of the feature vector[42].
The probability of a sample x belonging to class c can be formulated as:
[Equation]
where γ is a scaling hyper-parameter.
After training on S , the weights vectors [ w 1 , w 2 , ..., w N ] can be seen as prototypes of novel classes and used to predict the samples in the query set Q .
The classification result is based on the distance of the query image features to these learned prototypes.
Although the simple pre-training strategy has achieved promising performance with well-designed feature extractors, the ambiguity in classification caused by the few samples in the novel classes is still unsolved.
Besides, the ambiguity is further aggravated in the vicinity of the task centroid.
3.2 Task Centroid Projection Removing
In this work, we propose a simple transformation to remove the sample bias exacerbated by the task centroid, named Task Centroid Bias Removing(TCPR).
The overview of the proposed TCPR is shown in Figure 2(b), which consists of three steps: 1) given support data S from novel classes, we find the most related top k samples in the base set; 2) using the top k neighbors, we approximate the task centroid vector with the help of the statistics of the base set; 3) and, we alleviate the harmful effect of the approximated centroid by removing the component of novel features along the direction.
Step 1: Seeking the base neighbors.
Our work is based on the phenomenon that the domain shift between base and novel sets causes the skewness of novel class distribution to some specific directions [39].
As shown in Figure 2(a), we visualize the distributions of the base dataset, i.e., mini Imagenet (which is a coarse-grained dataset with a wide range of object categories) and the novel dataset, i.e., CUB (which is a fine-grained dataset that only contains various bird categories), showing that the novel classes are close to only some specific base classes and skew to the direction.
Inspired by the observation, we try to approximate the task centroid by looking back upon similar base samples.
In the N -way K -shot task, given novel support normalized features x ni ∈ S , || x ni ||
= 1 , i ∈
[1 , 2 , ..., N × K ] , extracted by the pre-trained feature extractor f θ , we calculate the mean of S by averaging the extracted features of all support samples, that is,
[Equation]
Using the mean of the support features, we seek the top k base samples with the closest angular similarity (evaluated by the cosine distance) between the ¯ x n and x bi ∈ D base , i ∈
[1 , 2 , ..., N base ] :
[Equation]
where the notation cos ( · , · ) denotes the cosine distance between two vectors and topK ( · ) is an operator to select the largest k elements in the distance set D cosine 2 .
The set D topK contains the most related k samples in base classes.
Step 2:
Approximating the task centroid.
We approximate the task centroid by a weighted aggregation of x bi , i ∈
D topK , i ∈ { 1 , 2 , ..., k } based on the cosine similarity in D cosine .
The weight w i of each x bi can be constructed as:
[Equation]
where p is the hyper-parameter controlling the relative weights of the base samples.
The approximated task centroid c task is formulated as:
[Equation]
where the notation L 2 ( · ) denotes the l 2 normalization.
The approximation of task centroid by the base neighbors, c task , is more accurate and stable than the means of support set, ¯ x n .
With only a few examples, ¯ x n is easy to overfit on the support data and far from mirroring the ground truth task centroid of the novel classes.
Since the novel classes skew in a certain direction toward relevant limited base samples, extrapolating the task centroid from the top k similar samples in base set can be seen as a more precise alternative.
We can change the hyper-parameter k to cover the distribution of novel classes with different domain gaps.
Step 3:
Removing the projection to the task centroid.
To reduce the sampling bias caused by the support samples in the vicinity of the task centroid, we propose to eliminate the effect of the direction of the task centroid by pushing novel features away from it.
For all the novel samples in both support and query set, x n ∈ S ∪ Q , we use a simple transformation which extracts the projection to the approximated task centroid c task :
[Equation]
After removing the component of features along the direction of c task , the distribution of novel classes is pushed away from the approximated centroid, shown in Figure 3.
Since the direction of task centroid can be seen as the commonality of classes of each task, it contains the non-discrimintaive information which inhibits the classification results.
TCPR projects out the harmful direction and constructs a better latent space.
[Figure]
3.3 The Connection of Task Centroid to Ambiguity in FSL
In this subsection, we analyze the classification ambiguity aggravated by the task centroid, and further reveal that it is a naturally-occurring phenomenon in FSL since the limited samples result in a poor estimation of the class prototype, and the small distribution margin among novel classes makes the ambiguity more severe.
For a better understanding, we design a toy simulation of a binary classification task.
Specifically, let D 1 = N ( µ 1 , Σ 1 ) , D 2 = N ( µ 2 , Σ 2 ) be two Gaussian distributions, where µ 1 =
[ − a, 0] , µ
=
[ a, 0] and Σ 1 = Σ 2 = I .
The task is simply to discriminate between samples from the two classes.
The task centroid of the two classes is the origin point (0 , 0) .
We use Nearest-Centroid Classifier [37] without any regularization to learn the decision boundaries, which first averages points from each class in the support set to form class prototypes and assigns query points to the class of the nearest prototypes.
We randomly sample 10 , 000 tasks for each simulation experiment.
The connection between classification accuracy and the average distance from two class prototypes to the task centroid is shown in Figure 4.
With the decrease of the distance, the average of accuracy results gets lower, while the variance of accuracy results gets larger, as shown in the 1 -shot task with a = 1 .
Intuitively, the sampling bias in the vicinity of task centroid ties with the number of shots in the FSL task.
With the increase of the shots, the variance of the class prototypes will diminish.
In K -shot task, the covariance matrix of class prototype is Σ
= I /K
When the number of shots is large enough, the class prototypes can estimate the class centroid accurately with a low variance and the sampling bias is alleviated.
This phenomenon is clearly shown in Figure 4(a)
when K = 10 .
The poor estimation of the class prototypes caused by limited labeled data is one of the reasons why the sampling bias decreases the performance of FSL.
On the other hand, the bias correlates with the discriminability of the two classes’ distribution.
We vary the value of a from a = 0 .
5 to a = 3 with a wide range to simulate distribution with different discriminability in 1 -shot tasks.
The two classes are more well-separated with larger a .
As shown in Figure 4(b), the bias problem begins to emerge with a = 0 .
5 , when there exists a large overlap between two classes’ distribution.
With the increase of a , the bias problem is more severe at first and is relieved later.
It almost disappears when a = 3 , where the inter-class distance is much larger than the intra-class variance.
The feature distribution which is suffering from the sampling bias is highly corresponding to the few-shot learning classification problem.
Recent studies have shown that the
[Table] high-quality image representations can be transferred from base to novel set [Table] but there does not exist enough margin among novel classes due to the domain gaps.
This leads to another factor causing the classification ambiguity aggravated by the task centroid.
Similar phenomenon can be observed among the real data in higher dimensional space, as illustrated in Figure 1(b) [Table]
Experiments
Datasets.
We evaluate our approach on four widely-used FSL datasets.
The mini ImageNet[31] is a coarse-grained dataset with distinct classes of animals or objects.
It is a subset of ILSVRC-12 [34], including 600 images of each class.
Following the previous work [31], the categories are split into 64 , 16 , 20 classes for training, validation and evaluation respectively.
tiered ImageNet is a much larger and more challenging dataset.
It is made up of 779 , 165 images from 608 classes sampled from a hierarchical category structure.
We adopt 351 classes as base categories, 97 as validation categories and 160 as novel categories as suggested in [32].
The CUB is a fine-grained dataset consisting of 11 , 788 images from 200 bird classes.
We spilt the dataset into 100 , 50 and 50 categories as base, validation and test categories, following [4].
Meta-dataset is a significantly larger-scale dataset which comprised of multiple datasets of diverse data distributions, presenting more realistic tasks.
Implementation Details.
We adopt the ResNet-18/12 [4, 33] and WRN [26] as the feature extractors f θ for a fair comparison with published results.
All the extracted features are normalized using the lnormalization.
We use the cosine classifiers to learn the decision boundaries of the novel set.
The p in Equation 4 is set to be 0 .
The number of related base neighbors are selected according to the degree of domain difference between base and novel set.
We set a larger k with a small task distribution shift, e.g., transferred from mini-train to mini-test, and a lower k with a larger distribution shift, e.g., transferred from mini-train to Quick Draw.
The details of the selection of k can be found in Section 4.3 Hyper-parameter selection.
All the ablation studies and analysis experiments are conducted with S2M2 [26].
We report the mean accuracy as well as the 95% confidence interval on 2000 randomly generated episodes. [Table] [Table]
4.1 Experimental Results
Note that any FSL method which follows the pre-training pipeline can be employed as the baseline in our method.
To show our method is agnostic to feature extractors, we use the algorithms proposed in Baseline++ [4] , S2M2 [26], Inv-Equ [33] to obtain the embedding features as baseline models, without loss of generality.
After applied TCPR transformation, we get 3% − 4% improvement on 1 -shot task and nearly 1% on 5 -shot task.
The improvements are larger in 1 -shot task than 5 -shot, since the sampling bias is more severe in 1 -shot task, as shown in 3.3.
Furthermore, we combine them to improve the generalization ability of brand-new images.
Given original images, the strategy randomly samples 9 patches with different sizes and shapes, followed by resizing these patches to 84 × 84 .
We achieve remarkable 70 .
67% and 86 .
61% accuracies for 1/5-shot tasks showing that the most state-of-the-art FSL methods still suffer from the biased problem caused by task centroid and the problem can be alleviated by our method.
The stable improvements validate the general applicability of TCPR for FSL.
We also observe stable improvement on tiered Imagenet shown in Table 2.
4.2 Statistical Analysis
The approximation of task centroids.
We approximate the task centroid of a novel task using four different statistic approximations, the mean of the support and query set ¯ x n S∪ Q , the mean of the support set ¯ x n S and the weighted summation of base neighbors c task .
We can observe from Tablethat: (1) The ¯ x n C novel is the most accurate approximation of the task centroid, which is not available in inductive few-shot tasks and can be seen as the transductive setting.
The improvements on all datasets by a large margin show the existence of the detrimental impact of task centroid, and it can be alleviated by our TCPR transformation; (2) Compared with the limited information in ¯ x n S , seeking the related samples in the base set as assistance, c task , can be a better estimation.
The comparison with alternative normalization operations.
We also compare different transformations on Meta-dataset shown in Table 3 based on pre-trained models.
The SimpleShot [44] subtracts the mean of whole training set features before l 2 normalization.
The improvements are limited when the train-test gaps are large.
The Z-score normalization [8] is applied to each feature vector independently, causing a loss of utilizing the task centroid information.
Our proposed TCPR achieves better performance with the assistance of the approximation of task centroid, which makes it insensitive to the base-novel gap.
The task centroid clusters different novel classes in the same direction and hurts the discriminability.
We visualize the cosine similarities among 20 novel class centers of mini Imagenet before and after the proposed transformation, as shown in Figure 5.
It is clear that the centers of novel classes are scattered in the same direction (task centroid of the novel set), which forms clusters with [Figure] large cosine similarity before transformation.
After the transformation removes the projection along the direction, samples of different classes in the feature space are more separable.
The projection on the task centroid vector distracts the neural network from the main objects.
Figure 6 shows the Grad-Cam activation maps of some query samples with and without the TCPR transformation.
We observe that our transformation by removing the projection of the task centroid helps the model adjust attention to the objects for classification with only a few support images.
Before transformation, skewness of features makes the classifier confused by the shared structure and distracts the neural network from the main objects.
After removing the projection on task centroid vector of task direction, it is easier for the classifier to capture the objects of interest.
4.3 Ablation Study
Hyper-parameter selection.
Selection of the number k of similar base neighbors plays an essential role in approximating the task centroid of novel classes.
We show how hyperparameter k influences the few-shot classification performance on different novel sets with varied domain differences in
Figure 7.
The optimal value of k is different according to the difference of the base-novel domain.
Evidently, datasets that are similar to the train split of mini Imagenet, like the test split of mini Imagenet and Coco, often result in a larger number of optimal k .
Datasets with less domain gap require small k , like the QDraw and Omniglot.
It is necessary to mention that the DTD is a texture dataset.
So, it is hard to find the most related base samples, using the whole training set can be a better estimation.
The results also manifest importance of a good approximation of task centroid since different approximations will cause significant change in accuracy .
Performance with different number of ways and shots.
We show the results with different numbers of ways and shots before and after the simple transformation in Table 4.
The transformation improvement is more significant when the number of shots is lower since the bias problem is more severe in the low-shot situation.
[Table]
Conclusion and Future Work
In this work, we disclose a phenomenon that few-shot classification performance would be very sensitive to the position of support samples if they are in the vicinity of the task centroid, which leads a decrease of accuracy with a large variance in different tasks.
A simple feature-level transformation, named TCPR, is proposed to address bias by removing the component of novel features to an approximated task centroid direction achieving consistent improvements on Meta-datasets.
In this paper, we consider the sampling bias effect of task centroid.
In the future, we will explore the influence of variance of data.
Besides, a self-adaptive selection algorithm of the number of k in base set is worth to explore to improve the generalization of TCPR.
Furthermore, we plan to apply our method to dense prediction FSL tasks like object detection/segmentation in the near future.
Acknowledgments and Disclosure of Funding
This paper was partially supported National Key Research and Development Program of China (No. 2018AAA0100204) and a key program of fundamental research from Shenzhen Science and Technology Innovation Commission (No. JCYJ20200109113403826) and the Major Key Project of PCL (No. PCL2021A06), Foundation of Zhejiang Province No.LQ20F030007 and the National
Natural Science Foundation of China No.62206256)
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Yes.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]