Mitigating Dataset Bias by Using Per-sample Gradient
Abstract
The performance of deep neural networks is strongly influenced by the training dataset setup.
In particular, when attributes having a strong correlation with the target attribute are present, the trained model can provide unintended prejudgments and show significant inference errors ( i.e., the dataset bias problem ).
Various methods have been proposed to mitigate dataset bias, and their emphasis is on weakly correlated samples, called bias-conflicting samples .
These methods are based on explicit bias labels provided by human.
However, such methods require human costs.
Recently, several studies have tried to reduce human intervention by utilizing the output space values of neural networks, such as feature space, logits, loss, or accuracy.
However, these output space values may be insufficient for the model to understand the bias attributes well.
In this study, we propose a debiasing algorithm leveraging gradient called PGD ( P er-sample G radient-based
D ebiasing).
PGD comprises three steps: (1) training a model on uniform batch sampling, (2) setting the importance of each sample in proportion to the norm of the sample gradient, and (3) training the model using importance-batch sampling, whose probability is obtained in step (2).
Compared with existing baselines for various datasets, the proposed method showed state-of-the-art accuracy for the classification task.
Introduction
Dataset bias [63, 57], is a bad training dataset problem that occurs when unintended easier-to-learn attributes (i.e., bias attributes) , having a high correlation with the target attribute, are present [56, 2].
This is due to the fact that the model can infer outputs by focusing on the bias features, which could lead to testing failures.
For example, most “camel” images include a “desert background,” and this unintended correlation can provide a false shortcut for answering “camel” on the basis of the “desert.”
In [48, 37], samples of data that have a strong correlation (like "desert background" in “camel” class images) are called “bias-aligned samples,” while samples of data that have a weak correlation (like “camel on the grass” images) are termed “bias-conflicting samples.”
To reduce the dataset bias, initial studies [29, 45, 58, 40] have frequently assumed a case where labels with bias attributes are provided, but these additional labels provided through human effort are expensive.
Alternatively, the bias-type, such as “background” is assumed in [38, 17, 7, 10, 13].
However, assuming biased knowledge from humans is still unreasonable since even humans cannot predict the type of bias that may exist in a large dataset [53].
Data for deep learning is typically collected by web-crawling without thorough consideration of the dataset bias problem.
Recent studies [35, 48, 30, 37, 54, 67] have replaced human intervention with DNN results.
They have identified bias-conflicting samples by using empirical metrics for output space ( e.g., training loss and accuracy).
For example, [48] suggested a “relative difficulty” based on per-sample training loss and thought that a sample with a high “relative difficulty” was bias-conflicting sample.
Most of the previous research has focused on the output space, such as feature space (penultimate layer output) [37, 30, 7, 54, 67], loss [48], and accuracy [35, 43].
However, this limited output space can impose restrictions on describing the data in detail.
Recently, as an alternative, model parameter space ( e.g., gradient [23, 28, 47]) has been used to obtain high-performance gains compared to output space approaches for various target tasks.
For example, [23] used gradient-norm to detect out-of-distribution detection samples and showed that the gradient of FC layer ∈
R h × c could capture joint information between feature and softmax output, where h and c are the dimension of feature and output vector, respectively.
Since the gradients of each data point ∈ R h × c constitute high dimensional information, it is much more informative than the output space, such as logit ∈ R c and feature ∈ R h .
However, there is no approach to tackle the dataset bias problem using a gradient norm-based metric.
In this paper, we present a resampling method from the perspective of the per-sample gradient norm to mitigate dataset bias.
Furthermore, we theoretically justify that the gradient-norm-based resampling method can be an excellent debiasing approach.
Our key contributions can be summarized as follows:
• We propose PGD, P er-sample G radient-norm based D ebiasing, a simple and efficient gradientnorm-based debiasing method.
PGD is motivated by prior research demonstrating [47, 23, 28] that gradient is effective at finding rare samples, and it is also applicable to finding the bias-conflicting samples in the dataset bias problem (See Section 3 and Appendix D).
• PGD outperforms dataset bias on various benchmarks, such as biased action recognition (BAR), biased FFHQ (BFFHQ), CelebA and CivilComments-WILD.
(See Section 4)
Dataset Bias Problem
Classification model.
We first describe the conventional supervised learning setting.
Let us consider the classification problem when a training dataset D n = { ( x i , y i ) } ni =1 with input image x i and corresponding label
y
i is given.
Assuming that there are c ∈ N \ { 1 } classes, y i is assigned to the one element in set C = { 1 , ..., c } .
Note that we focus on a situation where dataset D n does not have noisy samples, for example, noisy labels or out-of-distribution samples (e.g., SVHN samples when the task is CIFAR-10).
When input x i is given, f ( y i | x i , θ ) represents the softmax output of the classifier for label
y i .
It is derived from the model parameter θ ∈ R d .
The cross-entropy (CE) loss L CE is frequently used to train the classifier, and it is defined as L CE ( x i , y i ; θ )
= − log f
( y i | x i , θ ) .
Dataset bias.
Let us suppose that a training set D n is comprised of images, as shown in Figure 1, and that the objective is to classify the digits .
Each image can be described by a set of attributes, ( e.g., for the first image in Figure 1, it can be { digit 0 , red , thin ,...}).
The purpose of the training classifier is to find a model parameter θ that correctly predicts the target attributes, ( e.g., digit ).
Notably, the target attributes are also interpreted as classes .
However, we focus on a case wherein another attribute that is strongly correlated to the target exists, and we call these attributes bias attributes .
For example, in Figure 1, the bias attribute is color .
Furthermore, samples whose bias attributes are highly correlated to the target attributes are called bias-aligned (top three rows in Figure 1).
Conversely, weakly correlated samples are called biasconflicting (see the bottom row of Figure 1).
Therefore, our main scope is that the training dataset which have samples whose bias and target attributes are misaligned.
According to [48], when the bias attributes are easier-to-learn than the target attributes, dataset bias is problematic as the trained model may prioritize the bias attributes over the target attributes.
For example, for a model trained on the images in Figure 1, the model can output class 4 when the ( Orange , 0 ) image ( e.g., left bottom image) is given, due to the wrong priority, color which is an easier-to-learn attribute [48].
PGD: Per-sample Gradient-Norm-Based Debiasing
[Table]
In this section, we propose a novel debiasing algorithm, coined as PGD.
PGD consists of two models, biased f b and debiased f d with parameters θ b and θ d , respectively.
Both models are trained sequentially.
Obtaining the ultimately trained debiased model f d involves three steps: (1) train the biased model, (2) compute the sampling probability of each sample, and (3) train the debiased model.
These steps are described in Algorithm 1.
Step 1: Training the biased model.
In the first step, the biased model is trained on the mini-batches sampled from a uniform distribution U , similar to conventional SGD-based training, with data augmentation A .
The role of the biased model is twofold: it detects which samples are bias-conflicting and calculates how much they should be highlighted.
In doing so, the biased model f b is trained on the generalized cross-entropy (GCE) loss
L GCE [48, 37].
For an input image x and the corresponding true class y , L GCE is defined as L GCE ( x, y ; θ, α ) = 1 − f
( y | x,θ )
α α .
Note that α ∈ (0 , 1] is a hyperparameter that controls the degree of emphasizing the easy-to-learn samples, namely bias-aligned samples.
We set α = 0 .
7 as done by the authors of [68], [48] and [37].
Step 2: Compute the gradient-based sampling probability.
In the second step, the sampling probability of each sample is computed from the trained biased model.
Since rare samples have large gradient norm compared to the usual samples at the biased model [22], the sampling probability of each sample is computed to be proportional to its gradient norm so that bias-conflicting samples are over-sampled.
We propose the following sampling probability of each sample h ( x i , y i ) which is proportional to their gradient norm as follows:
[Equation]
where ∥·∥ rs denotes r square of the L s norm, and θ b is the result of step 1.
Note that computing the gradient for all samples requires huge computing resources and memory.
Therefore, we only extract the gradient of the final FC layer parameters.
This is a frequently used technique for reducing the computational complexity [6, 47, 28, 26, 27].
In other words, instead of h ( x i , y i ) , we empirically utilize ˆ [Equation] i ,
[Equation]
=
[Equation]
, where θ fc is the parameters of the final FC layer.
We consider r = 1 and s = 2 ( i.e., L 2 ), and deliver ablation studies on various r and s in Appendix ?? .
Step 3: Ultimate debiased model training.
Finally, the debiased model f d is trained using minibatches sampled with the probability h
( x i , y i ) obtained in stage 2.
However, [37] argued that just oversampling bias-conflicting samples does not successfully debias, and this unsatisfactory result stems from the data diversity, i.e., data augmentation techniques are required.
Hence, we used simple randomized augmentation operations
A such as random rotation and random color jitter to oversample the bias-conflicting samples.
Experiments
In this section, we demonstrate the effectiveness of PGD for multiple benchmarks compared with previous proposed baselines.
Detail analysis not in this section, e.g., training time, unbiased case study, easier to learn target attribute, sampling probability analysis, reweighting with PGD are described in the Appendix D.
4.1 Benchmarks
To precisely examine the debiasing performance of PGD, we used the BFFHQ, BAR, CelebA, and
CivilComments-WILDS datasets obtained from the real-world used to observe the situations in which general algorithms have poor performance due to bias attributes.
Note that BFFHQ and BAR are biased by using human prior knowledge, while CelebA and CivilComments-WILDS are biased datasets by nature.
A detailed explanation of each benchmark are presented in Appendix B.
4.2 Implementation.
Baselines.
We select baselines available for the official code from the respective authors among debiasing methods without prior knowledge on the bias.
We use eight methods on the various tasks: vanilla network, LfF [48], JTT [43] 3 , Disen [37], GEORGE [59], EIIL [14], BPA[54] and CNC [67].
Implementation details.
We use two types of networks: ResNet18 [21] and pretrained BERT.
For BFFHQ, it uses ResNet18 as a backbone network, and exactly the same setting presented by Disen [37].
For CelebA, we follows experimental setting of [54] which uses ResNet18 as a backbone network.
For CivilCommentsWILDS, we utilize exactly the same hyperparameters of [43] and utilize pretrained BERT.
Detail hyperparameters for CIFAR, BFFHQ, CelebA, CivilComments-WILDS are described in Appendix C.
To reduce the computational complexity in extracting the per-sample gradients, we use only a fully connected layer, similar to [6, 47, 28, 26, 27].
4.3 Results
[Table] [Table]
Similar to the results for the bias-feature-injected benchmarks, as shown in Table 4 and Table 2, PGD shows competitive performance among all the debiasing algorithms on the raw image benchmark (BAR, BFFHQ, and CelebA).
For example, for the BFFHQ benchmark, the accuracy of PGD is 1 .
43% better than that of Disen.
As in Table 2, PGD outperforms the other baselines on CivilCommentsWILDs, much more realistic NLP task.
Therefore, we believe PGD also works well with transformer, and it is applicable to the real-world.
4.4 Further analysis
To prove the superiority of PGD, we provide extensive analysis at the supplementary materials.
For example, we provide empirical analysis about sampling probability of PGD D.2, Parameter sensitivity D.5, Reweighting using per-sample gradient D.8.
Furthermore, we also provide theoretical evidence for understanding PGD at Appendix E.
Conclusion
We propose a gradient-norm-based dataset oversampling method for mitigating the dataset bias problem.
The main intuition of this work is that gradients contains abundant information about each sample.
Since the bias-conflicting samples are relatively more difficult-to-learn than bias-aligned samples, the bias-conflicting samples have a higher gradient norm compared with the others.
Through various experiments and ablation studies, we demonstrate the effectiveness of our gradient-normbased oversampling method, called PGD.
We are still working on a future project: case where the given training dataset is corrupted, such as with noisy labels.
We hope that this study will help improve understanding of researchers about the dataset bias problem.
the case of JTT [43], although the authors used bias label for validation dataset (especially, bias-conflicting samples), we tune the hyperparameters using a part of the biased training dataset for fair comparison.
Considering that JTT does not show significant performance gain in the results, it is consistent with the existing results that the validation dataset is important in JTT, as described in [24].
Acknowledgement
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT)
[No.2019-0-00075, Artificial
Intelligence Graduate School Program(KAIST), 10%] and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) [No.20220-00641, XVoice: Multi-Modal Voice Meta Learning, 90%] A Related Work
Debiasing with bias label.
In [19, 18], a debiased dataset was generated using human labor.
Various studies [3, 29, 45, 58, 62] have attempted to reduce dataset bias using explicit bias labels .
These studies [3, 29, 45, 58, 41, 40], used bias labels for each sample to reduce the influence of the bias labels when classifying target labels.
Furthermore, [61] proposed the EnD regularizer, which entangles target correlated features and disentangles biased attributes.
Several studies [3, 29, 62] have designed DNNs as a shared feature extractors and multiple classifiers.
In contrast to the shared feature extractor methods, [45] and [51] fabricated a classifier and conditional generative adversarial networks, yielding test samples to determine whether the classifier was biased.
Furthermore, [58] proposed a new overlap loss defined by a class activation map (CAM).
The overlap loss reduces the overlapping parts of the CAM outputs of the two bias labels and target labels.
The authors of [40, 41] employed bias labels to detect bias-conflicting samples and to oversample them to debias.
In [43], a reconstructing method based on the sample accuracy was proposed.
The authors of [43] used bias labels in the validation dataset to tune the hyper-parameters.
On the other hand, there has been a focus on fairness within each attribute [20, 66, 50, 1].
Their goal is to prevent bias attributes from affecting the final decision of the trained model.
Debiasing with bias context.
In contrast to studies assuming the explicit bias labels, a few studies [17, 64, 38, 7, 10, 13] assumed that the bias context is known.
In [17, 64, 38], debiasing was performed by directly modifying known context bias.
In particular, the authors of [17] empirically showed that
CNNs trained on ImageNet [16] were biased towards the image texture, and they generated stylized
ImageNet to mitigate the texture bias, while [38] and [64] inserted a filter in front of the models so that the influence of the backgrounds and colors of the images could be removed.
On the other hand, some studies [7, 13, 10], mitigated bias by reweighting bias-conflicting samples: [7] used specific types of CNNs, such as BagNet [9], to capture the texture bias, and the bias was reduced using the
Hilbert-Schmidt independence criterion (HSIC).
In the visual question answering (VQA) task, [13] and [10] conducted debiasing using the entropy regularizer or sigmoid output of the biased model trained on the fact that the biased model was biased toward the question.
Debiasing without human supervision.
Owing to the impractical assumption that bias information is given, recent studies have aimed to mitigate bias without human supervision [35, 48, 15, 30, 37]. [35] identified bias-conflicting samples by sorting the average accuracy of multiple train-test iterations and performed debiasing by training on the samples with low average accuracy.
In [2], each class is divided into two clusters based on IRMv1 penalty [5] using the trained biased model, and train the debiased model so that the output of two clusters become similar.
Furthermore, [30] used
Swap Auto-Encoder [49] to generate bias-conflicting samples, and [15] proposed the modification of the latent representation to generate bias-conflicting samples by using an auto-encoder.
[37] and [48] proposed a debiasing algorithm weighted training by using a relative difficulty score, which is measured by the per-sample training loss.
Specifically, [37] used feature mixing techniques to enrich the dataset feature information.
[54] and [59] proposed unsupervised clustering based debiasing method.
Recently, contrastive learning based method [67] and self-supervised learning method [31] are proposed.
On the other hand, there have been studies [42, 34, 32] that identify the bias attribute of the training dataset without human supervision.
B Benchmarks B.1 Synthetic datasets (Additional synthetic datasets)
To precisely examine the debiasing performance of PGD, we additionally checked the Colored
MNIST, Multi-bias MNIST, and Corrupted CIFAR datasets as synthetic datasets, which assume situations in which the model learns bias attributes first.
We reconstruct MNIST variants, while the others are directly downloaded from the official repositories 4 , 5 and run without any modification.
Colored MNIST
The MNIST dataset [36] is composed of 1-dimensional gray hand-written images.
The size of the image is 28 × 28 .
We inject color into these gray images to give them two main attributes, color and digit shape.
This benchmark comes from related works [48, 30, 37, 7].
First, we choose ten uniformly sampled RGB colors from { C i } i ∈ [10] ∈ R 3 × 10 .
Each sample ( x, y ) is colored by the following steps.
(1) Select bias-confilicting or bias-aligned: random sample u ∼ U (0 , 1) , and set the sample to bias-conflicting samples when u < ρ , otherwise as bias-aligned.
Note that ρ is a ratio of bias-conflicting samples and we use { 0 .
(2) Coloring: color the image with c ∼ N ( C i , σI ) , where i ̸
= y for bias-conflict, or i = y for bias-aligned.
To generate color variation, we inject color for each sample with added noisy i.e.,
C i + N (0 , 0 . 0001) .
We use 55 , 000 samples for training, 5 , 000 samples for validation ( i.e., 10%) and 10 , 000 samples for test.
Remark that test samples are unbiased test set, which means ρ = 90% .
Multi-Bias MNIST
The image size of Multi-bias MNIST is 56 × 56 .
This dataset aims to test the case where there are multiple biased attributes.
To do so, we inject totally 7 bias attributes, {digit color, object 1 (fashion), object 1 color, object 2 (Japanese character), object 2 color, object 3 (English character), object 3 color}, while the target attribute is digit shape.
We inject each bias independently into each sample, as with the colored MNIST case ( i.e., sampling, and injecting bias).
To generate unbiased test set, we also set ρ = 90% for all bias attributes.
As same with Colored MNIST, we use 55 , 000 samples for training, 5 , 000 samples for validation and 10 , 000 samples for test.
Corrupted CIFAR
This dataset is generayted by injecting filters to the CIFAR10 dataset [33].
This benchmark is motivated by the works [48, 37].
In this benchmark, the biased attribute and the target attribute are object and corruption, respectively.
Corruption examples are {Snow, Frost,
Fog, Brightness, Contrast, Spatter, Elastic, JPEG, Pixelate, Saturate}.
We download this benchmark from the official repository of [37].
This dataset is composed of 45 , 000 training dataset and 5 ,validation set, and 10 , 000 for test.
As same with prior datasets, test dataset is composed of unbiased samples.
B.2 Datasets in Section
We will explain the datasets utilized in Section 4.
Biased Action Recognition (BAR)
This dataset comes from the paper [48] for real-world image test.
The goal of this benchmark is classifying 6 actions {Climbing, Diving, Fishing, Racing, Throwing,
Vaulting}, while the places are biased.
Target and bias attributes pairs are (Climbing,RockWall), (Diving,Underwater), (Fishing, WaterSurface), (Racing, APavedTrack), (Throwing, PlayingField), and (Vaulting, Sky).
Bias-conflict samples, for example, is (Climbing, IceCliff), (Diving, Indoor), (Fishing, Forest), (Racing, OnIce), (Throwing, Cave), (Vaulting, Beach).
The number of samples for training is 1 , 941 , and test is 6 , 54 .
To split the training and validation samples, we use 10% validation samples, i.e., 1 , 746 images for training and 195 for validation.
We download training dataset from the official repository.
Biased FFHQ This BFFHQ benchmark is conducted in [37, 30].
Target and biased attributes for bias-aligned samples are (Female, Young), and (Male, Old).
Specifically, ‘Young’ represents, age ranging 10 to 29, and ‘old’ are from 40 to 59.
Oppositely, the bias-conflict samples are (Female, Old) and (Male, Young).
The number of training samples are 19 , 200 , validation samples are 1 , 000 , and test samples are 1 , 000 .
CelebA CelebA [44] is a common real world face classification dataset and each image hasattributes.
The goal is classifying the hair color(“blond" and “not blond") of celebrities which has a spurious correlation with the gender (“male" or “female") attribute.
In fact, only 6% of blond faces are male, so ERM shows poor performance on bias conflict subclass.
We report the average accuracy and the worst-group accuracy on the test dataset.
CivilComments-WILDS CivilComments-WILDS [8] is a dataset to classify whether an online comment is toxic or non-toxic.
Each data is a real online comment, curated on the Civil Comments platform that is a comment plug-in for independent news sites.
The mentions of certain demographic identities (male, female, White, Black, LGBTQ, Muslim, Christian, and other religion) cause the supurious correlation with the label.
Table 1 indicates the portion of toxic comments for each demographic identities [54].
[Table]
C Experiment details C.1 Settings
Architecture.
For the colored MNIST, and Multi-Bias MNIST datasets, we use simple convolutional networks consisting of three CNN layers with kernel size 4, and channel size {8, 32, 64} for each layer.
Also, we utilize average pooling at the end of each layer.
Batch normalization and dropout techniques are used for regularization.
Detailed network configurations are below.
For corrupted CIFAR, BAR, and BFFHQ, we utilize ResNet-18 which is provided by the open source library, torchvision .
For
CelebA, we follows experimental setting of [54] which uses ResNet18 as a backbone network.
For
CivilComments-WILDS, we utilize exactly the same hyperparameters of [43] and utilize pretrained
BERT.
SimConv-1. (conv1): Conv2d(3, 8, kernel_size=(4, 4), stride=(1, 1)) (bn1): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=
True, track_running_stats=
True) (relu1):ReLU() (dropout1):Dropout(p=0.5, inplace=
False) (avgpool1): AvgPool2d(kernel_size=2, stride=2, padding=0) (conv2): Conv2d(8, 32, kernel_size=(4, 4), stride=(1, 1)) (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=
True, track [Table] running
[Table]
True) (relu2):ReLU() (dropout2):Dropout(p=0.5, inplace=
False) (avgpool2): AvgPool2d(kernel_size=2, stride=2, padding=0) (conv3): Conv2d(32, 64, kernel_size=(4, 4), stride=(1, 1)) (relu3): ReLU() (bn3): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=
True, track_running_stats=
True) (dropout3):Dropout(p=0.5, inplace=False) (avgpool3): AdaptiveAvgPool2d(output_size=(1, 1)) (fc): Linear(in_features=64, out_features=$num_class, bias=
True) SimConv-2.
(conv1): Conv2d(3, 8, kernel_size=(7, 7), stride=(1, 1)) (bn1): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=
True, track_running_stats=
True)
(relu1):ReLU
[Table]
C.2 Baselines
(1) LfF [48] trains the debiased model by weighting the bias-conflicting samples based on the “relative difficulty” which is computed by the two loss values from the biased model and debiased model.
To emphasize bias-conflict samples, the authors utilize generalized cross entropy loss with parameter α = 0 .
We implement the LfF algorithm following the official code offered by the authors.
Loss function that this work proposed are as follows.
Details are in the original paper.
[Equation]
Note that W ( z ) is a relative difficulty and that GCE is a generalized cross-entorpy.
z denotes feature, which is the output of the penultimate layer, and C · is fully connected layer. (2) JTT [43] aims to debiasing by spliting dataset into correctly learned and failed to learned samples.
To do so, JTT trains the biased model first and split the given training dataset as follows:
[Equation]
Then, train the ultimate debiased model by oversampling D error-set with λ up times.
We set λ up as 1 ./ρ for all experiments.
We reproduce the results by utilizing the official code offered by the authors.
The main strength of PGD compared to JTT is that PGD does not need to set a hyperparameter.
(3) Disen [37] aims to debias by generating abundant features from mixing features between samples.
To do so, the authorus train the biased and debiased model by aggregating features from both networks.
This work also utilized “relative difficulty” that is proposed in LfF [48].
We reproduce the results utilizing the official code offered by the authors.
The loss function proposed in this work is as follows.
Details are provided in the original paper.
[Equation]
where
[Equation]
terms z
(4) GEORGE [59] aims to debias by measuring and mitigating hidden stratification without requiring access to subclass labels.
Suppose there are given n datapoints, x 1 , · · · , x n ∈ χ and associated superclass (target) labels y 1 , · · · , y n ∈ { 1 , · · · , C } .
In addition, associated with each datapoint x i has latent (unobserved) subclass label z i ∈ { 1 , · · · , K } .
GEORGE consists of 3 steps.
To do so, the authorus train the biased based on ERM first.
Next, to estimate approximate subclass (latent) label, apply UMAP dimensionality reduction [46] to the features of given training dataset at ERM model and cluster the output of the reduced dimension for the data of each superclass into K clusters, where K is chosen automatically.
The detail description of clustering process is provided in the original paper.
Lastly, to improve performance on these estimated subclass, they minimize the maximum per-cluster average loss(i.e.
( x, y ) ∼ ˆ P ˜ z ), by using the clusters as groups in the GDRO objective [52].
The loss function proposed in this work is as follows:
[Equation]
where f θ and L are parameterized feature extractor and classifier, respectively. (5) BPA[54] aims to debias by using the technique of feature clustering and cluster reweighting.
It consists of 3 steps.
To do so, the authorus train the biased model ˜ θ based on ERM first.
Next, at the biased model ˜ θ , cluster all training examples into K clusters based on the feature derived from the
ERM model ˜
θ , where K is hyperparameter.
Here, h ( x, y ; ˜ θ ) ∈ K = { 1 , · · · , K } denote the cluster mapping function of data ( x, y ) derived by ˜ θ .
At the last step, they calculate the proper importance weight, w k , to the k -th cluster, whekre k ∈ K and the final objective of debiasing framework is given by minimizing a weighted empirical risk as follows:
[Equation]
where w h ( x,y ;˜ θ ) ( θ ) denote the importance weight to the cluster h ( x, y ; ˜ θ ) at the model θ .
Concretely, for arbitrary iteration index T , w h ( x,y ;˜ θ ) ( θ T ) is derived from the momentum method based on the history set H T , defined by
[Equation]
where N k is number of the data belong to k -th cluster.
Details are provided in the original paper. (6) CNC [67] aims to debias by learning representations such that samples in the same class but different groups are close to each other.
CNC is composed of 2 steps: (1) Inferring pseudo group labels, (2) Supervised contrastive learning.
First, get ERM based model f ˆ θ and get pseudo prediction ˆ y , standard argmax over the final-layer outputs of f ˆ θ .
Next, training the debiased model based on supervised contrastive learning using pseudo prediction ˆ y .
The detail process of contrastive learning for each iteration is like below:
• From the selected batch, sample the one anchor data (x,y).
• Construct the set of positives samples { ( x + m , y + m ) } which is belong to the batch, satisfying y + m = y
and ˆ y + m ̸
= ˆ y .
• Similarly, construct the set of negative samples { ( x − n , y − n ) } which is belong to the batch, satisfying y − n ̸
= y and ˆ y − n = ˆ y .
• With loss of generality, assume cardinality of positive set and negative set are M and N , respectively.
• Weight update based on the gradient of the loss function ˆ L ( f θ ; x, y ) , the detail is like below:
[Equation]
f θ ; x, y ) . Here, λ ∈ [0 , 1] is a hyperparameter and ˆ L cross ( f θ ; x, y ) is an average cross-entropy loss over x , the M positives, and N negatives.
Moreover, f enc is the feature extractor part of f θ and the detail formulation of ˆ L supcon ( x, { x + m } Mm =1 , { x − n } Nn =1 ; f enc ) is like below:
[Equation]
(7) EIIL [14] proposes a novel invariant learning framework that does not require a prior environment knowledge.
EIIL is composed of 3-step process:
(1) ERM training (2) Environment inference (EI) (3)
Invariant Learning (IL).
First, get a biased model w ◦ Φ by minimizing ERM.
Note that Φ and w are feature extractor and classifier, respectively.
Next, based on feature extractor Φ , optimize the EI objective to infer environments of each training data.
The object of EI is sorts training examples D into D
∪ D 2 that maximally separate the spurious features so that facilitate effective invariant learning.
Concretely, get a q ⋆ = argmax q ∥∇ ˜ w =1 ˜ R e
( ˜ w ◦ Φ) ∥ , where ˜ R e ( ˜ w ◦ Φ) = 1 |D| (cid:80) |D| i =
1 q i ( e ) l ( ˜ w ◦ Φ( x i ) , y i ) .
Note that e ∈ { 1 , 2 } and q i (1)+ q i (2) =for all i ∈ { 1 , · · · , |D|} .
Based on q ⋆ , get a bernoulli sample ˆ q ∼ Bernoulli ( q ⋆ ) , and split D in to Dand D 2 based on result of binary value ˆ q .
Lastly, to get a debiased model, optimize the classifier w and feature extractor Φ by minimizing invariant learning objective (e.g. IRM or GroupDRO) from the gained two pseudo environment Dand D 2 .
C.3 Implementation Details
Image processing We train and evaluate with a fixed image size.
For colored MNIST case, 28 ×and Multi-bias MNIST , 56 × 56 , Corrupted CIFAR, 32 × 32 , and real-world datasets of 224 × 224 .
For all cases, we utilize random rotation and color jitter to avoid overfitting.
We use normalizing with mean 0 .
4822 , 0 . 4465) , and standard deviation (0 . 2023 , 0 . 1994 , 0 . 2010) .
Implementation For Table 3 and Table 4 reported in Section 4, we reproduce all experimental results referring to other official repositories: 6 7 8 9 .
Except for JTT, all hyperparameters for corrupted CIFAR, BAR, and BFFHQ follow previously reported parameters in repositories.
We grid-search for other cases, MNIST variants.
We set the only hyperparameter of PGD, q, as 0.7, which is proposed by the original paper [68].
A summary of the hyperparameters that we used is reported in Table 2. [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
For Table 2 reported in Section 4, we follow the implementation settings of CelebA and
CivilComments-WILDS, suggested by [54] and [43], respectively.
A summary of the hyperparameters that we used is reported in Table 2.
We conduct our experiments mainly using a single Titan
XP GPU for all cases.
https://
D Additional experiments D.1 Unbiased test accuracy on synthetic datasets
[Table] [Table]
Accuracy results.
In Table 3, we present the comparisons of the image classification accuracy for the unbiased test sets.
The proposed method outperforms the baseline methods for all benchmarks and for all different ratios.
For example, our model performance is 35 .
94% better than that of the vanilla model for the colored MNIST benchmark with a ratio ρ = 0 .
For the same settings, PGD performs better than Disen by 2 .
As pointed out in [57], colored MNIST is too simple to evaluate debiasing performance on the basis of the performance of baselines.
In Multi-bias MNIST case, other models fail to obtain higher unbiased test results, even though the ratio is high, e.g., 10% .
In this complex setting, PGD shows superior performance over other methods.
For example, its performance is higher by 36 . 15% and 35 .
63% compared with the performance of vanilla model and Disen for the ratio of 10% .
Similar to the results for the bias-feature-injected benchmarks, as shown in Table 4 and Table 2, PGD shows competitive performance among all the debiasing algorithms on the raw image benchmark (BAR and BFFHQ).
For example, for the BFFHQ benchmark, the accuracy of PGD is 1 .
43% better than that of Disen.
D.2 Correlation between gradient norm and bias-alignment of the CMNIST
To check if the per-sample gradient norm efficiently separates the bias-conflicting samples from the bias-aligned samples, we plot the gradient norm distributions of the colored MNIST (CMNIST).
For comparison, we normalized the per-sample gradient norm as follows:
[Equation]
As in Figure 6, the bias-aligned
D.3 PGD does not learn only the second-easiest feature
We provide the results of the following experimental setting: the target feature is color and the bias feature is digit shape, i.e., the task is to classify the color, not the digit shape.
Let us give an example of this task.
When one of the target classes is Red, this class is aligned with one of the digits (e.g., “0”).
In other words, bias-aligned samples in this class are (Red, “0”), and the bias-conflicting samples are (e.g., (Red, “1”), (Red, “2”), ..., (Red,”9”)).
Note that, as shown in LfF [48], color is empirically known to be easier to learn than digit shape, we think that the above scenario reflects the concern: whether PGD is only targeting to learn the second-easiest feature (digit shape).
Therefore, if the concern is correct, PGD may fail in this Color
[Figure]
target MNIST scenario since the model will learn digit shape.
However, as shown in the table below, vanilla, PGD, and LfF perform well in that case.
[Figure]
PGD
[Figure]
[Figure]
[Table]
We can also support this result by seeing the distribution of the normalized gradient norms, ∥∇ θ L CE ( x i , y i ; θ b ) ∥ / max ( x i ,y i ) ∈D n ∥∇ θ L CE
( x i , y i ; θ b ) ∥ ∈ [0 , 1] , extracted from the biased model θ b (trained in the Step 1 in Algorithm 1 of the Section 3).
If PGD aims to learn the second-easiest feature (digit shape), it will highlight bias-conflicting samples that have abundant digit shapes, so that the ultimate debiased model can consider them important.
[Table]
The numbers filled in the Table 6 is the number of data belonging to each bin category.
We can check that there are no bias-conflicting samples whose gradient norm is significantly larger than the bias-aligned samples.
In other words, PGD does not force the debiased model to learn the digit shape (i.e., the second-easiest feature) in this scenario.
This scenario brings the similar performance comparing to Vanilla.
D.4 PGD on unbiased datasets
[Table]
natural CIFAR10 cases.
In addition, we check whether PGD fails in the unbiased and conventional public datasets.
In this case, target is the first-easiest feature because no specific attribute makes bias.
To verify this, we report two types of additional results: (1) unbiased colored MNIST ( ρ = 90% ), and (2) conventional public datasets, i.e., CIFAR 10.
We follow experimental settings of colored MNIST for the unbiased colored MNIST.
On the other hand, we train ResNet18 [21] for CIFAR 10 with the SGD optimizer, 0 .
9 momentum, 5 e − 4 weight decay, 0 .
1 learning rate, CosineAnnealing
LR decay scheduler.
As in Table 7, PGD does not suffer a significant performance degradation in unbiased Colored MNIST.
Furthermore, it performs better than the vanilla model on the CIFAR 10 dataset.
This means that the training distribution that PGD changes does not cause significant performance degradation.
In other words, PGD can only balance the training data set, regardless of whether the training dataset is balanced or unbalanced.
D.5 Ablation study on GCE parameter α
[Table]
The only hyper-parameter used in PGD is the GCE parameter α .
We experimented with this value at 0 .according to the protocoal of LfF [48].
However, we need to compare the various cases α on the colored
MNIST benchmark for analysis.
As in Table 8, when the GCE parameter is 0 . 9 , the debiased model performs best.
This is because the biased model is fully focused on the bias feature, rather than seeing the target feature, which can be seen from the unbiased test accuracy of the biased model (see the bottom of Table 8).
D.6 Computation cost
[Table]
Debiasing requires more modules, which increases the computational cost.
The training time for the colosed MNIST ρ = 0 .
5% is reported in Table 9.
As in the top of Table 9, we report the computational cost of four algorithms, vanilla, LfF, Disen, and PGD.
Here, PGD spends a longer training time.
To check which parts generate longer time, we measure part-by-part cost, at the bottom of Table 9.
Note that Steps 1, 2, and 3 represent training the biased model, computing per-sample gradient norm, and training the debiased model, respectively.
We can conclude the following two facts.
(1) Step 2 (computing per-sample gradient norm) spends 4 .
3% of training time.
Resampling requires an additional cost of 1 m 27s.
D.7 Multi-stage vs Single-stage
[Table]
PGD computes per-sample gradient only once, between training the biased model and the debiased model.
However, an update of the persample gradient can be performed repeatedly at each epoch ( i.e., Single-stage).
In other words, the PGD can be modeled to run the following loop: updating the biased model → updating the sampling probability → updating the debiased model.
In this section, we justify why we compute per-sample gradient at once ( i.e., Multi-stage).
We report the performance of Multi-stage and Single-stage of PGD on the colored MNIST dataset.
As in
Table 10, the single-stage method has two characteristics:
(1) It takes a long training time compared to the multi-stage method.
(2) It has lower unbiased test accuracy compared to the multi-stage method.
The reason why it takes longer training time is that computing the per-sample gradient norm requires huge computation resources.
On the other hand, the sampling probability of the single-stage method changes training distribution over epochs, the debiased model suffers unstable training, so that it loses the debiasing performance.
D.8 Resampling versus Reweighting
[Table]
To support our algorithm design, we provide further experimental analysis, i.e., resampling versus reweighting.
Reweighting [48, 37] and resampling [43] are the two main techniques to debias by up-weighting bias-conflicting samples.
PGD is an algorithm that modifies the sampling probability by using the per-samplge gradient norm.
To check whether PGD works with reweighting, we examine the results of PGD with reweighting on colored MNIST datasets and report in Table 11.
We compute the weight for each sample as follows:
[Equation]
As in Table 11, PGD with resampling slightly outperforms PGD with reweighting.
As argued in [4], his gain from resampling can be understood by the arguments that resampling is more stable and better than reweighting.
[Figure]
D.9 Unbiased test accuracy on various norms.
Since, gradient norm can have various candidates, such as order of the norm, we report four configurations of gradient norms.
As shown in Figure 7, all norms have significant unbiased test performance.
Amongst them, the L 2 -norm square case shows lower unbiased performance than the other cases.
Therefore, it is recommended that any first power of L { 1 , 2 , ∞} -norms be used in PGD for overcoming the dataset bias problem.
This is quite different from the results with [23], which suggested that
L 1 -norm is the best choice in the research field of out-of-distribution detection.
D.10 Ablation study
[Table] [Figure] [Table] [Figure]
[Table]
Table 12 shows the importance of each module in our method: generalized cross entropy, and data augmentation modules.
We set the ratio ρ as 0 .
5% and 10% for colored
MNIST and multi-bias MNIST, respectively.
We observe that GCE is more important that data augmentation for colored MNIST.
However, data augmentation shows better performance than GCE for multi-bias MNIST.
For all cases, the case where both are utilized outperforms the other cases.
E Mathematical Understanding of PGD
This section provides a theoretical analysis of per-sample gradient-norm based debiasing.
We first briefly summarize the maximum likelihood estimator (MLE) and Fisher information (FI) which are ingredients of this section.
We then interpret the debiasing problem as a min-max problem and deduce that solving the min-max problem can be phrased as minimizing the trace of the inverse
FI.
Since handling the trace of the inverse FI is very difficult owing to its inverse computation, we look at a glance by relaxing it into a one-dimensional toy example.
In the end, we conclude that the gradient-norm based re-sampling method is an attempt to solve the dataset bias problem.
E.1 Preliminary
Training and test joint distributions.
The general joint distribution P ( x, y | θ ) is assumed to be factored into the parameterized conditional distribution f
( y | x, θ ) and the marginal distribution P ( x ) , which is independent of the model parameter θ , i.e., P ( x, y | θ ) =
P ( x ) f ( y | x, θ ) .
We refer to the model f
( y | x, θ ⋆ ) that produces the exact correct answer, as an oracle model, and to its parameter θ ⋆ as the oracle parameter.
The training dataset D n is sampled from { ( x i , y i ) } ni =1 ∼ p ( x ) f ( y | x, θ ⋆ ) , where the training and test marginal distributions are denoted by p ( x ) and q ( x ) , respectively.
Here, we assume that both marginal distributions are defined on the marginal distribution space M =
{P ( x ) | (cid:82) x ∈X P ( x ) dx = 1 } , where X means the input data space, i.e., p ( x ) , q ( x ) ∈ M .
The space H of sampling probability h .
When the training dataset D n is given, we denote the sampling probability as h ( x ) which is defined on the probability space H 10 :
[Equation]
Maximum likelihood estimator (MLE).
When h ( x ) is the sampling probability, we define MLE ˆ θ h ( x ) , D n as follows:
[Equation]
Note that MLE ˆ θ h ( x ) , D n is a variable controlled by two factors: (1) a change in the training dataset
D n and (2) the adjustment of the sampling probability h ( x ) .
If h ( x ) is a uniform distribution U ( x ) , then ˆ θ U ( x ) , D n is the outcome of empirical risk minimization (ERM).
Fisher information (FI).
FI, denoted by I P ( x ) ( θ ) , is an information measure of samples from a given distribution P ( x, y | θ ) .
It is defined as:
[Equation]
FI provides a guideline for understanding the test cross-entropy loss of MLE ˆ θ U ( x ) , D n .
When the training set is sampled from p ( x ) f ( y | x, θ ⋆ ) and the test samples are generated from q ( x ) f
( y | x, θ ⋆ ) , we can understand the test loss of MLE ˆ θ U ( x ) , D n by using FI as follows.
Theorem 1.
Suppose Assumption 1 in Appendix F and Assumption 2 in Appendix G hold, then for sufficiently large n = |D n | , the following holds with high probability: [Equation] [Equation] n )
[Equation]
[Equation]
Here is the proof sketch.
The left-hand side of equation 4 converges to the Fisher information ratio (FIR) Tr (cid:2)
I p ( x ) ( θ ⋆ ) − 1 I q ( x ) ( θ ⋆ ) (cid:3) related term.
Then, FIR can be decomposed into two trace terms with respect to the training and test marginal distributions p ( x ) and q ( x ) .
Finally, we show that the term Tr[ I p ( x ) ( θ ⋆ ) − 1 ] which is defined in the oracle model parameter can be replaced with
Tr[ I p ( x ) (ˆ θ U ( x ) , D n ) − 1 ] .
The proof of Theorem 1 is in Appendix D. Note that Theorem 1 means that the upper bound of the test loss of MLE ˆ θ U ( x ) , D n can be minimized by reducing Tr
[ I p ( x ) ( ˆ θ U ( x ) , D n ) − 1 ] .
Empirical Fisher information (EFI).
In practice, the exact FI equation 3 cannot be computed since we do not know the exact data generation distribution P ( x ) f ( y | x, θ ) .
For practical reasons, the empirical Fisher information (EFI) is commonly used [25, 11] to reduce the computational cost of gathering gradients for all possible classes when x is given.
In this study, we used a slightly more generalized EFI that involved the sampling probability h ( x ) ∈ H as follows:
[Equation]
Note that the conventional EFI is the case when h ( x ) is uniform.
EFI provides a guideline for understanding the test cross-entropy loss of MLE ˆ θ h ( x ) , D n .
E.2 Understanding dataset bias problem via min-max problem
Debiasing formulation from the perspective of min-max problem.
Improving generalization is understandable as reducing (min) the loss of test samples that are difficult (max) to infer.
There was a trial for formulating the dataset bias problem when multiple training datasets from different training distributions are given [5].
The problem is seen differently when we can only control the sampling probability h ( x ) ∈ H on the given one training dataset D n .
The problem, in our view, can be defined from the left-hand side of equation 4 in a practical point of view as follows:
Definition 1.
When the training dataset D n ∼ p ( x ) f ( y | x, θ ⋆ ) is given, the debiasing objective is
[Equation]
[Equation] The meaning of Definition 1 is that we have to train the model ˆ θ h ( x ) , D n so that the loss of the worst case test samples ( max q ( x ) ) is minimized by controlling the sampling probability h ( x ) ( min h ( x ) ).
Note that since we cannot control the given training dataset D n and test marginal distribution q ( x ) , the only controllable term is the sampling probability
h ( x ) .
Therefore, from Theorem 1 and EFI, we design a practical objective function for the dataset bias problem as follows:
[Equation]
E.3 Meaning of PGD in terms of equation 7.
In this section, we present an analysis of PGD with respect to equation 7.
To do so, we try to understand equation 7, which is difficult to directly solve.
It is because computing the trace of the inverse matrix is computationally expensive.
Therefore, we intuitively understand equation 7 in the one-dimensional toy scenario.
One-dimensional example.
We assume that D n comprises sets M and m such that elements in each set share the same loss function 12 ( θ + a ) 2 for all elements in M and 12 ( θ − a ) 2 for all elements in m with a given constant a .
In other words, there are only two distinguishable points, and we construct the training data from only these two points.
We also assume that each sample of M and m has the set dependent probability mass h M ( x ) and h m ( x ) , respectively.
With these settings, our objective is to determine h ⋆ ( x ) =
arg min h ( x ) ∈H Tr[ˆ
I h ( x ) (ˆ θ h ( x ) , D n ) − 1 ] .
Thanks to the model’s simplicity, we can easily find h ⋆ ( x ) in a closed form with respect to the gradient at ˆ θ U ( x ) , D n for each set, i.e., g M (ˆ θ U ( x ) , D n ) and g m (ˆ θ U ( x ) , D n ) .
Theorem 2.
Under the above setting, the solution of ( h ⋆M ( x ) , h ⋆m ( x )) =
arg min
[Equation]
x ) ∈H Tr[ˆ I h ( x ) (ˆ θ h ( x ) , D n ) [Equation] 1 ] is:
[Equation]
where Z = | M || g M (ˆ θ U ( x ) , D n ) | + | m ||
g m (ˆ θ U ( x ) , D n ) | , and | M | and | m | denote the cardinality of M and m , respectively.
The proof of Theorem 2 is provided in Appendix E. Note that h ⋆M ( x ) and h ⋆m ( x ) are computed using the trained biased model with batches sampled from the uniform distribution U ( x ) .
It is the same with the second step of PGD.
Vanilla
PGD (Ours)
T r [I U ( θ h , D n ) - 1 ] 0 2 × 104 ×
Bias conﬂict ratio ρ 0 . 0 0 5 0 .
9 T r 𝐼 $ ! "
𝜃 $ ! " , 𝒟 ! % & (a) Colored MNIST
T r [I U ( θ h , D n ) - 1 ] 0 2 × 10 6 4 ×6 6 × 10 6 8 ×
Bias conﬂict ratio ρ 0 .
9 T r 𝐼 $ ! "
𝜃 $ ! " , 𝒟 ! % & (b) Multi-bias MNIST
Tr[
[Equation]
I h (
[Equation] θ h [Equation] [Equation]
[Equation]
PGD : h ( x ) =
ˆ h ( x ) , and vanilla:
[Equation] ( x ) = U ( x ) .
PGD tries to minimize equation 7.
Theoremimplies that equation 7 can be minimized by sampling in proportion to their gradient norm.
Because the fundamental of PGD is oversampling based on the gradient norm from the biased model, we can deduce that PGD strives to satisfy equation 7.
Furthermore, we empirically show that PGD reduces the trace of the inverse of EFI in the high-dimensional case, as evident in Figure 8.
F Backgrounds For Theoretical Analysis
F.1 Notations Summary.
For convenience, we describe notations used in Appendix E, Appendix F, G, and H.
[Table]
sampling probability of
[Table]
sample in
[Table]
satisfying h (
[Table]
[Table]
x -
[Table]
distribution of
[Table] [Table]
distribution of test [Table]
[Table]
) conditional distribution with model parameter θ -
[Table]
) general joint distribution with model parameter θ
[Table] ( x ) f
[Table]
| [Table] [Table]
) joint distribution [Table]
[Table]
all possible h ( x ) [Table] D n [Table]
[Table] [Table] } M set [Table]
P [Table] [Table]
- supp [Table] ) Support set of P ( [Table]
[Table]
[Table]
F.2 Main Assumption
Here, we organize the assumptions that are used the proof of Theorems.
These are basically used when analyzing models through Fisher information as follows.
The assumptions are motivated by [60].
Assumption 1. (A0).
The general joint distribution P ( x, y | θ ) is factorized into the conditional distribution f ( y | x, θ ) and the marginal distribution P ( x ) , not depend on model parameter θ , that is:
[Equation]
Thus, the joint distribution is derived from model parameter θ and the marginal distribution
P ( x ) , which is determined from the task that we want to solve.
Without loss of generality, we match the joint distribution’s name with the marginal distribution.
(A1).
(Identifiability): The CDF P θ (whose density is given by P ( x, y | θ ) ) is identifiable for different parameters.
Meaning that for every distinct parameter vectors θ 1 and θ 2 in Ω , P θand P θ 2 are also distinct.
That is,
[Equation]
where X , { 1 , · · · , c } and Ω are input, label and model parameter space, respectively. (A2).
The joint distribution P θ has common support for all θ ∈ Ω . (A3).
(Model Faithfulness) :
For any x ∈ X , we assume an oracle model parameter θ ⋆ that generates y true ( x ) , a true label of input x with the conditional distribution f
( y true ( x ) | x, θ ⋆ ) = 1 . (A4).
(Training joint): Let p ( x ) denote the training marginal with no dependence on the parameter.
Then, the set of observations in D n ≜ { ( x 1 , y 1 ) .
· · · ( x n , y n ) } are drawn independently from the training/proposal joint distribution of the form p ( x, y | θ ⋆ ) ≜ p ( x ) f ( y | x, θ ⋆ ) , because we don’t think the existence of mismatched label data situation in the training data. (A5).
(Test joint): Let q ( x ) denote the test marginal with no dependence on the parameter.
The unseen test pairs are distributed according to the test/true joint distribution of the form q ( x, y | θ ⋆ ) ≜ q ( x ) f ( y | x, θ ⋆ ) , because we don’t think the existence of mismatched label data situation in the test task. (A6).
(Differentiability): The log-conditional log f ( y | x, θ ) is of class C 3 (Ω) for all ( x, y ) ∈
X × { 1 , 2 , · · · , c } , when being viewed as a function of the parameter.(A7).
The parameter space Ω is compact
and there exists an open ball around the true parameter of the model θ ⋆ ∈ Ω .
(A8).
(Invertibility): The arbitrary Fisher information matrix I P ( x ) ( θ ) is positive definite and therefore invertible for all θ ∈ Ω .
(A9).
{ ( x, y ) ∈ supp ( q ( x, y | θ ⋆ )) | ∇ 2 θ log q ( x, y | θ ⋆ ) is singular } is a measure zero set.
Compare to [60], we modify (A3) so that the oracle model always outputs hard label i.e., f ( y true ( x ) | x, θ ⋆ )
= 1 and add (A9) which is not numbered but noted in the statement of Theorem 3 and Theorem 11 in [60].
F.3 Preliminaries
We organize the two background knowledge, maximum likelihood estimator (MLE) and Fisher information (FI), needed for future analysis.
F.3.1 Maximum Likelihood Estimator (MLE)
In this section, we derive maximum likelihood estimator in the context of classification problem with sampling probability h ( x ) .
Unless otherwise stated, training set D n = { ( x i , y i ) } n i =1 is sampled from p ( x, y | θ ⋆ ) .
For given probability mass function (PMF) h ( x ) on D n , in abstract h ( x ) ∈ H , we define
MLE ˆ θ h ( x ) , D n as follows:
[Equation]
In equation 9 and equation 10, (A4) and (A0) of the Assumption 1 in Appendix F was used, respectively.
Note that MLE ˆ θ h ( x ) , D n is a variable controlled by two factors: (1) a change in the training dataset D n and (2) the adjustment of the sampling probability h ( x ) .
If h ( x ) is a uniform distribution U ( x ) , then ˆ θ U ( x ) , D n is the outcome of empirical risk minimization (ERM).
F.3.2 Fisher information (FI)
General definition of FI.
Fisher information (FI), denoted by I P ( x ) ( θ ) , is an information measure of samples from the given distribution P ( x, y | θ ) ≜ P ( x ) f ( y | x, θ ) .
It is defined as the expected value of the outer-product of the score function ∇ θ log P ( x, y | θ ) with itself, evaluated at some θ ∈ Ω .
[Equation]
Extended version of FI.
Here, we summarize extended version of FI, which can be derived by some assumptions.
These variants of FI are utilized in the proof of Theorems.
• (Hessian version) Under the differentiability condition (A6) of Assumption 1 in Appendix F, FI can be written in terms of the Hessian matrix of the log-likelihood:
[Equation]
• (Model decomposition version) Under the factorization condition (A0) of Assumption 1 in Appendix F, equation 12 and equation 13 can be transformed as follows:
[Equation]
Specifically, equation 14 and equation 15 can be unfolded as follows:
[Equation]
Hereinafter [Equation] we define I p ( x ) ( θ ) and I q ( x ) ( θ ) as the FI derived from the training and test
marginal,
respectively
F.3.3 Empirical Fisher information (EFI)
When the training dataset D n is given [Equation] we denote the sampling probability as h ( x ) which is defined on the probability space H :
[Equation]
Practically, the training dataset D n is given as deterministic.
Therefore, equation 14 can be refined as empirical Fisher information (EFI).
This reformulation is frequently utilized, e.g., in [25, 11], to reduce the computational complexity of gathering gradients for all possible classes ( i.e., expectation with respect to f ( y | x, θ ) as in equation 14).
Refer the (cid:80) cy =1 term of equation 16.
Different with prior
EFI which is defined on the case when h ( x ) is uniform, U ( x ) , we generalize the definition of EFI in terms of h ( x ) ∈ H as follows:
[Equation]
Note that (a) holds owing to equation 17.
F.3.4 Stochastic Order Notations o p and O p
For a set of random variables X n and a corresponding set of constant a n , the notation X n = o p ( a n ) means that the set of values X n /a n converges to zero in probability as n approaches an appropriate limit.
It is equivalent with X n /a n = o p (1) , where X n = o p (1) is defined as:
[Equation]
The notation X n = O p ( a n ) means that the set of values X n /a n is stochastically bounded.
That is ∀ ϵ > 0 , ∃ finite M > 0 , N > 0 s.t. P ( | X n /a n | > M ) <
ϵ for any n > N .
G Theorem
In this section, we deal with some require sub-lemmas which are used for the proof of Lemmawhich is main ingredient of the proof of Theorem 3.
G.1 Sub-Lemmas
Lemma 1 ([39], Theorem 5.1) .
When P −→ denotes convergence in probability, and if (A0) to (A7) of the Assumption 1 in Appendix F hold, then there exists a sequence of MLE solutions { ˆ θ U ( x ) , D n } n ∈ N
that ˆ θ U ( x ) , D n P −→ θ ⋆ as n −→ ∞ , where θ ⋆ is the ‘true’ unknown parameter of the distribution of the sample.
Proof.
We refer to [39] for a detailed proof.
Lemma 2 ([39], Theorem 5.1) .
Let { ˆ θ U ( x ) , D n } n ∈ N be the MLE based on the training data set D n . If (A0) to (A8) of the Assumption 1 in Appendix F hold, then the MLE ˆ θ U ( x ) , D n has a zero-mean normal asymptotic distribution with the covariance equal to the inverse Fisher information matrix, and with the convergence rate of 1/2:
[Equation]
where D −→ represents convergence in distribution.
Proof.
We refer to [39] for a detailed proof, which is based on Lemma 1.
Lemma 3 ([65], Theorem 9.18) .
Under the (A0) to (A8) of the Assumption 1 in Appendix F hold, we get
[Equation]
where D −→ represents convergence in distribution.
Proof.
We refer to [65] for a detailed proof, which is based on Lemma 2.
Lemma 4.
([55], Chapter 1) Let { θ n } be a sequence of random vectors.
If there exists a random vector ˜ θ such that θ n D −→ ˜ θ , then ∥ θ n − ˜ θ ∥ 2 = O p (1) , where ∥ · ∥ 2 denote the L 2 norm.
Proof.
We refer to [55] for a detailed proof.
Lemma 5.
([60], Theorem 27) Let { θ n } be a sequence of random vectors in a convex and compact set Ω ⊆ R d and θ
⋆ ∈ Ω be a constant vector such that ∥ θ n − θ ⋆ ∥ 2 = O p ( a n ) where a n −→ 0 (as n −→ ∞ ).
If g : Ω −→ R is a C 3 function, then
[Equation]
Proof.
We refer to [55] for a detailed proof.
Lemma 6.
If (A0) and (A3) of the Assumption 1 in Appendix F hold, then ∇ θ log P ( x, y true ( x ) | θ ⋆ )
=
⃗ 0 for any joint distribution P ( x, y | θ ⋆ ) .
Proof.
[Equation]
On the first equality, (A0) of the Assumption 1 in Appendix F is used.
At the second equality, (A3) of the Assumption 1 in Appendix F is used.
Lemma 7.
If (A0) to (A8) of the Assumption 1 in Appendix F hold and the case ∇ 2 θ log p ( x, y true ( x ) | θ ⋆ ) is non-singular for given data ( x, y true ( x ))
satisfies, then the asymptotic distribution of the log-likelihood ratio is a mixture of first-order Chi-square distributions, and the convergence rate is one.
More specifically:
[Equation]
where { λ i ( x, y true ( x )) } di =1 are eigenvalues of I p ( x ) ( θ ⋆ ) − 12 (cid:8) −∇ 2 θ log p ( x, y true ( x ) | θ ⋆ ) (cid:9) I p ( x ) ( θ ⋆ ) − 12 .
Proof.
The proof is based on taylor expansion theorem.
Remind that we deal with the data ( x, y true ( x )) satisfying ∇ 2 θ log p ( x, y true ( x ) | θ ⋆ ) is non-singular.
From the property √ n (ˆ θ U ( x ) , D n − θ ⋆ ) D −→ N ( ⃗ 0 , I p ( x ) ( θ ⋆ ) − 1 ) derived from Lemma 3, one concludes that √ n ∥ ˆ θ U ( x ) , D n − θ ⋆ ∥ 2 = O p (1) and therefore ∥ ˆ θ U ( x ) , D n − θ ⋆ ∥ 2 = O p ( 1 √ n ) by the Lemma 4.
Thus, by the Lemma 5, log
[Equation]
hold.
By the Lemma 3 and the property ∇ θ log p ( x, y true ( x ) | θ ⋆ ) =
⃗ 0 derived by Lemma 6, we can obtain
[Equation]
Define Γ ( x, y true ( x )) as − I p ( x ) ( θ ⋆ ) − 12 ∇ 2 θ log p ( x, y true ( x ) | θ ⋆ )
I p ( x ) ( θ ⋆ ) − 12 and rewrite the righthand-side element-wise 13 as
[Equation]
where { λ i ( x, y true ( x )) } di =1 are eigenvalues of Γ ( x,
y true ( x )) .
Thus, desired property is obtained.
G.2 Main Lemma
In this section, we derive main Lemma which represents the test cross entropy loss can be understood as Fisher information ratio (FIR) [60].
G.2.1 Main Lemma statement and proof
Lemma 8 (FIR in expected test cross entropy loss with MLE) .
If the Assumption 1 in Appendix F holds, then
[Equation]
Proof.
We prove Lemma 8 via two steps.
First we show that the expected cross entropy loss term can be rewritten in terms of log-likelihood ratio.
Then, we prove that the expected log-likelihood ratio can be asymptotically understood as FIR.
Step 1: Log-likelihood ratio We show that the expected log-likelihood ratio can be formulated as expected test cross-entropy loss as follows:
This property holds because,
[Equation]
Since (A0) of Assumption 1 in Appendix F, equation 21 and equation 22 hold.
At equation 23, the properties, (i) Supp ( q ( x, y | θ ⋆ )) ⊆ W and (ii) f
( y | x, θ ⋆ ) =
1 ∀ ( x, y ) ∈ W , was used which is derived by (A3) of the Assumption 1 in Appendix F.
Step 2: FIR Here, we show that the expected test loss of MLE can be understood as FIR.
By (A0) in Assumption 1 in Appendix F, trivially
[Equation]
holds.
Since equation [Equation] and (A9) in Assumption 1 in Appendix F
[Equation]
Supp ( q ( x, y | θ ⋆ )) can be replaced by,
[Equation]
when calculate expectation.
We can get a result of Lemma 8 as follows:
[Equation]
x )
[Equation]
| x
[Equation]
⋆ )
[Equation]
lim n
[Equation]
n
[Equation]
) p
[Equation]
[Equation]
[Equation]
=
E ( x
[Equation]
y true ( x )) ∼
[Equation]
f
( y | x
[Equation]
θ ⋆ )
[Equation]
lim
[Equation] ) (cid:34) [Equation] x
[Equation]
) p
[Equation] x [Equation] ( x )
[Equation]
( x ) [Equation] D n ) (cid:35) [Equation]
(cid:35) (28) = E ( x [Equation] y true ( x ))
[Equation]
( x ) f (
[Equation]
| x [Equation] θ ⋆ ) (cid:34)(cid:32) [Equation]
(cid:88)
[Equation]
(cid:2)
[Equation]
(cid:3)(cid:33)
[Equation]
S (cid:35) (29) =
E ( x
[Equation]
y true ( x ))
[Equation]
( x
[Equation]
[Equation]
x
[Equation]
⋆ ) (cid:34)(cid:32)
[Equation]
(cid:88)
[Equation]
( x
[Equation]
( x )
[Equation]
(cid:33)
[Equation]
(cid:35) (30) = E ( x [Equation] y true ( x )) ∼
[Equation]
( x ) [Equation] ( y | x [Equation]
θ ⋆ ) [Equation] (cid:104) I p ( x ) ( θ ⋆ ) − 12 (cid:8) −∇ 2 θ log p ( x
[Equation]
y true ( x ) | θ ⋆ ) (cid:9) I p ( x ) ( θ ⋆ ) − 12 (cid:105) 1 S (cid:21) (31) = E ( x,y ) ∼ [Equation] ( x ) f ( y | x
[Equation]
34) equation 26 holds from Step 1.
From equation 25, equation 27 satisfied.
Since ( x, y ) is sampled from q ( x ) f
( y | x.θ ⋆ ) , equation 28 and equation 32 holds.
From equation 29 to equation 31, the result of
Lemma 7 is used.
equation 33 can be obtained thanks to (A0).
Lastly, equation 34 holds because trace satisfies the commutative law about matrix multiplication.
The last term, Tr (cid:2)
I p ( x ) ( θ ⋆ ) −
I q ( x ) ( θ ⋆ ) (cid:3) is named as Fisher information ratio (FIR) since it is understanble as a ratio when the scalar case.
G.3 Theorem
In this section, we finally prove Theorem 3.
To do so, we additionally follow assumptions in [60].
G.3.1 Additional Assumption
Assumption 2.
We assume to exist four positive constants L 1 , L 2 , L 3 , L 4 ≥ 0 such that following properties hold ∀ x ∈ X, y ∈ { 1 , · · · , c } and θ ∈ Ω :
• I ( θ, x ) = −∇
2 θ log f
( y | x, θ ) is independent of the class labels y. • ∇ θ log f
( y | x, θ ⋆ ) T I q ( x ) ( θ ⋆ ) − 1 ∇ θ log f
( y | x, θ ⋆ ) ≤ L• ∥
I q ( x ) ( θ ⋆ ) − 1 / 2 I ( θ ⋆ , x ) I
q ( x ) ( θ ⋆ ) − 1 / 2 ∥
≤ L• ∥
I q ( x ) ( θ ⋆ ) − 1 / 2 ( I ( θ ′ , x ) − I ( θ ′′ , x ))
I q ( x ) ( θ ⋆ ) − 1 / 2 ∥
≤ L 3 ( θ ′ − θ ′′ ) T I q ( x ) ( θ ⋆ )( θ ′ − θ ′′ ) • − L 4 ∥
θ − θ ⋆ ∥
I ( θ ⋆ , x ) ⪯ I ( θ, x ) − I ( θ ⋆ , x ) ⪯ L 4 ∥
θ − θ ⋆ ∥
I ( θ ⋆ , x [-)G.3.
-])
x , D n
Lemma 9.
Suppose Assumption 1 in Appendix F and Assumption 2 in Appendix G hold, then with high probability:
[Equation]
Proof.
It is shown in the proof of Lemma 2 in [12] that under assumptions mentioned in the
Assumption 2, the following inequalities hold with probability 1 − δ ( n ) :
[Equation]
where β ( n ) and 1 − δ ( n ) are proportional to n , which is size of training set D n .
Note that for any marginal distribution P ( x ) , I P ( x ) ( θ ) =
E
x ∼P ( x )
[ I ( θ, x )] holds because of the independence for the class labels y of I ( θ, x ) .
Taking the expectation to the equation 36 with respect to the marginal p ( x ) and q ( x ) , then:
[Equation]
Since I p ( x ) ( θ ⋆ ) and I p ( x ) ( ˆ θ U ( x ) , D n ) are assumed to be positive definite, we can write equation 37 in terms of inverted matrices 15 :
[Equation]
equation 38 is equivalent to
[Equation]
From equation 39, β ( n ) −
[Equation]
[Equation] [Equation]
[Equation]
[Equation]
[Equation]
[Equation] [Equation]
[Equation]
[Equation] [Equation]
satisfies.
Thus,
[Equation]
[Equation]
holds when taking n → ∞ to the equation 40.
Note that β ( n ) is proportional to n .
G.4 Statement and proof of Theorem
Theorem 3.
Suppose Assumption 1 in Appendix F and Assumption 2 in Appendix G hold, then for sufficiently large n = |D n | , the following holds with high probability:
[Equation] [Equation]
Proof.
Because of the (A8) of Assumption 1 in Appendix F, I p ( x ) ( θ ⋆ ) − 1 and I q ( x ) ( θ ⋆ ) are positive definite matrix.
Thus, Tr (cid:2) I p ( x ) ( θ ⋆ ) − 1 I q ( x ) ( θ ⋆ ) (cid:3) ≤ Tr (cid:2) I
[Equation] x [Equation] ( θ ⋆ )
[Equation]
(cid:3) Tr (cid:2) I q ( x ) ( θ ⋆ ) (cid:3) holds.
From the result of Lemma 8 and 9 in Appendix G,
[Equation]
holds with high probability.
Thus, desired result is obtained.
Note that Theorem 3 means that the upper bound of the test loss of MLE ˆ θ U ( x ) , D n can be minimized by reducing Tr (cid:104) I p ( x ) (ˆ θ U ( x ) , D n ) − 1 (cid:105) when training marginal p ( x ) is the only tractable and controllable variable.
H Theorem
In this section, we introduce the motivation of gradient norm based importance sampling.
To get the motivation, we introduce the debiasing object problem for a given D n under sampling probability h ( x ) and provide evidence of solution of this problem in the toy example setting because of the problem difficulty.
H.1 Practical objective function for the dataset bias problem
Remark that right-hand side term of equation 42 are controlled by the training and test marginals p ( x ) , and q ( x ) .
Since we can only control the training dataset D n not p ( x ) and q ( x ) , we can design a practical objective function for the dataset bias problem by using EFI and Theorem 3 as follows:
[Equation]
Then, Tr( B − A ) = (cid:80) di =1 u i ( B − A ) u
Ti where ˆ
I h ( x ) ( θ ) is an empirical Fisher information matrix.
Remark that EFI is defined as:
[Equation]
Here, h ( x ) describes the sampling probability on D n , which is the only controllable term.
We deal with equation 43 in the toy example because of the problem difficulty.
H.2 One-dimensional Toy Example Setting
For simplicity, we assume that D n comprises sets M and m and the samples in each set share their same loss function and same probability mass.
The detail is like below:
• For the given a ∈ R , at the model parameter θ ∈ R , 12 ( θ + a ) 2 and 12 ( θ − a ) 2 loss function arise for all data in M and m , respectively.
• ˆ θ h, D n denote the trained model from the arbitrary PMF h ( x ) ∈ H which has a constraint having degree of freedom 2, ( h M ( x ) , h m ( x )) .
• Concretely, each samples of M and m has a probability mass h M ( x ) and h m ( x ) , respectively. i e., | M | · h M ( x ) + | m | · h m ( x ) = 1 , where | M | and | m | denote the cardinality of M and m , respectively. • Let g M ( θ ) and g m ( θ ) denote the gradient of each sample in M and m at θ ∈ R , respectively.
Then, | M | · h M ( x ) · g M (ˆ θ h ( x ) , D n ) + | m | · h m ( x ) ·
g m (ˆ θ h ( x ) , D n ) = 0 hold by the definition of ˆ θ h ( x ) , D n .
• In this settings, our objective can be written as finding
[Equation]
and this is equivalent to find ( h ⋆M ( x ) , h ⋆m ( x )) .
H.3 Statement and proof of Theorem
In this section, we introduce the motivation of the gradient norm based importance sampling in the toy example setting.
Theorem 4.
Under the above setting, the solution of ( h ⋆M ( x ) , h ⋆m ( x )) =
arg min h ( x ) ∈H Tr (cid:104) ˆ I h ( x ) (ˆ θ h ( x ) , D n ) − 1 (cid:105) is:
[Equation]
where Z = | M || g M (ˆ θ U ( x ) , D n ) | + | m ||
g m (ˆ θ U ( x ) , D n ) | , and | M | and | m | denote the cardinality of M and m , respectively.
Proof.
The trained model ˆ θ h ( x ) , D n ∈ [ − a, a ] trivially holds for any h ( x ) ∈ H .
By the loss function definition in the toy setting, g M ( θ ) =
θ + a and g m ( θ ) =
θ − a , ∀ θ .
Thus, | g M ( θ ) | + | g m ( θ ) | = 2 a satisfies for ∀ θ ∈ [ − a, a ] .
Since the gradient is scalar in the toy setting, ˆ I h ( x ) (ˆ θ h ( x ) , D n ) is also scalar and same as unique eigenvalue, that is,
[Equation]
Thus, our problem is deciding h M ( x ) and h m ( x ) that maximize | M | · h M ( x ) · { g M
[Equation]
Below 3 constrains are hold for arbitrary θ ∈ [ − a, a ] and h ( x ) ∈ H , because of the toy setting. | M | · h M ( x ) + | m | · h m ( x ) = 1 . (probability definition)
[Equation]
Note that convex linear sum of the sample’s gradient w.r.t.
h ( x ) =
( h M ( x ) , h m ( x )) is zero at the trained model ˆ θ h ( x ) , D n .
[Equation]
Note that this is derived by the property of predefined loss function at θ ∈ [ − a, a ] . 2nd constraint is equivalent to | M | · h M ( x ) · (2 a + g m (ˆ θ h ( x ) , D n ))
+ | m | · h m ( x ) · g m (ˆ θ h ( x ) , D n ) = 0 .
⇔ g m (ˆ θ h ( x ) , D n ) = − 2 a | M | · h M ( x ) .
Because of the 3rd constraint, g M (ˆ θ h ( x ) , D n ) = 2 a (1 − | M | · h M ( x )) .
Then the objective is, maximizing
[Equation]
equation 45 is maximized when | M | · h M ( x ) = 12 , and it means | m | · h m ( x ) = 12 .
Thus, h ⋆M ( x ) = | m | 2 | M |·| m | , h ⋆m ( x ) =
| M | 2 | M |·| m | .
This result is related with the trained model ˆ θ U ( x ) , D n , where U M ( x ) =
U m ( x ) = 1 | M | + | m | .
At ˆ θ U ( x ) , D n , | M | g M (ˆ θ U ( x ) , D n ) + | m | g m (ˆ θ U ( x ) , D n ) =
0 satisfies and this is equivalent to | M | : | m | =
| g m (ˆ θ U ( x ) , D n ) | : | g M (ˆ θ U ( x ) , D n ) | .
Thus, it is coincide with our intuition that setting the sampling probability h for set M and m with proportion to | g M (ˆ θ U ( x ) , D n ) | and |
g m (ˆ θ U ( x ) , D n ) | helps to minimize the trace of the inverse empirical
Fisher information.